Job Description

ETL/PySpark Data Engineer

The ETL/PySpark Data Engineer will play a crucial role in designing, developing, and maintaining data pipelines that ensure high-quality, reliable data for our analytics and reporting needs. This position is vital for transforming raw data into meaningful insights, driving data-driven decisions across the organization. The ideal candidate will have a strong background in data engineering, proficiency in PySpark, and a passion for data-driven solutions.

Experience Required:

3+ years

Key Responsibilities:

Design and implement ETL processes to extract, transform, and load data from various sources.
Develop and optimize PySpark scripts for large-scale data processing.
Collaborate with data scientists and analysts to understand data requirements and deliver data solutions.
Ensure data quality and integrity throughout the data pipeline lifecycle.
Troubleshoot and resolve data processing issues to maintain pipeline performance.
Automate repetitive data tasks to improve efficiency and reduce manual effort.
Participate in the design and implementation of data warehousing solutions.
Create and maintain documentation for data processes and data models.
Conduct performance tuning and optimization of data pipelines.
Stay updated with the latest trends and technologies in data engineering.

Qualification Criteria:

Bachelor's degree in Computer Science, Data Science, or a related field.
Proven experience in ETL processes and data pipeline development.
Strong proficiency in PySpark and Python.
Experience with data warehousing solutions such as Amazon Redshift, Snowflake, or Google BigQuery.
Familiarity with database management systems like MySQL, PostgreSQL, or MongoDB.
Excellent problem-solving skills and attention to detail.
Strong understanding of data modeling and data architecture.
Ability to work collaboratively in a fast-paced, team-oriented environment.
Strong communication skills to articulate technical concepts to non-technical stakeholders.
Experience with CI/CD tools and practices.