Job Description
As an ETL Engineer, you will be a part of our dynamic Information Technology team. You will be responsible for designing, developing, and supporting ETL solutions using Python, Pyspark, SQL, and Azure. You will be working closely with various stakeholders to gather and define data requirements, and will be instrumental in the design and implementation of data extraction, transformation, and load (ETL) processes.
Responsibilities of ETL Engineer
- Design, develop, and maintain ETL processes using Python, Pyspark, SQL, and Azure.
- Collaborate with various stakeholders to understand and define data requirements.
- Develop and implement ETL procedures for data migration and integration.
- Ensure the accuracy and integrity of data through the design and implementation of appropriate error handling processes.
- Perform data quality checks, validation, and performance tuning.
- Collaborate with data architects and modelers to design and implement data models.
- Develop and maintain technical documentation related to ETL processes.
- Troubleshoot and resolve any issues related to ETL processes.
- Stay updated with the latest ETL and data warehousing trends and technologies.
- Special emphasis on working with cloud platforms like AWS, GCP.
- The ideal candidate should have strong analytical thinking skills, excellent communication management abilities, and a positive attitude. Experience with AWS and GCP will be highly valued.
Functional Competencies
- ETL: The candidate should have a strong understanding of ETL (Extract, Transform, Load) processes and be able to design and implement complex data pipelines. This includes the ability to extract data from various sources, transform it to fit business needs, and load it into the end system.
- Python: Proficiency in Python is required as it is one of the most commonly used programming languages in data engineering. The candidate should be able to write efficient, reusable, and reliable Python code.
- PySpark: The candidate should have experience with PySpark, the Python library for Apache Spark. This includes the ability to use PySpark for big data processing and analytics.
- SQL: The candidate should have a strong understanding of SQL (Structured Query Language) and be able to write complex queries to manipulate and analyze data.
- Azure: Experience with Azure, Microsoft's cloud computing service, is required. The candidate should be familiar with its data storage and processing services.
Behavioral Competencies
- Analytical Thinking: The candidate should have strong analytical skills, including the ability to analyze complex data sets and derive insights from them.
- Communication Management: The candidate should have excellent communication skills, including the ability to clearly explain complex technical concepts to nontechnical stakeholders.
- Positive Attitude: The candidate should have a positive attitude and be able to maintain a positive work environment, even in challenging situations.
Good to have skills
- AWS: The candidate should have experience with AWS (Amazon Web Services), Amazon's cloud computing service. This includes familiarity with its data storage and processing services.
- GCP: The candidate should have experience with GCP (Google Cloud Platform), Google's cloud computing service. This includes familiarity with its data storage and processing services.
Employee type: FullTime
Work Location: Hyderabad
