Senior Data Engineer

5-7 Years

Not Disclosed

Job Description

Senior Data Engineer with 5+ years of experience

Mandatory Skills:

-Extensive experience in developing ETL pipelines,

- Databricks

-scheduling using Azure Data Factory (ADF) and/or Apache Airflow,

- performance tuning using SQL.

- SQL, Python, Spark, PySpark, and Azure.

- Fabric experience preferred.

Key Responsibilities:

ETL Pipeline Development:
Design, develop, and maintain ETL pipelines to extract, transform, and load data from various sources into Azure data platforms.
Optimize ETL processes for performance and scalability.
Implement data quality and validation checks within ETL processes.
Scheduling and Orchestration:
Develop and manage data integration workflows using Azure Data Factory (ADF) and/or Apache Airflow.
Schedule and monitor ETL jobs to ensure timely data processing and availability.
Performance Tuning:
Optimize SQL queries and data processing tasks for performance.
Utilize notebooks for development and performance tuning of ETL processes.
Data Integration and Management:
Develop and maintain data models and schemas.
Ensure data integrity, consistency, and security across data platforms.
Collaborate with data engineers, solution architects, and other stakeholders to understand business requirements and translate them into technical solutions.

Qualifications:

Bachelor's degree in computer science, Information Technology, or a related field.
5-6 years of experience as a Data Engineer in developing ETL pipelines and scheduling using ADF and/or Apache Airflow.
Strong technical skills in SQL, Python, Spark, PySpark, and Azure.
Proven experience in performance tuning and optimization of SQL and ETL processes.
Strong understanding of Data Architecture and Data Modeling.
Experience in healthcare and pharmaceutical industries is preferred.
Excellent problem-solving and analytical skills.
Strong communication and collaboration skills.

Preferred Skills: