Senior Data Engineer Databricks, ADF, PySpark

Key Responsibilities:

• Build scalable ETL pipelines and implement robust data solutions using Azure technologies.
• Manage and orchestrate workflows with Azure Data Factory (ADF), Databricks, ADLS Gen2, and Azure Key Vault.
• Design, maintain, and optimize secure and efficient data lake architectures.
• Collaborate with stakeholders to gather requirements and translate them into detailed technical specifications.
• Implement CI/CD pipelines to enable automated, seamless data deployment leveraging Azure DevOps.
• Monitor and troubleshoot data quality, performance bottlenecks, and scalability issues in production pipelines.
• Write clean, modular, and reusable PySpark code adhering to Agile development methodologies.
• Maintain thorough documentation of data pipelines, architecture designs, and best practices for team reuse.

Must-Have Skills:

• 6+ years of experience in Data Engineering roles.
• Strong expertise with SQL, Python, PySpark, Apache Spark.
• Hands-on experience with Azure Databricks, Azure Data Factory (ADF), ADLS Gen2, Azure DevOps, and Azure Key Vault.
• Deep knowledge of data warehousing concepts, ETL development, data modeling, and governance.
• Familiarity with Agile software development lifecycle (SDLC) and containerization tools like Docker.
• Commitment to clean coding practices and maintaining high-quality codebases.

Good to Have Skills:

• Experience with Azure Event Hubs and Logic Apps.
• Exposure to Power BI for data visualization.
• Strong problem-solving skills with a background in logic building and competitive programming.