SENIOR DATA ENGINEER - Azure Databricks

Key Responsibilities:

• Willingness to work in second shift to support global stakeholders and business operations (CST (US)).
• Actively support BAU operations (L2 & L3) including incident triage, root cause analysis, bug fixes, data issue resolution, and performance tuning.
• Support BAU operations including incident triage, root cause analysis, bug fixes, and performance tuning.
• Build and maintain documentation, runbooks, and operational playbooks for pipelines and workflows.
• Enable observability through logging, lineage capture, and monitoring frameworks.
• Contribute to reusable frameworks, accelerators, and standardization of development practices.
• Work with downstream consumers (BI, analytics, data science) to optimize data availability and usability.
• Design, build, and maintain scalable data pipelines and workflows on modern data platforms (primarily Databricks and Azure stack).
• Implement and operationalize DataOps practices including monitoring, and alerting for data pipelines.
• Translate business/data requirements into efficient data models, transformations, and reusable components.
• Own ingestion, transformation, and curation layers (bronze/silver/gold) ensuring data quality, consistency, and performance.
• Collaborate with architects, SMEs, and cross-functional teams (platform, BI, and support) for seamless delivery and handoffs.

Skills & Experience:

• 610 years of experience in data engineering and pipeline development.
• Strong hands-on expertise in Databricks (PySpark, Delta Lake, Workflows, notebooks).
• Proficiency in Python, SQL, and distributed data processing frameworks.
• Experience with Azure data services such as ADF, ADLS, Azure DevOps, and integration patterns.
• Hands-on experience in building and maintaining ETL/ELT pipelines at scale.
• Nice to have: CI/CD, version control, testing, and monitoring knowledge
• Familiarity with data quality frameworks, logging, and observability tools.
• Working knowledge of data governance concepts (catalog, lineage, access control).
• Experience supporting production environments with SLAs and incident management.