Senior Data Engineer ML Operations Specialist (Databricks)

We are seeking an experienced Senior Data Engineer ML Operations Specialist to lead the design, implementation, and optimization of our data lakehouse architecture using Databricks. The ideal candidate will architect end‑to‑end data pipelines, orchestrate machine learning workflows, and ensure data quality and governance across our AWS cloud environment. This role requires deep expertise in Spark, PySpark, Python, and modern ML tooling such as MLflow, TensorFlow, and PyTorch, as well as infrastructure automation with Terraform or CloudFormation. The successful applicant will collaborate closely with data scientists, product teams, and DevOps to deliver scalable, reliable data solutions that drive business insights.

Responsibilities

• Design, develop, and maintain scalable data lakehouse solutions on Databricks, ensuring high performance and reliability.
• Architect and implement ETL/ELT pipelines using Spark, PySpark, and Python, integrating with AWS services such as S3, Redshift, and Glue.
• Build and manage ML pipelines, including data preprocessing, model training, deployment, and monitoring using MLflow or equivalent tools.
• Automate infrastructure provisioning and configuration with Terraform or CloudFormation, ensuring reproducible environments.
• Develop and enforce data quality standards using tools like DQX, and implement automated data validation and monitoring.
• Create and maintain RESTful APIs, Docker containers, and Kubernetes deployments to expose data services and ML models.
• Collaborate with cross‑functional teams to translate business requirements into technical specifications and deliver actionable insights.
• Continuously evaluate and adopt new technologies and best practices to improve data processing efficiency and model performance.

Requirements

• Proven experience in advanced data engineering, including data modeling, ETL/ELT design, and architecture.
• Strong proficiency with Databricks, Spark, PySpark, and Python for data engineering.
• Hands‑on experience with ML pipeline orchestration, MLflow, TensorFlow, and PyTorch.
• Experience with infrastructure as code tools such as Terraform or CloudFormation.
• Knowledge of data quality tooling (e.g., DQX) and SQL for data manipulation.
• Familiarity with AWS cloud services, REST APIs, Docker, and Kubernetes.
• Excellent problem‑solving skills and ability to work independently in a fast‑paced environment.
• Strong communication skills to collaborate with data scientists, product managers, and stakeholders.