Exp: 6 to 11 yrs
Key Responsibilities
- Design and develop scalable data pipelines using Databricks (Spark, PySpark, SQL)
- Build and optimize ETL/ELT workflows for large-scale data processing
- Work with Delta Lake for data management, performance tuning, and reliability
- Collaborate with data scientists to productionize ML models on Databricks
- Implement ML pipelines including data preparation, feature engineering, and model deployment
- Ensure data quality, governance, and security best practices
- Integrate Databricks solutions with cloud platforms (AWS/Azure/GCP)
- Monitor, troubleshoot, and optimize performance of data workflows
- Contribute to CI/CD pipelines and automation for data and ML workflows
Required Skills & Qualifications
- 6–11 years of experience in Data Engineering / Big Data ecosystem
- Strong hands-on experience with Databricks, Apache Spark, and PySpark
- Proficiency in SQL and Python
- Experience with Delta Lake, Unity Catalog (preferred)
- Strong understanding of data modeling and data warehousing concepts
- Experience working on cloud platforms (AWS / Azure / GCP)
- Exposure to Machine Learning workflows (training, deployment, monitoring)
- Familiarity with ML libraries like scikit-learn, MLflow, TensorFlow, or similar
- Experience with workflow orchestration tools (Airflow, ADF, etc.)
- Strong problem-solving and debugging skills
Good to Have
- Experience with MLflow for experiment tracking and model management
- Knowledge of MLOps practices
- Experience in real-time/streaming pipelines (Kafka, Structured Streaming)
- Exposure to Data Governance & Security frameworks
- Certification in Databricks or cloud platforms
Soft Skills
- Strong communication and stakeholder management skills
- Ability to work in agile, fast-paced environments
- Collaborative mindset with a focus on continuous learning
