Lead Software Engineer

Exp: 6 to 11 yrs

Key Responsibilities

• Design and develop scalable data pipelines using Databricks (Spark, PySpark, SQL)
• Build and optimize ETL/ELT workflows for large-scale data processing
• Work with Delta Lake for data management, performance tuning, and reliability
• Collaborate with data scientists to productionize ML models on Databricks
• Implement ML pipelines including data preparation, feature engineering, and model deployment
• Ensure data quality, governance, and security best practices
• Integrate Databricks solutions with cloud platforms (AWS/Azure/GCP)
• Monitor, troubleshoot, and optimize performance of data workflows
• Contribute to CI/CD pipelines and automation for data and ML workflows

Required Skills & Qualifications

• 6–11 years of experience in Data Engineering / Big Data ecosystem
• Strong hands-on experience with Databricks, Apache Spark, and PySpark
• Proficiency in SQL and Python
• Experience with Delta Lake, Unity Catalog (preferred)
• Strong understanding of data modeling and data warehousing concepts
• Experience working on cloud platforms (AWS / Azure / GCP)
• Exposure to Machine Learning workflows (training, deployment, monitoring)
• Familiarity with ML libraries like scikit-learn, MLflow, TensorFlow, or similar
• Experience with workflow orchestration tools (Airflow, ADF, etc.)
• Strong problem-solving and debugging skills

Good to Have

• Experience with MLflow for experiment tracking and model management
• Knowledge of MLOps practices
• Experience in real-time/streaming pipelines (Kafka, Structured Streaming)
• Exposure to Data Governance & Security frameworks
• Certification in Databricks or cloud platforms

Soft Skills

• Strong communication and stakeholder management skills
• Ability to work in agile, fast-paced environments
• Collaborative mindset with a focus on continuous learning