Posting title: Lead Data Engineer (Databricks)
Experience: 7+ Years
Location: Pune/ Ahmedabad
Work mode: Hybrid
Primary skills: Databricks, Data Modelling/ Data Lakehouse, Gen AI, SQL, Python/Scala, Azure Data Factory (ADF)/AWS Glue, Docker, Team Lead
Qualification:Any Technical Degree
Roles & Responsibilities:
- Lead the design, development, and optimization of data solutions using Databricks, ensuring they are scalable, efficient, and secure.
- Collaborate with cross-functional teams to gather and analyse data requirements, translating them into robust data architectures and solutions.
- Develop and maintain ETL pipelines, leveraging Databricks and integrating with Azure Data Factory as needed.
- Implement machine learning models and advanced analytics solutions, incorporating Generative AI to drive innovation.
- Ensure data quality, governance, and security practices are adhered to, maintaining the integrity and reliability of data solutions.
- Provide technical leadership and mentorship to junior engineers, fostering an environment of learning and growth.
- Stay updated on the latest trends and advancements in data engineering, Databricks, Generative AI, and Azure Data Factory to continually enhance team capabilities.
Required Qualifications:
- Bachelor's or master's degree in computer science, Information Technology, or a related field.
- 7+ to 10 years of experience in data engineering, with a focus on Databricks.
- Proven expertise in building and optimizing data solutions using Databricks and integrating with Azure Data Factory/AWS Glue.
- Proficiency in SQL and programming languages such as Python or Scala.
- Strong understanding of data modelling, ETL processes, and Data Warehousing/Data Lakehouse concepts.
- Familiarity with cloud platforms, particularly Azure, and containerization technologies such as Docker.
- Excellent analytical, problem-solving, and communication skills.
- Demonstrated leadership ability with experience mentoring and guiding junior team members.
Preferred Skills:
- Experience with Generative AI technologies and their applications.
- Familiarity with other cloud platforms, such as AWS or GCP.
- Knowledge of data governance frameworks and tools.
