Key Responsibilities:
- Data Pipeline Development: Design and develop robust data processing pipelines and analytics solutions using Databricks.
- Architecture & Modeling: Architect scalable and efficient data models and storage solutions tailored for the Databricks environment.
- Migration & Integration: Collaborate with architecture and engineering teams to migrate existing solutions to Databricks.
- Performance Optimization: Tune and optimize Databricks clusters and job performance to meet SLAs and evolving business needs.
- Governance & Compliance: Implement best practices for data governance, security, and compliance on the Databricks platform.
- Mentorship: Guide and mentor junior engineers, offering technical support and best practice recommendations.
- Innovation & Learning: Stay informed about emerging data engineering trends and new features in Databricks and related technologies.
Required Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related technical discipline.
- 5 to 8 years of overall experience in data engineering, with a minimum of 2 years focused on Databricks.
- Hands-on proficiency in Python, Scala, or SQL.
- Solid understanding of distributed computing principles and big data frameworks like Apache Spark.
- Experience with major cloud platforms (AWS, Azure, or GCP) and their respective data services.
- Demonstrated ability to build scalable and reliable data solutions in dynamic environments.
- Strong analytical and problem-solving skills with high attention to detail.
- Effective communication and collaboration skills within cross-functional teams.
Preferred Qualifications:
- Familiarity with containerization and orchestration tools such as Docker and Kubernetes.
- Understanding of DevOps practices for automating deployments and monitoring data pipelines.
