AWS Databricks with Python

Key responsibilities for this role include designing, developing, and maintaining ETL/ELT data pipelines in Databricks using Spark, and integrating Databricks with various AWS services like S3, Glue, Lambda, and Redshift. They contribute to data lake and data warehouse architecture and modeling, orchestrate workflows with tools like Databricks Workflows or Airflow, and optimize performance and troubleshoot issues. Collaboration with stakeholders to define data requirements, implement quality checks, and ensure data governance is also crucial.

Required Skills & Qualifications

Required skills include proven experience in data engineering with Databricks and AWS data services, proficiency in Python or Scala and advanced SQL, and expertise in Apache Spark. In-depth knowledge of core AWS data services and a solid understanding of data warehousing principles, ETL/ELT, and data modeling are essential. Familiarity with DevOps, Git, and orchestration tools is also needed. A bachelor's degree is typical, and relevant certifications like AWS Certified Data Engineer or Databricks Certified Data Engineer are beneficial.