Job Description: Data Engineer with Databricks Expertise
Job Summary
We are seeking a highly skilled Data Engineer with Databricks certifications and extensive practical experience working with the Databricks Lakehouse platform and its supporting capabilities. The ideal candidate will possess a robust understanding of AWS services relevant to data engineering and demonstrate proficiency in data pipeline development, programming, data modeling, and governance.
Key Responsibilities
- Design and implement data pipelines using AWS and Databricks.
- Manage and orchestrate data workflows, ensuring efficient data ingestion processes.
- Utilize Delta Lake for data storage and management, ensuring secure data sharing across platforms with Delta Sharing.
- Implement data governance and access control using Unity Catalog.
- Apply the Medallion architecture (Bronze, Silver, Gold layers) to projects.
- Develop and manage ETL pipelines using AWS Glue.
- Leverage Amazon S3 for data storage and management.
- Utilize AWS Lambda and Step Functions for serverless computing in data processing.
- Apply data warehousing solutions and best practices in data transformation.
- Monitor and implement data quality checks and governance practices.
- Collaborate with data scientists, analysts, and other stakeholders in an Agile environment.
- Deliver large-scale data processing and analytics projects within cloud environments.
Required Skills And Qualifications
- Databricks certifications with practical experience in the Databricks Lakehouse platform.
- Proficient in AWS services: Glue, S3, Lambda, Step Functions, and data warehousing solutions.
- Strong programming skills in Python, Scala, or Java.
- Proficient in SQL for querying and manipulating data.
- Experience in data modeling for Lakehouse architecture.
- Knowledge of ETL best practices and data transformation techniques.
- Understanding of data governance frameworks and practices.
- Proven track record of delivering data engineering projects, particularly in cloud environments.
- Experience with large-scale data processing and analytics projects.
- Ability to work in Agile teams and collaborate effectively with various stakeholders.
Preferred Qualifications
- Advanced AWS certifications.
- Experience with additional cloud platforms (e.g., Azure).
- Familiarity with other data engineering tools and frameworks.
