Overview
We are looking for a skilled Data Engineer with strong experience across Databricks, SQL, Python, and AWS data services. The ideal candidate should be proficient in building end-to-end data pipelines, working with large datasets, and ensuring efficient data processing and transformation.
Key Responsibilities
- Design, develop, and maintain end-to-end data pipelines using Databricks.
- Work extensively with SQL and Python to support data transformations, processing, and analysis.
- Implement and optimize data workflows across AWS services such as S3, Secrets Manager, and other foundational cloud components.
- Leverage AWS Athena, Glue, and Redshift for data extraction, cataloging, processing, and warehousing.
- Develop and maintain shell scripts for automation and operational workflows.
- Work with PostgreSQL databases for data modeling, querying, and optimization.
- Collaborate with cross-functional teams to understand data requirements and ensure data quality, consistency, and availability.
- Troubleshoot data issues, optimize query performance, and ensure efficient data processing.
- Apply strong data engineering and data understanding skills to solve complex business challenges.
Required Skills & Experience
- 45 years of experience in Data Engineering.
- Strong hands-on experience with:
- Databricks (end-to-end development)
- SQL (advanced querying, performance tuning)
- Python (data processing, automation)
- Solid knowledge of AWS Cloud services including S3, Secrets Manager, and other basic AWS components.
- Experience with Athena, Glue, and Redshift.
- Basic understanding of Shell scripting.
- Strong data modeling, data pipeline design, and data troubleshooting skills.
- Experience with PostgreSQL.
Nice to Have
- Exposure to data governance, data validation, or CI/CD for data pipelines.
- Experience with any orchestration tools (e.g., Airflow, AWS Step Functions).
