Data Engineer
Job Description
We are looking for an experienced Data Engineer (5+ years) with strong hands-on expertise in Databricks, DLT, PySpark, Python, SQL, and CI/CD for our client.
Key Responsibilities & Requirements
- Databricks DLT: Hands-on experience in creating, executing, troubleshooting, and maintaining DLT pipelines, particularly for real-time data — Mandatory.
- Unit Testing: Hands-on experience with PyTest, mocking, unit testing, and integration testing — Mandatory.
- PySpark: Strong experience with DataFrames, joins, transformations, and aggregations.
- Python & SQL: Strong hands-on experience in both Python and SQL — Mandatory.
- Git: Experience with branching, commit, push, pull, and merge.
- Bamboo / CI-CD: Experience with build and deployment processes.
- Repository / Code Review: Experience with PR creation, code review, and code push.
- JIRA: Experience with ticket creation, bug tracking, and sprint workflows.
- Deployment & Debugging: Experience managing Dev → QA → Prod deployment flows and troubleshooting issues across environments.
More Info
About Company
Skills:
Unix Shell Scripting, Cloudformation, Pyspark, Spark, Python, Sql, Service Catalog
Skills:
Pyspark, AWS Glue, automation, Scripting, data pipelines, data quality frameworks, cloud-based data processing, ETL workflows
Skills:
T-sql, SQL Server, Azure Sql, Sql, ELT, Devops, Git, Azure Data Factory, Data Lake, Data Warehousing, Python, Etl