Data Engineer with Databricks, AWS

'Data Engineer – AWS Databricks (Offshore)

Experience 5–8 years

Primary Responsibilities

• Support data engineering activities for enterprise data modernization initiatives using AWS Databricks and Lakehouse architecture.
• Analyze and validate existing data ingestion pipelines, CDC processes, AWS S3 landing zone, and Bronze layer datasets.
• Perform source-to-Bronze and Bronze-to-Silver data validation, reconciliation, and data quality assessment.
• Analyze existing transformation logic, including deduplication, standardization, cleansing, and business rule implementations.
• Develop and maintain data transformation pipelines using Databricks, Spark, PySpark, and SQL.
• Support implementation and validation of Silver and Gold layer data structures aligned with enterprise data models.
• Perform data profiling to identify completeness, accuracy, consistency, and quality issues.
• Develop SQL queries and validation scripts for data reconciliation and testing activities.
• Collaborate with Data Modelers and Business Analysts to validate data structures, mappings, and business rules.
• Support source-to-target mapping documentation and technical lineage documentation.
• Troubleshoot data pipeline issues, performance challenges, and data quality defects.
• Implement logging, error handling, monitoring, and operational support processes for data pipelines.
• Participate in technical reviews, defect resolution, testing cycles, and production readiness activities.
• Follow engineering standards, coding practices, version control, and deployment processes.
  
  

Mandatory Skills

AWS Databricks, Apache Spark, PySpark, SQL, Python, Data Engineering, ETL/ELT Development, Data Transformation, Data Validation, Data Profiling, AWS S3, Delta Lake, Data Warehouse Concepts.

Preferred Skills

Unity Catalog, Delta Live Tables (DLT), Databricks Workflows, CDC processing, AWS Glue, Informatica, SSIS, Data Quality Frameworks, CI/CD, Git, Terraform, Power BI data model understanding.