Data Engineer

Primary skills: Python, SQL, data lakes, azure

Experience: 4+ years

Immediate Joiner

Key Responsibilities

Pipeline Development & Automation

· Design, build, and maintain CI/CD pipelines to automate deployment of DQ rules and data services across environments.

· Optimize data pipelines and jobs for efficiency, scalability, and enterprise-grade reliability within Azure and Databricks.

· Implement best practices in version control, testing, and automated deployments.

Data Quality & Rule Development

· Develop, optimize, and maintain DQ rules in PySpark/Python for self-serve capabilities.

· Design and implement profiling frameworks for rule generation and automated remediation.

· Ensure DQ frameworks align with enterprise standards, governance, and audit requirements.

UI & Self-Serve Integration

· Collaborate with front-end teams (Node.js/Angular) to enable rule configuration, validation, and monitoring via a low-code UI.

· Develop APIs and services to expose DQ results and outputs for dashboards and self-service tools.

Collaboration & Governance

· Partner with Data Stewards, GPOs, and business owners to translate requirements into technical solutions.

· Contribute to BRDs, SRDs, and design reviews for DQ rules and pipelines.

· Provide inputs on governance, compliance, and change management processes.

· Support periodic reviews and continuous improvements of pipelines, rules, and dashboards.

Required Skills & Qualifications

• Strong programming skills in PySpark, Python, and SQL.
• Hands-on experience with Databricks (clusters, notebooks, Delta Lake).
• Experience designing and implementing CI/CD pipelines (Azure DevOps / GitHub Actions / Jenkins).
• Familiarity with Node.js & Angular for UI integration.
• Solid understanding of data governance, DQ frameworks, and MDM concepts.
• Strong analytical, problem-solving, and stakeholder engagement skills.
• Cloud expertise, preferably Azure Data Lake, Data Factory, and Synapse.

Education & Experience

• Bachelor's or Master's degree in Computer Science, Information Systems, or related field.
• 4–8 years of relevant experience in data engineering, CI/CD, and DQ rule development.
• Proven expertise in Databricks, Azure data services, and low-code self-serve frameworks.