Lead Data Engineer

Labcorp Drug Development

Bengaluru, India

5-7 Years

This job is no longer accepting applications

Posted 6 months ago

Job Description

Job Description:

We are looking for aLead Data Engineerwith strong Databricks expertise who can also design and buildaccelerators for automation. The role requires hands-on development inDatabricks (PySpark, Spark SQL, Delta Lake,Autoloader, DLT etc)along with experience in creating frameworks fortest data generation, report validation, and backend validation. You will be leading technical work, mentoring team members, and building reusable solutions that speed up project delivery.

Key Responsibilities:

.Design, develop, and optimizeETL/ELT pipelineson Databricks usingPySpark, Spark SQL,Delta Lake, DLT live.

.Buildautomation acceleratorssuch as:

Synthetic test data generatorsfor large-scale data sets.

Automated report validation toolsto compare dashboards with backend data.

Backend validation frameworksfor reconciliation and data quality checks.

.Implementdata validation frameworks(Great Expectations, Deequ, or custom solutions).

.Ensuredata quality, lineage, and governanceusingUnity Catalog.

.Collaborate with QA, BI, and business teams to integrate accelerators into workflows.

.Drive best practices in coding, performance tuning, and cost optimization on Databricks.

.Lead and mentor data engineers, review code, and set technical standards.

Must-Have Skills

.5-7years of experience in data engineering, with at least 3-5 years hands-on experience in databricks

.Databricks:PySpark, Spark SQL, Delta Lake, Delta Live Tables (DLT).

.Automation & Accelerators:Experience creating test data generators, validation frameworks, report reconciliation tools.

.Programming language:Strong inPython,PySparkandSQL.

.Validation Tools:open source tools likeGreat Expectations, Deequ, or equivalent custom frameworks.

.Data Pipelines:Batch and streaming pipeline design.

.Data Governance:Unity Catalog for access, lineage, and compliance.

.DevOps/CI-CD:Git, Azure DevOps, Jenkins, or GitHub Actions for deployment automation.

.Cloud & Storage:Azure (preferred) / AWS / GCP with hands-on inADLS/S3/GCS.

Good-to-Have Skills

.Streaming:Kafka, EventHub, or Kinesis.

.Infra-as-Code:Terraform for Databricks and cloud provisioning.

.Reporting Tools:Power BI, Tableau, or Looker (for validation accelerators).

.Testing:Exposure to pytest or unittest for automation.

Qualifications:

.Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Labcorp is proud to be an Equal Opportunity Employer:

Labcorp strives for inclusion and belonging in the workforce and does not tolerate harassment or discrimination of any kind. We make employment decisions based on the needs of our business and the qualifications and merit of the individual. Qualified applicants will receive consideration for employment without regard to race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), family or parental status, marital, civil union or domestic partnership status, sexual orientation, gender identity, gender expression, personal appearance, age, veteran status, disability, genetic information, or any other legally protected characteristic. Additionally, all qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable law.

We encourage all to apply

If you are an individual with a disability who needs assistance using our online tools to search and apply for jobs, or needs an accommodation, please visit ouror contact us at Formore information about how we collect and store your personal data, please see our.