Senior Data Engineer

Data Engineer – Databricks

Experience: 7+ Years

Work Mode: Remote

Shift Timing: 12:30 PM to 9:30 PM IST

• Primary Skills: Databricks, Pyspark, Python, SQL, Microsoft Fabric, Azure Data Factory, DataOps practices, and performance optimization

Job Summary:

We are looking for experienced Data Engineers with strong hands-on expertise in Databricks, PySpark, Python, SQL, and big data processing. The ideal candidate will have proven experience designing and developing scalable data ingestion pipelines, analytical data models, and high-performance data solutions.

Experience with Microsoft Fabric, Azure Data Factory, DataOps practices, and performance optimization will be an added advantage.

Key Responsibilities:

• Design, develop, and maintain scalable data engineering pipelines using Databricks and PySpark.
• Develop robust data ingestion and transformation pipelines for batch and large-scale data processing.
• Work with Databricks for data processing, transformation, orchestration, and analytics.
• Develop efficient and optimized PySpark and Python code for big data workloads.
• Write complex SQL queries for data transformation, validation, analysis, and troubleshooting.
• Design and implement analytical data models to support reporting and business analytics.
• Build and maintain data pipelines using Azure Data Factory (ADF).
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Apply DataOps practices to improve data pipeline reliability, automation, deployment, and operational efficiency.
• Identify and resolve performance bottlenecks across data pipelines, queries, Spark jobs, and Databricks workloads.
• Collaborate with data architects, analysts, application teams, and other stakeholders to deliver scalable data solutions.
• Follow best practices for data security, governance, testing, version control, and CI/CD

Required Skills:

• 7+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Strong expertise in PySpark and big data processing.
• Strong programming skills in Python.
• Advanced SQL skills.
• Experience developing data ingestion and ETL/ELT pipelines.
• Experience in analytical data modeling and data transformation.
• Good understanding of distributed data processing and scalable data architectures.
• Strong troubleshooting and performance optimization skills.

About Atain:

(http://www.atain.com)

Atain is an enterprise orchestration partner that aligns people, processes, and platforms with AI to redefine work and deliver assured outcomes. It is enabled through the SMART Framework driving superior metrics with AI-redefined work, the TechBud platform, a GenAI foundation built on zero-pilot architecture and ISO 42001 standards, a robust partner ecosystem, and deep domain expertise. Atain's Unified Hub continuously senses, predicts, and re-orchestrates work across people, processes, and platforms to deliver assured outcomes. The centers of skill and scale combine deep expertise with AI to deliver assured outcomes. For more information, visit www.atain.com

Atain is ISO 27001:2013, CMMI SVC Level 5 and ISAE-3402 compliant for IT, and COPC Certified v6.0, ISO 27001:2013 and PCI DSS 3.2 certified for BPO processes. The organization follows Six Sigma rigor for process improvements.

It is our policy to provide equal employment opportunities to all individuals based on job-related qualifications and ability to perform a job, without regard to age, gender, gender identity, sexual orientation, race, color, religion, creed, national origin, disability, genetic information, veteran status, citizenship or marital status, and to maintain a non-discriminatory environment free from intimidation, harassment or bias based upon these grounds.