Job Title: Data Engineer
Location: Coimbatore
Experience: 35 Years
Mandatory Certification: Databricks Data Engineer Professional
About the Role
We are looking for a skilled Data Engineer with hands-on experience in building scalable data pipelines and working extensively with Databricks. The ideal candidate should have strong data engineering fundamentals, expertise in cloud platforms, and the ability to collaborate with cross-functional teams to deliver reliable data solutions.
Key Responsibilities
- Design, develop, and maintain ETL/ELT pipelines using Databricks and related technologies.
- Build and optimize data architectures, data models, and data workflows.
- Work with large-scale datasets to ensure efficient data ingestion, transformation, and processing.
- Implement Delta Lake best practices, including schema evolution, time travel, and ACID transactions.
- Collaborate with data scientists, analysts, and business teams to understand data requirements.
- Monitor and improve data pipeline performance and reliability.
- Ensure data security, compliance, and governance practices are followed.
- Troubleshoot and resolve issues related to data processing, orchestration, and pipeline failures.
- Automate repetitive processes to improve efficiency and scalability.
Required Skills & Qualifications
- 35 years of experience as a Data Engineer or in a similar role.
- Databricks Data Engineer Professional certification (Mandatory).
- Strong expertise in Databricks, Apache Spark (PySpark/Scala), and Delta Lake.
- Hands-on experience with cloud platforms (AWS, Azure, or GCP Azure preferred if using Azure Databricks).
- Proficiency in SQL and experience working with relational and non-relational databases.
- Strong understanding of data warehousing concepts and performance optimization.
- Solid experience with ETL/ELT pipelines, data modeling, and data processing frameworks.
- Experience in Power BI for building dashboards, data visualizations, and analytics reports.
- Ability to integrate Power BI with Databricks or other data sources.
- Experience with Git, CI/CD pipelines, and data orchestration tools (Airflow, Azure Data Factory, etc.).
- Strong problem-solving and analytical skills.
Preferred Qualifications
- Experience with Azure Data Factory, Azure Synapse, AWS Glue, or similar data integration tools.
- Knowledge of MLflow, Unity Catalog, or modern data governance frameworks.
- Familiarity with containerization tools such as Docker and Kubernetes.
Soft Skills
- Strong communication and collaboration skills.
- Ability to work in a fast-paced and dynamic environment.
- Attention to detail with a strong focus on data accuracy and quality.
- Team player with a proactive and solution-oriented mindset.
