Job Description: Senior Data Engineer

Experience: 5+ Years

Department: Data Engineering & Analytics

Job Type: Full-Time

Location: Ahmedabad

Job Summary

We are looking for an experienced Senior Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable data pipelines and modern data platforms. The ideal candidate should have strong hands-on expertise in Python, SQL, ETL/ELT, Databricks, Snowflake, Apache Airflow, and cloud-based data engineering solutions.

The candidate will be responsible for building robust data pipelines, implementing lakehouse and data warehouse architectures, orchestrating complex workflows, and optimizing data processing performance. Experience with Microsoft Purview and enterprise data governance frameworks is highly desirable.

The role requires collaboration with data architects, analysts, data scientists, and business stakeholders to deliver reliable, secure, governed, and analytics-ready data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, SQL, PySpark, and modern data engineering technologies.
• Build and optimize data processing solutions using Databricks, Apache Spark, Delta Lake, and lakehouse architecture.
• Design and manage data warehouse solutions using Snowflake, including ingestion, transformation, performance optimization, and cost management.
• Develop, schedule, and orchestrate complex data pipelines and dependencies using Apache Airflow DAGs.
• Configure Airflow operators, sensors, task dependencies, retries, scheduling, logging, and failure notifications.
• Integrate Apache Airflow with Databricks, Snowflake, cloud storage, APIs, and other data services.
• Implement batch and near-real-time data pipelines integrating databases, REST APIs, third-party applications, and cloud platforms.
• Implement incremental data loading, Change Data Capture (CDC), schema evolution, and data partitioning.
• Design dimensional data models, star schemas, slowly changing dimensions (SCD), and analytical data structures.
• Establish data quality checks, validation rules, reconciliation processes, and automated pipeline monitoring.
• Implement data governance, metadata management, data lineage, data cataloging, and data classification using Microsoft Purview and related tools.
• Support data discovery, sensitive data classification, business glossary management, and data ownership practices.
• Implement data access controls, least-privilege principles, and sensitive data protection.
• Optimize SQL queries, Spark jobs, Snowflake workloads, and cloud infrastructure costs.
• Implement CI/CD, automated testing, version control, and deployment workflows for data engineering solutions.
• Troubleshoot production data issues, perform root-cause analysis, and maintain pipeline reliability.
• Conduct code reviews, mentor junior engineers, and contribute to data engineering architecture and best practices.

Required Technical Skills

1. Programming & Data Processing

• Strong proficiency in Python and SQL.
• Hands-on experience with PySpark and Apache Spark.
• Experience developing production-grade ETL/ELT pipelines.
• Knowledge of incremental processing, CDC, data validation, and performance tuning.

2. Databricks

• Experience with Databricks, Apache Spark, PySpark, Spark SQL, and Delta Lake.
• Understanding of Bronze, Silver, and Gold layers in medallion architecture.
• Experience with Delta tables, optimization, partitioning, and incremental processing.
• Familiarity with Databricks Workflows and Unity Catalog.

3. Snowflake

• Hands-on experience with Snowflake data warehousing and Snowflake SQL.
• Understanding of virtual warehouses, databases, schemas, and query optimization.
• Experience with data loading, transformation, and workload cost optimization.
• Knowledge of Snowpipe, Streams, Tasks, and CDC concepts.
• Familiarity with Snowpark, role-based access control, and data security.

4. Apache Airflow & Workflow Orchestration

• Hands-on experience developing and maintaining Apache Airflow DAGs for production data pipelines.
• Strong understanding of DAG design, task dependencies, scheduling, retries, sensors, operators, and error handling.
• Experience integrating Airflow with Databricks, Snowflake, cloud storage, databases, and APIs.
• Knowledge of backfills, catchup behavior, idempotency, and dependency management.
• Experience monitoring DAG execution, troubleshooting failed tasks, and configuring alerts.
• Understanding of Airflow connections, variables, secrets management, and environment-specific configurations.
• Familiarity with Airflow deployment, CI/CD, and production operations is an advantage.

5. Cloud Technologies

Hands-on experience with at least one major cloud platform:

• AWS: S3, AWS Glue, Lambda, Redshift, EMR.
• Microsoft Azure: Azure Data Factory, Azure Data Lake Storage, Azure Databricks, Synapse Analytics.
• Google Cloud: BigQuery, Cloud Storage, Dataflow.

Experience integrating cloud services with Databricks, Snowflake, and Airflow is preferred.

6. Data Governance & Microsoft Purview

• Experience with Microsoft Purview for data cataloging, discovery, and governance.
• Understanding of metadata management, data lineage, data classification, and business glossaries.
• Knowledge of data ownership, stewardship, and governance policies.
• Familiarity with sensitive data identification, PII protection, access governance, and data lifecycle management.
• Understanding of governance integration with Databricks, Snowflake, and cloud data services.
• Familiarity with Unity Catalog and Snowflake governance capabilities is an advantage.

7. Databases & Data Modeling

• Experience with PostgreSQL, SQL Server, MySQL, or other relational databases.
• Strong understanding of dimensional modeling, star schemas, and slowly changing dimensions.
• Familiarity with Parquet, Avro, JSON, and other common data formats.
• Knowledge of data warehouse, data lake, and lakehouse design principles.

8. DevOps & CI/CD

• Experience with Git and version control.
• Familiarity with CI/CD pipelines, automated testing, and deployment automation.
• Knowledge of Docker, logging, monitoring, alerting, and production support.
• Experience managing code promotion across development, testing, and production environments.

Preferred Qualifications

• Experience working across Databricks and Snowflake within enterprise data platforms.
• Experience orchestrating multi-stage pipelines using Apache Airflow.
• Microsoft Purview implementation or administration experience.
• Knowledge of dbt, Fivetran, Airbyte, or similar data integration tools.
• Exposure to Power BI, Tableau, or other BI platforms.
• Understanding of GDPR, CCPA, and other applicable data privacy regulations.
• Experience preparing governed datasets for AI, machine learning, and LLM-based applications.
• Relevant certifications in Databricks, Snowflake, cloud data engineering, or data governance are an advantage.

Experience & Qualifications

• Minimum 5+ years of professional experience in data engineering.
• Strong hands-on experience building and supporting production data pipelines.
• Practical experience with Databricks, Snowflake, and Apache Airflow.
• Experience with cloud data platforms, orchestration, and enterprise data architectures.
• Strong problem-solving, communication, documentation, and stakeholder management skills.
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or equivalent practical experience.

Key Competencies

• Strong ownership of data engineering deliverables.
• Ability to design scalable, secure, and maintainable data solutions.
• Strong understanding of data governance and data quality principles.
• Excellent troubleshooting and root-cause analysis skills.
• Ability to collaborate with cross-functional teams.
• Commitment to engineering quality, documentation, security, and continuous improvement.

What We Offer

• Opportunities to work with modern data platforms, including Databricks and Snowflake.
• Exposure to Apache Airflow orchestration and enterprise data governance using Microsoft Purview.
• Opportunities to contribute to data platform architecture and technical decision-making.
• Collaborative engineering environment and professional development opportunities.