We are seeking an experienced Senior Data Engineer to lead the design, development, and optimization of our modern data platform. This is a hands-on technical leadership role responsible for building scalable data pipelines, defining data models, enforcing engineering best practices, and ensuring delivery of high-quality, governed data products.
The ideal candidate will have deep expertise in Databricks, Lakehouse architecture, ETL/ELT development, and data modeling, along with the ability to mentor engineers and drive technical excellence across the data engineering function.
Key ResponsibilitiesData Platform Design & Development
- Design, develop, and maintain scalable data ingestion, transformation, and serving pipelines using Databricks.
- Build and manage data pipelines across Bronze, Silver, and Gold layers following Medallion Architecture principles.
- Develop production-grade solutions leveraging Databricks Autoloader, Delta Live Tables (DLT), Delta Lake, and Unity Catalog.
- Design canonical data models aligned with business domains, including conformed dimensions, business entities, and shared reference data.
- Implement Slowly Changing Dimensions (SCD Type 2) and other historical data management patterns.
- Ensure platform scalability, reliability, maintainability, and cost optimization.
Data Quality & Governance
- Implement robust data quality checks, validation frameworks, and automated testing strategies.
- Establish monitoring and alerting mechanisms to ensure data freshness and SLA compliance.
- Manage data governance, metadata, lineage, and access controls through Unity Catalog.
- Collaborate with governance and security teams to ensure compliance with organizational and regulatory standards.
Performance Optimization
- Optimize data pipelines for performance, latency, scalability, and cost efficiency.
- Drive best practices around partitioning, file management, compute utilization, and query optimization.
- Ensure resilient pipeline design with proper retry mechanisms, idempotency, and failure recovery strategies.
Technical Leadership
- Define and enforce engineering standards, coding practices, naming conventions, and architectural guidelines.
- Lead code reviews and ensure adherence to quality and documentation standards.
- Mentor and support Data Engineers through technical guidance and knowledge sharing.
- Contribute to architecture decisions and technology roadmap planning.
Cross-Functional Collaboration
- Partner with Data Architects, Data Modelers, Product Owners, and Application Teams to translate business requirements into scalable technical solutions.
- Collaborate with integration teams to deliver data products suitable for API consumption and analytics platforms.
- Support end-to-end delivery of enterprise-grade data solutions.
Required Qualifications- 6–10+ years of experience in Data Engineering, including technical leadership responsibilities.
- Strong expertise in SQL and large-scale ETL/ELT pipeline development.
- Hands-on experience with Databricks, including:
- Autoloader
- Delta Live Tables (DLT)
- Delta Lake
- Unity Catalog
- Strong understanding of Lakehouse and Medallion Architecture.
- Expertise in data modeling, dimensional modeling, and SCD Type 2 implementation.
- Experience building scalable data platforms and governed data products.
- Strong understanding of data governance, metadata management, and access controls.
- Proven experience conducting code reviews and establishing engineering best practices.
- Excellent analytical, problem-solving, and communication skills.
Preferred Qualifications
- Strong experience with PySpark and Apache Spark.
- Experience working with cloud platforms such as AWS, Azure, or GCP.
- Knowledge of CI/CD, Infrastructure as Code, and DevOps practices for data platforms.
- Experience with API-driven data consumption patterns.
- Familiarity with data observability and monitoring tools.
What We Offer
- Opportunity to work on modern cloud-native data platforms.
- Exposure to large-scale enterprise data engineering initiatives.
- Collaborative and innovation-driven environment.
- Career growth and leadership opportunities.
- Competitive compensation and benefits package.
Skills
Databricks | Delta Lake | Delta Live Tables | Unity Catalog | Autoloader | SQL | PySpark | Apache Spark | Data Engineering | ETL | ELT | Data Modeling | SCD Type 2 | Lakehouse Architecture | Medallion Architecture | Data Governance | Performance Optimization | Cloud Data Platforms
Skills: medallion,data bricks,pipelines
