Job Description
Position Overview
We are seeking an experienced Data Engineer to join our innovative data team data team and help build scalable data infrastructure, software consultancy, and development services that powers business intelligence, analytics, and machine learning initiatives. The ideal candidate will design, develop, and maintain robust high-performance data pipelines and solutions while ensuring data quality, reliability, and accessibility across the organization working with cutting-edge technologies like Python, Microsoft Fabric, Snowflake, Dataiku, SQL Server, Oracle, PostgreSQL, etc.
Experience: 5 to 7 years
Location: Bengaluru
Employment Type: Full-time / Permanent
Key Responsibilities
Data Pipeline Development
Design, build, and maintain highly efficient and scalable ETL/ELT pipelines to process large volumes of structured and unstructured data
Implement real-time and batch data processing solutions using modern data engineering tools (e.g., Fabric, Spark, Kafka, Airflow, )
Optimize data workflows for performance, reliability, and cost-effectiveness
Monitor pipeline health and implement automated alerting and recovery mechanisms Data Architecture & Infrastructure
Collaborate with architects to design and implement data warehouse and Data Warehouse, Data Lake, and Lakehouse solutions (e.g., Fabric, Snowflake).
Build and maintain cloud-based data infrastructure (Azure or AWS, GCP)
Implement data governance frameworks and ensure compliance with data privacy regulations
Design schema and data models for optimal query performance and scalability Data Quality & Integration
Develop data validation frameworks and implement quality checks throughout pipelines
Integrate data from multiple sources including APIs, databases, streaming platforms, and third-party services
Create and maintain data documentation, lineage tracking, and metadata management
Troubleshoot data quality issues and implement corrective measures Collaboration & Support
Partner with data scientists, analysts, and business stakeholders to understand data requirements
Support analytics teams with data access, modeling, and optimization
Participate in code reviews and maintain high standards for data engineering practices
Mentor junior team members and contribute to team knowledge sharing
Skills
Knowledge of machine learning workflows and MLOps practices
Familiarity with data visualization tools (Tableau, Looker, Power BI)
Experience with stream processing and real-time analytics
Experience with data governance and compliance frameworks (GDPR, CCPA)
Contributions to open-source data engineering projects
Relevant Cloud certifications (e.g., Microsoft Certified: Azure Data Engineer Associate, AWS Certified Data Engineer, Google Cloud Professional Data Engineer).
Specific experience or certifications in Microsoft Fabric, or Dataiku, Snowflake.
Required Qualifications
Technical Skills
Programming Languages: Proficiency in Python
Cloud Platforms: Hands-on experience with Azure (Fabric, Synapse, Data Factory, Event Hubs)
3+ years of experience in data engineering or related roles with 5+ years of Software Engineering Experience.
Databases: Strong SQL skills and experience with both relational (Microsoft SQL Server, PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases
Version Control: Proficiency with Git and collaborative development workflows
Proven track record of building production-grade data pipelines handling large-scale data or solutions.
Desired experience with containerization (Docker) and orchestration (Kubernetes) technologies.
