Data Engineer

logfix scm solutions pvt ltd

Chennai, India

3-5 Years

Save

Posted 9 hours ago
Be among the first 10 applicants

Early Applicant

Job Description

Key Responsibilities

Data Engineering & Pipeline Development Design, build, and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Microsoft Fabric Data Pipelines.

Develop high-performance ingestion, transformation, and orchestration workflows for enterprise data workloads.

Build scalable data processing solutions using PySpark, Spark SQL, and Fabric Lakehouse.

Develop and maintain data models in Microsoft Fabric Lakehouse / Warehouse environments. Database & Storage Management Work with Azure Dedicated SQL Pool (Synapse/Fabric) for large-scale analytical workloads.

Perform partitioning, indexing, performance tuning, and query optimization.

Manage and optimize Relational Databases (Azure SQL DB, SQL Server, MySQL, PostgreSQL).

Build solutions with NoSQL databases such as MongoDB, Cosmos DB, or Cassandra. Data Quality, Governance & Optimization Implement data quality checks, validation rules, and monitoring for pipeline reliability.

Ensure data governance, security, and compliance across all environments. Optimize compute cost and performance across Fabric, SQL Pools, and storage layers.

Collaboration & Stakeholder Support Work closely with Data Architects, BI Teams, and Business Analysts for data modelling and reporting needs.

Required Skills & Qualifications

3+ years of hands-on experience as a Data Engineer in cloud data platforms.

Strong expertise in: Microsoft Fabric (Lakehouse, Warehouse, Pipelines, One Lake). Azure Data Factory (ADF) pipelines, mapping/ dataflows, triggers. Azure Dedicated SQL Pool (Synapse/Fabric) performance tuning & distributed SQL.

Strong SQL skills for complex query building, optimization, and analytical workloads. Strong coding experience in Python and PySpark for data engineering workflows.

Experience handling large datasets (GB/ TB scale) and distributed processing frameworks. Knowledge of Relational Databases (Azure SQL DB, SQL Server, MySQL, PostgreSQL).

Knowledge of NoSQL Databases (MongoDB, Cosmos DB, Cassandra).

Hands-on experience with Git, CI/CD, DevOps for data pipelines.