Data Engineer
Data Engineer
KPI PartnersThis job is no longer accepting applications
Job Description
We are looking for an experienced Senior Microsoft Fabric Data Engineer to join our growing Data Engineering team. The ideal candidate will have strong expertise in Microsoft Fabric, data pipeline development, and Oracle EBS integration. You will play a key role in designing, developing, and orchestrating scalable data pipelines to support enterprise analytics and reporting.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Microsoft Fabric.
- Build and manage Data Factory pipelines within Microsoft Fabric.
- Schedule and orchestrate Notebooks, Copy Activities, and Dataflow Gen2 executions.
- Create and manage dependency chains between pipeline activities.
- Implement retry policies, timeout handling, and error recovery mechanisms.
- Configure dynamic pipelines using parameters, variables, and expressions.
- Set up automated notifications and alerts for pipeline failures.
- Design and orchestrate Bronze → Silver → Gold data processing workflows following the Medallion Architecture.
- Integrate and transform data from Oracle EBS and other enterprise data sources.
- Collaborate with business stakeholders, data analysts, and engineering teams to deliver reliable and scalable data solutions.
- Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
Mandatory Skills
- Strong hands-on experience with Microsoft Fabric
- Experience building Data Factory Pipelines in Microsoft Fabric
- Hands-on experience with Notebook orchestration
- Experience with Dataflow Gen2
- Strong understanding of Copy Activity and data movement
- Experience scheduling and orchestrating end-to-end data pipelines
- Knowledge of Pipeline Parameters, Variables, and Dynamic Expressions
- Experience implementing Retry Policies, Timeout Handling, and Failure Notifications
- Hands-on experience designing Bronze → Silver → Gold (Medallion Architecture) data pipelines
- Experience integrating data from Oracle EBS
- Strong SQL skills
- Excellent problem-solving and troubleshooting abilities
Good to Have
- Experience with Azure Data Services
- Knowledge of PySpark or Spark
- Experience with CI/CD for data engineering
- Familiarity with Power BI
- Experience with Git or Azure DevOps
More Info
Key Skills
Retry Policies
Dynamic Expressions
Bronze ? Silver ? Gold Medallion Architecture
Copy Activity
Timeout Handling
Notebook orchestration
Pipeline Parameters
Dataflow Gen2
Failure Notifications
Microsoft Fabric
Variables
About Company
Skills:
Java, Hadoop, Cassandra, Scala, Dynamodb, Spark, Kafka, Data Governance, Data Modeling, Sql, ELT, Devops, Hive, Agile, MongoDB, Hbase, Etl, Data Lakes, Accumulo, Data quality practices, Map-Reduce, Cloud-based technologies
Skills:
Pyspark, Sql, Etl, ELT, Data Modeling, Python, Data Warehousing, Git, Orchestration tools, Data lakes
Skills:
snowflake , Sql, Pl Sql, Ibm Datastage, Unix Shell Scripting, Apis, Data quality validations, Reporting datasets, Airflow, Data transformations, data models, Data pipelines
Skills:
snowflake , T-sql, Snowpipe, Plsql, Data Warehousing, Advanced Sql, Python, DataStream ETL Fundamentals, SnowSQL, Snowflake Cortex AI Functions, Stored Procedures, Time Travel and Fail Safe
Skills:
Java, Pyspark, Scala, Apache Spark, Data Modeling, Data Warehousing, ELT, Gcp, Data Integration, Databricks, Azure, Data Warehouse, Python, AWS, Etl, Data quality and performance optimization, Strong SQL and database knowledge