Data Engineer

Role Summary

The Data Engineer will support data ingestion, transformation, profiling, documentation, and Microsoft Fabric environment operations for a data and actuarial discovery sandbox. This role will work closely with senior data engineering, actuarial, product, IT, and security teams to help build reliable data pipelines, maintain Lakehouse tables, validate data quality, and support controlled access to curated datasets.

Key Responsibilities

• Support ingestion of sample source data from cloud storage into Microsoft Fabric using ADF pipelines, OneLake landing folders, and Lakehouse tables.
• Assist with the creation and maintenance of bronze, silver, and gold tables based on the agreed discovery data model.
• Build and run data profiling scripts and notebooks to identify file structures, schemas, row counts, null rates, duplicate records, timestamp ranges, and field-level quality issues.
• Help standardize common source data fields such as entity identifiers, transaction or event IDs, timestamps, event types, measurement units, and geography fields.
• Support data transformation work required to prepare daily summary, event-summary, exposure, behavioral, and contextual feature datasets.
• Document source tables, transformations, assumptions, data quality findings, known limitations, and operational procedures for actuarial users.
• Support ad hoc data requests from the actuarial team during discovery and help refine datasets based on feedback.
• Assist with Microsoft Fabric workspace setup, Lakehouse configuration, pipeline permissions, access controls, and environment support.
• Help configure and validate secure connections between Microsoft Fabric and external cloud platforms, including authentication handling, service account/key usage, and privileged access management considerations.
• Monitor pipeline runs, ingestion failures, storage usage, compute activity, and basic cost/performance indicators during the discovery build.
• Provide crossover support to the product build effort, including Fabric configuration, Lakehouse setup, Power BI access support, and troubleshooting with IT/security teams.

Required Skills and Experience

• Foundational experience in data engineering, data pipelines, ETL/ELT, data transformation, or cloud data platforms.
• Working knowledge of SQL and Python exposure to Spark, notebooks, or distributed data processing is preferred.
• Basic understanding of data modeling, data quality checks, schema validation, and table maintenance.
• Familiarity with Microsoft Azure, Microsoft Fabric, ADF pipelines, OneLake, Lakehouse tables, or similar cloud data services.
• Ability to analyze source data, identify quality issues, and document findings clearly for technical and non-technical audiences.
• Strong attention to detail, willingness to learn, and ability to work collaboratively with senior engineers and actuarial stakeholders.

Preferred Qualifications

• Exposure to insurance, actuarial, mobility, operational, or risk analytics data.
• Experience working with GCP, Azure, Power BI, or cloud-to-cloud data ingestion patterns.
• Understanding of access controls, workspace permissions, data retention, and secure data handling practices.
• Experience using Git, Azure DevOps, or similar tools for version control, task tracking, and documentation.