Data Engineer / Data Architect

Job Summary

Design and optimize cloud based data platforms that power analytics and decision making for a global property and casualty insurance business using AWS PySpark Python and Databricks within a hybrid work model ensuring resilient data pipelines reliable governance and scalable architectures that enable accurate risk assessment and faster claims processing.

Responsibilities

Design and implement scalable data architectures on AWS that support end to end data flows for property and casualty insurance underwriting pricing claims and policy administration use cases

Build robust PySpark and Python based data pipelines in Databricks that ingest transform and curate structured and semi structured insurance data from multiple internal and external sources

Optimize Databricks workflows and Spark jobs to achieve predictable performance cost efficiency and reliability for large volumes of policy premium exposure and claims data

Develop reusable data models and canonical data sets that standardize property and casualty insurance entities such as policies coverages claims and reinsurance treaties across business lines

Establish and maintain data quality rules data validation checks and reconciliation processes that improve accuracy timeliness and completeness of critical regulatory and financial reporting datasets

Collaborate with actuaries underwriters claims analysts and product teams to translate analytical and reporting requirements into well defined data engineering solutions that create measurable business value

Implement secure data access patterns on AWS using appropriate services and configurations that protect sensitive customer and financial information while enabling governed self service analytics

Create and maintain technical documentation including data flow diagrams data dictionaries and architecture blueprints that provide clear traceability from source systems to curated consumption layers

Automate testing monitoring and alerting for data pipelines using Python and cloud native tools to proactively identify data anomalies and pipeline failures before they affect downstream consumers

Partner with enterprise and solution architects to align data engineering solutions with target cloud architecture standards helping to modernize legacy insurance platforms and decommission redundant assets

Drive continuous improvement by profiling datasets identifying performance bottlenecks and implementing tuning strategies that shorten data delivery cycles for reporting dashboards and advanced analytics

Support business continuity by implementing backup recovery and environment management strategies in AWS and Databricks that reduce operational risk and minimize downtime for critical data services

Mentor peers and junior engineers by sharing best practices in PySpark coding standards data modeling for insurance and cloud based engineering patterns that raise the overall quality of team deliverables

Qualifications

Possess eight to twelve years of experience in data engineering or data architecture roles with a consistent focus on delivering production grade solutions in complex enterprise environments

Demonstrate strong hands on expertise with AWS data and compute services including data storage processing orchestration and security components used to run modern analytical workloads

Exhibit advanced proficiency in PySpark and Python for building scalable transformation logic unit tests and utility frameworks that can be reused across multiple insurance data pipelines

Bring proven experience working with Databricks to develop optimize and operate collaborative notebooks jobs and clusters tailored for large scale property and casualty data processing

Show deep functional understanding of property and casualty insurance concepts including policies coverages rating factors claims lifecycle and regulatory reporting and apply this knowledge to data designs

Apply solid knowledge of dimensional and relational data modeling practices to design curated data marts and semantic layers that serve analytics actuarial studies and operational reporting

Utilize strong skills in SQL data profiling and data quality assessment to diagnose source system issues and drive corrective actions with business and technology stakeholders

Demonstrate effective communication and collaboration abilities to work in a hybrid model with distributed teams conveying complex data topics in clear and concise business language

Display familiarity with modern DevOps and DataOps practices such as version control continuous integration automated testing and environment promotion for data engineering solutions

Show commitment to continual learning by staying updated on evolving AWS Databricks and open source data engineering capabilities and by actively incorporating relevant innovations into project delivery