Azure Databricks

Responsibilities

• Design and optimize scalable data engineering solutions on Azure Databricks that enable advanced analytics and AI workloads for enterprise stakeholders and downstream applications
• Configure and administer Databricks Unity Catalog to establish centralized governance robust access controls and consistent data security policies across all workspaces and environments
• Develop automate and monitor Databricks Workflows that orchestrate complex data pipelines ensuring reliable scheduling error handling and recovery for critical business processes
• Implement and manage Databricks Delta Lake structures to support incremental data ingestion versioning and time travel capabilities that improve data quality and analytical flexibility
• Build fine tuned Databricks SQL queries dashboards and data models that deliver actionable insights to business teams while maintaining efficient performance on large datasets
• Collaborate with data scientists analysts and application teams to translate analytical requirements into well structured data solutions that accelerate experimentation and deployment
• Document data models pipeline logic governance rules and operational procedures in a clear concise manner to promote transparency maintainability and knowledge sharing across teams
• Apply rigorous data quality checks validation frameworks and monitoring practices to ensure that curated datasets remain accurate consistent and fit for AI and reporting use cases
• Partner with cloud infrastructure and security teams to align Databricks configurations networking and identity controls with enterprise standards and regulatory expectations
• Optimize cluster configuration storage layout and resource utilization in Azure Databricks to balance performance reliability and cost efficiency for production and development workloads
• Contribute to continuous improvement initiatives by evaluating new Databricks features and best practices then guiding adoption that strengthens data platform capability and resilience
• Support domain oriented data solutions that can incorporate life and annuities insurance concepts where relevant helping business teams better understand risk policy behavior and customer outcomes
• Champion responsible and ethical use of data by promoting governance practices that protect customer information support regulatory compliance and enhance societal trust in data driven decisions

Qualifications

• Demonstrate strong hands on expertise in Azure Databricks including consistent practical experience with Databricks Unity Catalog Workflows SQL and Delta Lake in complex environments
• Bring solid experience in designing and operating large scale data pipelines with a minimum of nine years in data engineering or AI data specialist roles and exposure to modern cloud data stacks
• Show familiarity with life and annuities insurance data structures processes and metrics as a beneficial domain skill that can deepen the impact of analytics solutions for insurance stakeholders
• Exhibit advanced proficiency in structured query development performance tuning and data modeling practices that support reporting machine learning and self service analytics
• Apply sound knowledge of data governance security and compliance concepts to configure Unity Catalog permissions data classification and audit readiness for enterprise usage
• Demonstrate strong problem solving and debugging skills to investigate pipeline failures data anomalies and performance bottlenecks while driving sustainable corrective actions
• Communicate clearly with technical and nontechnical partners providing explanations of data solutions trade offs and risks that support informed decision making and collaboration
• Manage work effectively in an office based day shift model by coordinating with cross functional teams aligning priorities and ensuring consistent availability for stakeholder interactions