Lead Data Engineer

Job Title - Senior Data Engineer

Overview of Providence:

At Providence, we use our voice to advocate for vulnerable populations and needed reforms in health care. We pursue innovative ways to transform health care by keeping people healthy, and making our services more convenient, accessible and

affordable for all. In an increasingly uncertain world, we are committed to high-quality, compassionate health care for everyone-regardless of coverage or ability to pay. We help people and communities benefit from the best health care model for the future-today.

Together, our 119,000-plus caregivers/employees serve in 51 hospitals, more than 1000 clinics and a comprehensive range of health and social services across Alaska, California, Montana, New Mexico, Oregon, Texas and Washington in United States.

Providence Information Service (commonly known as Information technology) is aimed to digitally enable our vision of Health for a better world. We make use of multiple technologies spread across Microsoft, Oracle, JAVA, Python and many new open sources.

Providence Global Center recently launched in Hyderabad, India as Global Capability Center for Providence looking to leverage the India talent to help meet our global vision and scale our Information Services and products to the world of Cloud.

About Healthcare Intelligence

Healthcare Intelligence is the pillar that focuses on creating intelligent data products within Providence and provides unique opportunities in Data Engineering, Operations, Data Science, BI Reporting and Data Analytics on cloud stack. We are a group of data professionals who work towards enabling decisions that improve patient and caregiver experience.

About the role

A Data Engineer is responsible for designing and modelling Data lake solutions/data ingestion pipelines by collaborating with our customers and understanding requirements on digital transformation. Able to navigate through ambiguous situations, to deal with aggressively changing environment. The drive to collaborate, gather feedback, solve problems, and tackle challenges through test and learn is highly valuable in this position. Should work closely with Software Engineers, Data Scientists, Azure & Network Administrator teams to build a scalable & compliant data system andproduction-ready ML pipelines. Work with different stakeholders as SME for Data Engineering, providing guidance ondata readiness for AI initiativesand architectural support forAzure-native AI services.

Responsibilities

• Be the owner for data engineering solutions within Healthcare Intelligence. Ensure the continuity of data processes and the associated batch jobs.
• Partner with leadership, engineers, program managers, data analysts and scientists to understand data needs.
• Identify, design and implement/coordinate implementation of scalable processes and infrastructure to have good governance of automated data processes.
• Manage the end-to-end data solutions of our customers: from raw data analysis to data flow and predictive framework configurations.
• Manage performance, capacity, availability, security and compliance of data platform and data solutions.
• Should be able to work on a problem independently and prepare client ready deliverable with minimal supervision. Elicit, analyze, and validate customer data from ingestion to production
• Monitor all data update processes and outputs to ensure predictive quality
• Communicate with customers to discuss any issues with received data and help them identify and fix data issues
• Solve day-to-day Data problems and customer challenges
• Own the automation, deployment and operation of data pipelines on MS Azure
• Build tools and mechanism to monitor and optimize different parts of the systems
• Build custom integrations between cloud-based systems using APIs
• Develop automated data quality profiling and cleansing routines to ensure AI-ready datasets that meet the strict accuracy requirements of clinical intelligence
• Proactively evaluate and integrate emerging Azure technologies and industry trends to modernize the existing data stack and reduce technical debt.
• Monitor and manage Azure consumption and compute costs, implementing strategies to optimize resource utilization without compromising performance.
• Build and oversee comprehensive telemetry, alerting, and monitoring dashboards to proactively identify data drifts or pipeline latencies before they impact downstream customers
• Lead technical design reviews and provide mentorship to junior engineers, fostering a culture of continuous learning and operational excellence.

Skills and qualifications

• Required: Working experience on Azure Data technologies like Datalake ADLSGen2, Azure DataBricks, Azure Data factory, Azure SQL, Azure Synapse etc.
• Required: Experience in creating data orchestration using ADF and optimizing them through regular monitoring.
• Exposure to Snowflake Cloud data Platform
• Familiarity with Linux/Unix scripting, Python, SQL Queries and Database concepts required. Exposure to cloud onboard from legacy data sets.
• Experience working with large datasets and building data processing platforms
• Fluent and fast with T-SQL, query analysis and optimization. Deep skills in modeling transactional databases and data warehouses.
• Exposure and understanding of Agile process and DevOps
• Good to have: Experience in batch scheduling & rationalization through Control M
• Excellent interpersonal, verbal and written communication skills
• Excellent analytical and critical thinking capabilities and problem solving skills