Data Engineer II - Databricks, Python/Java, PySpark

Job responsibilities

• Organizes, updates, and maintains gathered data that will aid in making the data actionable
• Demonstrates basic knowledge of the data system components to determine controls needed to ensure secure data access
• Uses enterprise-authorized AI capabilities within the work environment to accelerate data analysis support and technical documentation (e.g., clarifying requirements and drafting data definitions), validating outputs and handling data according to sensitivity and security requirements.
• Applies reuse-first, AI-assisted approaches to improve data quality checks and model/change validation routines, ensuring results are validated and aligned to resiliency and security expectations.
• Be responsible for making custom configuration changes in one to two tools to generate a product at the business or customer request
• Updates logical or physical data models based on new use cases with minimal supervision
• Adds to team culture of diversity, opportunity, inclusion, and respect

Required qualifications, capabilities, and skills

• Formal training or certification on data engineering concepts and 2+ years applied experience
• Basic knowledge of the data lifecycle and data management functions. Experience across the data lifecycle, building Data frameworks, working with Data lakes. Experience with Batch and Real time Data processing with Spark or Flink.
• Working experience with both relational and NoSQL databases. Experience in ETL data pipelines both batch and real-time data processing, Data warehousing, NoSQL DB. Experience working with Databricks, Python/Java, PySpark etc.
• Working knowledge of AWS Glue and EMR usage for Data processingand in building services using Spring Boot or Flask and deploying them on AWS EKS or Kubernetes.
• Working knowledge of using enterprise-authorized AI capabilities within the work environment to support data engineering workflows with strong validation habits and awareness of data sensitivity.
• Ability to review and validate AI-assisted outputs (e.g., query suggestions or model change summaries) before use, escalating when uncertain and following data handling requirements.
• Significant experience with statistical data analysis and ability to determine appropriate tools to perform analysis
• Basic knowledge of data system components to determine controls needed