Job Description
Transform and process source data into data products and enable egression to other platforms
Develop efficient Data Lake frameworks using Scala, PySpark, or Python, with robust data quality and security controls.
Create reusable libraries and config driven pipelines for enterprise wide ingestion and integration
Ensure compliance with data governance and security standards using tools like AWS DataZone, Lake Formation, AWS Glue and MWAA
Communicate effectively with internal stakeholders and contribute to team knowledge sharing
Experience with CI/CD tools such as GitHub, GitHub Actions, ArgoCD, and automated deployment pipelines.
Experience with observability tools like Cloudwatch, Prometheus, or Grafana
Exposure to AI/ML technologies, with the ability to integrate or leverage AI models to enhance data processing, automation, or system intelligence
Experience using DevSecOps methodologies and best practice to automate deployments and testing
Demonstrable in Python, SQL, Spark, and Scala for distributed data processing.
Demonstrable experience with AWS Glue, Lambda, Redshift, Athena, Kinesis, and EMR
Exposure to AWS Marketplace, API Gateway, and data monetisation strategies.
Performance tuning of ETL jobs and query optimisation for cost efficiency
Design and develop next generation data platforms and pipeline solutions.
Build and maintain backend services and integrations using Java Spring Boot, ensuring performance, scalability, and security.
Collaborate with engineering peers to co create and contribute to shared practices and frameworks
Apply modern coding standards to deliver high quality, reliable solutions that address core business needs and reduce risk.
Strong experience in Java, Spring Boot, and RESTful API development.
