Job Purpose
Seeking a highly skilled Senior Data Engineer to design, build, and maintain scalable data platforms, ETL processes, and AI-enabled data solutions. The ideal candidate will have strong expertise in modern data engineering practices, cloud-native technologies, DevOps automation, and generative AI integrations. This role requires hands-on experience developing robust data pipelines, managing containerized applications, and supporting enterprise-grade data infrastructure.
Key Responsibilities
- Design, develop, and maintain scalable ETL workflows and data pipelines that support business-critical applications and analytics.
- Build and manage data orchestration workflows using Apache Airflow.
- Develop high-quality, production-ready solutions using Python.
- Implement and maintain CI/CD pipelines leveraging GitHub Actions.
- Manage source control, branching strategies, and code reviews through GitHub repositories.
- Deploy, monitor, and optimize applications running on Azure Kubernetes Service (AKS).
- Develop and maintain containerized solutions using Docker.
- Integrate and support Generative AI solutions utilizing OpenAI and Anthropic Claude models.
- Collaborate with data scientists, ML engineers, and business stakeholders to deliver reliable data products.
- Ensure data quality, monitoring, observability, security, and governance standards are met.
- Troubleshoot production issues and drive continuous improvement across data engineering platforms.
- Document architecture, processes, and best practices for enterprise data solutions.
Key competencies
Essential Skills
- Bachelor's or master's degree in computer science, Engineering, Information Technology, or a related field.
- 7–10 years of experience in data engineering, software engineering, or related technical disciplines.
- Strong hands-on experience with:
- Python
- GitHub and Git-based development workflows
- GitHub Actions CI/CD
- Azure Kubernetes Service (AKS)
- Apache Airflow
- Docker
- Building and maintaining enterprise-scale ETL processes and data pipelines
- Experience designing and operating cloud-native, distributed data platforms.
- Strong understanding of software engineering best practices, testing frameworks, and deployment automation.
- Experience working with OpenAI and Claude (Anthropic) models in production environments.
- Experience working with REST APIs and external data integrations.
- Excellent problem-solving, debugging, and performance optimization skills.
- Strong communication and stakeholder management capabilities.
Nice to Have
- Experience with S&P Xpressfeed data platforms.
- Experience with Snowflake data warehousing and analytics solutions.
- Familiarity with data governance, data quality frameworks, and monitoring tools.
- Exposure to machine learning and AI/ML operations (MLOps) environments.
- Experience working in Agile/Scrum development teams
Skills:
- Data Engineering: ETL, Data Pipelines, Data Integration, Data Orchestration, Data Quality
- Cloud & DevOps: Azure Kubernetes Service (AKS), Docker, GitHub, GitHub Actions, CI/CD
- Programming: Python
- Workflow Orchestration: Apache Airflow
- AI & Generative AI: OpenAI, Claude, LLM Integrations
- Preferred Data Platforms: Snowflake, S&P Xpressfeed
