Senior Data Engineer

Role Overview

We are looking for a highly skilled Senior Platform Engineer of 7+years of experience to design and implement a next-generation observability and reliability platform for distributed data pipelines.

This platform will automatically discover lineage, collect metrics, and aggregate logs across the technology stack, enabling proactive monitoring and rapid troubleshooting of failures. You will work closely with reliability and data engineering teams to build intelligent systems that enhance operational efficiency and resilience.

Key Responsibilities

· Design and develop end-to-end observability solutions for distributed data pipelines

· Build systems that automatically capture lineage, metrics, and logs across heterogeneous data platforms

· Develop MCP proxies, orchestrators, and scalable data pipelines using Python

· Implement alerting and monitoring frameworks to detect and resolve failures proactively

· Integrate with modern observability stacks including metrics, logs, and tracing systems

· Build and optimize REST/GraphQL APIs with async patterns and robust session management

· Collaborate with reliability engineers to enable faster root cause analysis and troubleshooting

· Develop intuitive UI dashboards (React or equivalent) for monitoring and insights

· Work with cloud-native architectures, primarily on AWS

Required Skills & Experience

· Strong experience in Python & PySpark for data engineering (Must)

· Strong experience with Databricks platform for data processing (Must)

· Strong expertise in Python for building scalable backend systems, orchestrators, and pipelines

· Experience with REST and/or GraphQL APIs, async programming, and session handling

· Hands-on experience with cloud platforms (AWS preferred)

· Deep understanding of distributed systems, data pipelines, and observability concepts

· Experience with data quality, anomaly detection, or validation systems

· Hands on experience in:

o LLM orchestration frameworks

o RAG (Retrieval-Augmented Generation)

o MCP (Model Context Protocol)

o AWS Bedrock or equivalent AI platforms

o Experience with AWS Nova Pro

· Frontend experience with React JS or similar frameworks

Good to Have

· Experience building AI-powered observability or AIOps platforms

· Exposure to data lineage tools and metadata management systems

· Familiarity with Kubernetes and containerized deployments

· Knowledge of CI/CD pipelines and DevOps practices

What You'll Build

· A unified platform that provides:

o Automatic data lineage discovery

o Centralized logging, metrics, and tracing

o Intelligent alerting for failures

o AI-assisted troubleshooting workflows

Why Join Us

· Opportunity to build a cutting-edge observability platform from scratch

· Work at the intersection of Data Engineering, AI/ML, and Cloud

· High ownership and impact in a fast-moving, innovation-driven environment