Role Summary
We are seeking an experienced Observability Engineer with strong expertise in Splunk, Datadog, Grafana, and Prometheus to design, implement, and optimize enterprise-wide monitoring solutions.
The ideal candidate will be responsible for building end-to-end observability capabilities across infrastructure, applications, databases, and cloud environments. This role requires hands-on experience with log ingestion, APM instrumentation, synthetic monitoring, anomaly detection, custom metrics, dashboard creation, and alerting strategies.
The candidate will work closely with architecture, application, infrastructure, and operations teams to implement monitoring solutions based on High-Level Designs (HLDs) and Low-Level Designs (LLDs).
Key Responsibilities
- Design and implement end-to-end observability solutions across infrastructure, applications, databases, and cloud environments.
- Configure and manage Splunk, Datadog, Grafana, and Prometheus platforms.
- Build and maintain monitoring dashboards, alerts, and reports aligned with business and operational requirements.
- Develop and optimize log ingestion pipelines, parsing rules, indexing strategies, and retention policies.
- Implement APM instrumentation, distributed tracing, and custom metrics collection.
- Configure and manage synthetic monitoring, anomaly detection, and database monitoring.
- Create and maintain service health indicators, SLI/SLO frameworks, and performance baselines.
- Translate HLDs and LLDs into scalable monitoring architectures and operational dashboards.
- Collaborate with development and infrastructure teams to troubleshoot performance issues and conduct root cause analysis (RCA).
- Define monitoring standards, best practices, and observability governance models.
- Support incident response activities and drive continuous improvements in monitoring coverage and reliability.
Required Skills & Experience
- 6+ years of experience in observability, monitoring, or site reliability engineering roles.
- Strong expertise in Splunk administration, configuration, and log analytics.
- Hands-on experience with Datadog, Grafana, and Prometheus.
- Experience implementing Application Performance Monitoring (APM) and distributed tracing solutions.
- Proficiency in log ingestion, custom metrics, synthetic testing, anomaly monitoring, and database monitoring.
- Experience designing monitoring solutions based on HLDs and LLDs.
- Strong understanding of infrastructure monitoring across servers, containers, cloud platforms, and network components.
- Knowledge of OpenTelemetry, metrics collection frameworks, and observability best practices.
- Strong analytical and troubleshooting skills with experience in root cause analysis.
Preferred Qualifications
- Experience with cloud platforms such as AWS, Azure, or GCP.
- Knowledge of Kubernetes, Docker, and microservices architectures.
- Experience with Infrastructure as Code (Terraform, Ansible) is a plus.
- Familiarity with DevOps, CI/CD pipelines, and SRE principles.
Key Skills
Splunk | Datadog | Grafana | Prometheus | Observability | APM | Log Ingestion | Synthetic Monitoring | Anomaly Detection | Custom Metrics | Database Monitoring | Distributed Tracing | OpenTelemetry | Dashboarding | Alerting | Infrastructure Monitoring | HLD | LLD | SRE | RCA
