Join us as we work to create a thriving ecosystem that delivers accessible, high-quality, and sustainable healthcare for all.
Join Our Cloud Infrastructure Engineering & Operations Division as a Senior Site Reliability Engineer
We are seeking a highly skilled Senior Site Reliability Engineer to elevate our Cloud Infrastructure Engineering & Operations team. Your primary mission will be to enhance the performance, reliability, and scalability of our platforms by spearheading the development of a world-class observability ecosystem that drives business success.
The Team:
The Logging, Metrics, and Monitoring (LMM) team is at the forefront of building and delivering observability services and tools for our engineering communities within the Cloud Engineering & Operations, and Research & Development zones. Our solutions are critical-used daily by hundreds of developers to develop, monitor, troubleshoot, and optimize our web services. We manage large-scale, distributed, fault-tolerant systems that collect and host vast volumes of log and metric data, enabling data-driven decision-making across the organization.
Our work has a direct, measurable impact on the productivity of our engineering teams across athenaNation, empowering them to innovate faster and operate more reliably.
In this role, you will tackle a diverse set of challenges-from fine-tuning system performance and scaling services to debugging complex issues. You will partner closely with development teams to deliver new monitoring features, improve existing tools, and solve pressing engineering problems-all within an agile environment that leverages both private and public cloud platforms.
Job Responsibilities
- Automate the deployment, configuration, and management of logging, metrics, and monitoring services leveraging Puppet and Infrastructure as Code best practices to ensure reliable and scalable operations.
- Proactively troubleshoot and resolve complex production incidents, leveraging deep Linux system administration and engineering expertise to minimize downtime.
- Lead cross-functional projects from conception through delivery, including designing scalable technical solutions, managing timelines, and ensuring successful implementation.
- Architect and implement comprehensive monitoring strategies by developing metrics, dashboards, and alerting criteria to enable proactive service performance management and dynamic scaling.
- Collaborate closely with engineering teams during weekly on-call rotations to swiftly diagnose and resolve high-impact issues, fostering a culture of reliability.
- Partner with development teams to enhance their logging and telemetry capabilities, improving observability and operational efficiency.
- Mentor and guide team members on best practices for incident response, system tuning, and service reliability.
Required Qualifications
- 5-8 years of hands-on experience managing mission-critical production environments with a focus on Linux system administration and DevOps practices.
- Expertise on Amazon Web Services and Cloud Native Approaches.
- Experience working on Microservices, production grade infrastructure.
- Proven expertise in managing and optimizing large-scale logging and data platforms such as Kafka, OpenSearch/Elasticsearch, and log forwarding agents like Vector or Fluentd.
- Extensive experience with configuration management tools such as Puppet or Ansible, automating deployment and operations at scale.
- Scripting experience with Python or Bash.
- Demonstrated success troubleshooting and resolving issues in Linux-based production services, including participating actively in on-call rotations.
- Proficiency in scripting and programming languages including Bash, Python, and Golang for automation, tooling, and integrations.
- Strong expertise in Infrastructure as Code using Terraform and AWS CloudFormation to build resilient, repeatable deployment workflows.
- Ability to rapidly adapt to evolving technology environments and business priorities with a bias toward reliability and automation.
Additional Qualifications
- Experience managing large-scale production server fleets (thousands of nodes) with high availability and fault tolerance.
- Deep subject matter expertise in technologies such as Graphite, ClickHouse, Prometheus, Grafana, Docker, Jenkins, and Git.
- Familiarity with AWS cloud architecture, deployment, and operational best practices, with hands-on experience deploying scalable cloud-native applications.
- Proficiency with protocol analyzers like tcpdump and Wireshark for network troubleshooting and performance diagnostics.
