Software Engineer – Backend (Python)

Job Timing – 6:30pm IST to 3:30am IST

Remote role

Immediate Joiner

(Log Intelligence & Observability)

We are building a sophisticated service designed to transform raw log data into actionable answers. The system takes user input, fetches relevant logs from sources like Splunk, and processes them for delivery. As part of this flow, you will help architect intermediate caching layers and data processing pipelines to ensure fast, reliable access to distributed log data.

While Splunk is our primary focus today, our vision is a log-agnostic platform capable of extracting and aggregating data from a variety of sources to provide a unified troubleshooting experience.

Core Responsibilities

You will be responsible for the end-to-end flow of log retrieval and processing, ensuring that our users get the data they need with minimal latency.

Pipeline Development: Build and optimize the Input-to-Answer workflow: taking user input, executing Splunk queries, caching outputs, and delivering final results.
API & Integration: Develop RESTful Flask APIs and integrate with the Splunk API across five distributed computing log layers.
Cross-Platform Expansion: Architect the system to eventually support log extraction from other sources, including ELK (Elasticsearch/Logstash/Kibana), Graylog, and more.
Caching & Optimization: Implement intermediate caching layers, potentially utilizing alternative log processing solutions to speed up data delivery.
Security & Validation: Manage service-to-service authentication (JWT/Shared Secrets) and implement query validation and workload analysis.

Required Technical Skills

Python & Flask: Proficiency in Python 3.9+ and the Flask framework for microservices.
Advanced Splunk Querying: Strong experience with Splunk Query Language (SPL) and the Splunk SDK for Python is essential. You should be comfortable writing complex searches to extract specific insights from high-volume data.
Kubernetes Knowledge: Deep understanding of Kubernetes logs is required to effectively validate and interpret the content being analyzed.
Troubleshooting: Proven ability to troubleshoot distributed systems using logs. You should understand how to trace an issue across multiple services and nodes.
Testing: Strong commitment to quality, specifically requiring experience writing Integration Tests and unit tests using Pytest.
Containerization: Experience with Docker and Kubernetes/Helm for deployment.

Preferred Qualifications (Nice-to-Have)

Distributed Systems: Hands-on experience with Spark or Flink. Since our log-fetching mechanisms are part of a distributed architecture, this experience is highly valuable for understanding the data lifecycle.
Multi-Stack Experience: Familiarity with ELK, Graylog, ClickHouse, or OpenSearch, as we plan to integrate these into our extraction engine.
Advanced Observability: Experience with platform engineering or advanced log management (index management, retention policies).
Specialized Tooling: Familiarity with internal tools such as Whisper, Mosaic, or Rio.

Tech Stack

Python, Splunk, Kubernetes, Flask API, Integration Testing