SRE - Observability Expert

Key Responsibilities:

• Design, build, and maintain flexible and secure data ingestion, streaming, transformation, analytics, and data lake storage solutions.
• Develop and maintain infrastructure-as-code using Terraform for Kubernetes clusters (AKS) and Databricks environments.
• Build and manage data pipelines using Azure Data Factory, Amazon Data Pipeline, Apache Spark, Databricks, and other orchestration tools.
• Implement streaming and batch processing solutions using Kafka, Apache Storm, Spark Streaming, Apache Flink, and Kappa architecture.
• Architect storage strategies for performance, accessibility, availability, and cloud cost optimization.
• Manage data ingestion and distribution using Event Hubs, Kafka topics, ADLS2, REST APIs, and related connectors.
• Set up and manage open-source tools like Airflow, Druid, Kafka, OpenSearch, and Superset.
• Automate and integrate tasks using Python scripting; build high-performance APIs using FastAPI.
• Handle cloud services, IAM, auto-scaling, high availability, elasticity, and networking requirements.
• Implement federated access control for multi-tenant cloud environments.
• Manage data catalogs per domain and ensure governance, compliance, and security best practices.
• Research and implement emerging technologies to continuously evolve the data platform.
• Collaborate in Agile/Scrum environments and apply DevSecOps practices, including Day 2 operations with Datadog.