Senior Software Engineer

5-7 Years

Save

Early Applicant

Job Description

Required Experience

5+ years of software engineering experience with a focus on data infrastructure, platform engineering, and distributed systems
Strong proficiency in at least one programming language - Python/Go/Scala
Deep understanding of Apache Spark architecture and experience running Spark workloads in production
Solid experience with AWS services, particularly EMR, EKS, S3 etc.
Experience with Data Platform infrastructure on Kubernetes and container orchestration
Strong understanding of data engineering concepts, workflows, and best practices
Proven ability to debug complex issues in distributed systems
Knowledge of data quality frameworks and tooling (Great Expectations, Deequ, Monte Carlo, etc.)
Experience with data lineage tools and metadata management (Apache Atlas, DataHub, OpenLineage, etc.)
Understanding of data governance principles and compliance requirements
Experience with Apache Airflow setup
AWS Certification is a plus (but not required)

Responsibilities

Partner with data engineering teams to understand requirements and translate them into data platform capabilities
Build tooling and automation to improve developer experience and reduce operational overhead
Establish monitoring, alerting, and observability practices for platform health and performance
Drive platform upgrades, migrations, and adoption of new technologies
Own and maintain the Apache Airflow orchestration platform, ensuring reliable DAG execution for data teams across the organization
Manage and optimize the Apache Spark platform on EMR on EKS, supporting large-scale data processing workloads
Design and implement platform improvements to enhance reliability, scalability, and performance
Troubleshoot complex distributed systems issues and provide timely resolution for platform incidents
Optimize resource utilization and cost efficiency across AWS infrastructure
Create and maintain technical documentation, runbooks, and best practices
Participate in on-call rotation and incident response to ensure platform availability
Conduct capacity planning and scaling to accommodate organizational growth