Lead Software Engineer

Key Responsibilities

Design and develop enterprise-grade automation frameworks for GTS platforms.

Build reusable libraries for validating Spark transformations and data pipelines.

Automate Airflow DAG validation and orchestration testing.

Develop Regression Suites Covering

Data quality,Data transformations,Workflow execution,Failure handling and recovery

Create automated comparison frameworks between source and target systems.

Implement performance, stability, and volume testing for Spark applications.

Automate test execution within CI/CD pipelines.

Enable nightly regression execution and automated reporting.

Work closely with Engineering teams to shift testing left and improve delivery quality.

Technical Skills

Java (Mandatory)

Python (Mandatory)

Automation Frameworks – Any of the following

JUnit,TestNG,Cucumber BDD (strongly preferred),REST Assured,Mockito,Data Technologies

Apache Spark / PySpark,SQL/Big Query,XML/JSON processing,Data validation frameworks

Orchestration & CI/CD – Data processing pipeline experience Mandatory

Commercial in Confidence

GitHub Actions,Jenkins Maven Gradle Cloud Platforms - Any GCP (preferred) AWS Required Cloud Experience – either one GCP (mandated) Dataproc BigQuery Cloud Storage Pub/Sub

Cloud Logging IAM Monitoring & Alerting AWS (optional) S3 EMR Lambda IAM CloudWatch Preferred Experience

Building automation frameworks for distributed systems.

Data pipeline validation in cloud environments.

Cross-cloud validation (GCP ↔ AWS).

Data migration testing.Performance testing of Spark jobs.

Regression testing of Airflow-orchestrated workloads.

Enterprise-scale data platforms handling high-volume batch processing.

Ideal Candidates Profile (for both QA and Dev)

The Ideal Recruit Should Be Able To

Create automation frameworks from scratch

Commercial inConfidence

Build regression suites for Spark pipelines

Validate Airflow DAG orchestration end-to-end

Perform AWS vs GCP data comparisons

Automate data quality verification

Integrate testing into CI/CD pipelines

Develop framework utilities for XML/JSON/Avro comparison

Work independently with minimal supervision

Drive automation strategy and improve release confidence across GTS platforms