Job Title: Data QA Engineer
Location: Bangalore
Experience: 5 to 8 Years
Job Description
We are seeking a highly skilled and detail-oriented
Software Test Engineer (Data QA) with hands-on experience in validating
real-time data streaming platforms,
cloud-native architectures, and
modern data ecosystems. The ideal candidate will be proficient in testing complex data pipelines, ensuring data integrity, and validating system reliability within distributed environments.
Key Responsibilities
- Design, implement, and execute manual and automated test strategies for real-time streaming use cases using Azure Service Bus, Event Hubs, Kafka, and Azure Functions.
- Validate Spark Streaming applications, including unbounded data flows, streaming DataFrames, checkpoints, and streaming joins.
- Develop and execute test plans for containerized microservices deployed on Kubernetes, focusing on scalability, resilience, and fault tolerance.
- Test and verify data ingestion, transformation, and consumption workflows across open table formats such as Delta Lake, Apache Iceberg, and Hudi.
- Collaborate with engineering and data teams to ensure data quality, reliability, and compliance across distributed systems.
Mandatory Skills (Must Have)
- 58 years of experience in data quality assurance or data testing.
- Hands-on experience in real-time data streaming platforms Azure Service Bus, Event Hubs, or Kafka.
- Strong understanding of Spark Streaming and real-time data validation concepts.
- Experience in Kubernetes-based deployments and microservices testing.
- Proficiency in SQL, Python, or any scripting language for data validation and automation.
- Solid understanding of data lake architectures, ETL pipelines, and data governance principles.
Good To Have Skills
- Experience with observability stacks such as Prometheus, Grafana, or ELK for performance monitoring and troubleshooting.
- Exposure to analytical databases and query engines like Trino, StarRocks, or ClickHouse.
- Understanding of data mesh architecture and domain-oriented data quality frameworks.
- Familiarity with Azure Cloud ecosystem and CI/CD pipelines for data workflows.
