Job Role - QA with Azure and Databricks
Experience Required - 7 to 14 Years
Location - Indore/ Pune/ Chennai/ Bangalore
Immediate joiners are preferred.
- Key Responsibilities
- 1) Test Strategy & Planning
- Define and maintain test strategies for data pipelines, notebooks, jobs, and APIs across Databricks & Azure.
- Translate business rules and data quality SLAs into testable acceptance criteria.
- Establish risk-based test plans across functional, integration, system, and non-functional (performance, scalability, security) testing.
- 2) Data & Pipeline Testing (Databricks / Spark / Delta)
- Develop automated tests for Spark transformations (e.g., PySpark, Spark SQL) and Delta Lake features (MERGE, time travel, schema evolution).
- Implement data quality checks (completeness, accuracy, consistency, referential integrity, drift detection).
- Validate batch and streaming pipelines, job dependencies, and SLAs.
- Create reproducible test datasets and use Delta Live Tables (if applicable) to validate expectations.
- 3) Azure Cloud QA
- Test data flows across Azure Data Factory / Synapse pipelines, Databricks Jobs, Azure Storage/ADLS, Azure SQL/SQL MI, Event Hub/Kafka, and Key Vault access patterns.
- Verify IAM/ACLs/Unity Catalog permissions, secrets handling, and network boundary controls.
- Validate infrastructure configurations across environments (Dev/Test/Prod) and support IaC validation (e.g., Bicep/Terraform plan diffs).
- 4) Automation & DevOps
- Integrate tests into CI/CD (Azure DevOps/GitHub Actions): unit, integration, data-validation, and post-deployment smoke tests.
- Implement quality gates (coverage thresholds, schema checks, contract tests, static checks like Pylint/flake8).
- Build test utilities and shared libraries for data assertions, fixtures, and synthetic data generation.
- Use feature branches, PR checks, and merge policies to enforce quality.
- 5) Observability & Reliability
- Instrument tests and pipelines with logging & metrics (e.g., Azure Monitor, Log Analytics, Databricks metrics).
- Define SLIs/SLOs for data quality and pipeline reliability; participate in incident reviews and RCA.
- Establish quality dashboards (test pass rate, DQ incidents, recovery time, flaky test rate).
- 6) Collaboration & Governance
- Partner with data engineers, platform engineers, product owners, and data stewards.
- Contribute to test data management practices and a data contract approach with upstream/downstream teams.
- Document test plans, runbooks, and traceability from requirements to tests.
- Required Qualifications
- 38+ years in QA/Test Engineering with 2+ years in data platform or Spark-based environments.
- Hands-on with Databricks: Notebooks, Jobs, Delta Lake, cluster policies.
- Strong Python for tests/utilities; solid SQL for validation and profiling.
- Experience testing ETL/ELT patterns, data partitioning, late-arriving data, slowly changing dimensions, idempotency.
- Azure experience: Data Factory, ADLS Gen2, Key Vault, Azure DevOps/GitHub.
- CI/CD for data: pipelines, artifacts, environments, approvals, test stages and post-deploy validations.
- Familiar with schema validation and API testing (REST/Databricks Jobs API).
- Clear understanding of security (RBAC/ABAC, managed identities), cost & performance considerations.
- Nice-to-Have
- Unity Catalog, data lineage, and governance testing.
- Streaming (Structured Streaming, Delta Live Tables), event-driven validation.
- Performance testing of Spark jobs (cluster sizing, shuffle hotspots, skew).
- Exposure to privacy & regulatory controls (PII masking, GDPR/CCPA, SOX).
About InfoBeans:
InfoBeans is a global digital transformation and product engineering company, enabling businesses to thrive through innovation, agility, and cutting-edge technology solutions. With over 1,700 team members across the globe, we specialize in custom software development, enterprise solutions, cloud, AI/ML, UX, automation, and digital transformation services.
At InfoBeans, we live by our core purpose of Creating WOW!for our clients, team members, and the community. Our collaborative culture, growth opportunities, and people-first approach make us one of the most trusted and rewarding workplaces.
