V
Lead Software Engineer
V
- Posted 11 hours ago
- Be among the first 10 applicants
Job Description
We are looking for a detail-oriented QA Engineer – Data Engineering to ensure the quality, accuracy, completeness, and reliability of data pipelines, ETL/ELT workflows, data lakes, and data warehouse solutions.
The candidate will be responsible for validating data ingestion, transformation logic, source-to-target reconciliation, schema changes, and data processing workflows. The ideal candidate will have strong hands-on experience with SQL, Python, data validation, ETL testing, and automated data quality frameworks. Familiarity with AWS data services, distributed data processing, and pipeline observability is highly desirable
Analyze data engineering requirements and develop comprehensive test plans, test scenarios, and test cases.
Validate end-to-end data pipelines, including ingestion, transformation, processing, and loading.
Perform source-to-target data reconciliation to verify record counts, field mappings, and transformation accuracy.
Develop automated data validation scripts using Python and SQL.
Test ETL/ELT workflows for data completeness, accuracy, consistency, uniqueness, and integrity.
Validate structured and semi-structured datasets, including CSV, JSON, XML, and Parquet.
Test data transformations, aggregations, joins, filtering logic, and business rules.
Identify duplicate records, missing values, invalid data, schema mismatches, and unexpected data changes.
Data pipeline validation, E2E testing automation, data quality checks, observability validation
The candidate will be responsible for validating data ingestion, transformation logic, source-to-target reconciliation, schema changes, and data processing workflows. The ideal candidate will have strong hands-on experience with SQL, Python, data validation, ETL testing, and automated data quality frameworks. Familiarity with AWS data services, distributed data processing, and pipeline observability is highly desirable
Analyze data engineering requirements and develop comprehensive test plans, test scenarios, and test cases.
Validate end-to-end data pipelines, including ingestion, transformation, processing, and loading.
Perform source-to-target data reconciliation to verify record counts, field mappings, and transformation accuracy.
Develop automated data validation scripts using Python and SQL.
Test ETL/ELT workflows for data completeness, accuracy, consistency, uniqueness, and integrity.
Validate structured and semi-structured datasets, including CSV, JSON, XML, and Parquet.
Test data transformations, aggregations, joins, filtering logic, and business rules.
Identify duplicate records, missing values, invalid data, schema mismatches, and unexpected data changes.
Data pipeline validation, E2E testing automation, data quality checks, observability validation
More Info
Key Skills
AWS data services
pipeline observability
automated data quality frameworks
distributed data processing
About Company
Skills:
Xpath, Json, Xslt, Sql, Openshift, Xml, Rest Apis, Kubernetes, SmartCOMM, Cloud-based Deployments, Thunderhead NOW, Document template design and development, SmartDX
Skills:
Java, Software Development Life Cycle, full stack development, Agile Methodologies, Python, cloud native experience, AI-assisted software development tools
Lead Software Engineer | Developer | SQL Server | MSBI | Lead | FTESociete Generale Global Solution Centre
Skills:
.Net Technologies, C#, T-sql, Ssrs, Dts, Ssas, Xml, Msbi, Xsd, SSIS, MS SQL Server 2017 or above
Skills:
Database Management, Network security, Python Scripting, Bash Scripting, Git, Network configuration, Docker, C++ Programming, Multi-threading, ROS2 framework, Linux systems programming in C, Common network protocols, File I/O, Server-client application architecture, CMake build system, Socket communication