Quality Engineer

Project Role : Quality Engineer

Project Role Description : Enables full stack solutions through multi-disciplinary team planning and ecosystem integration to accelerate delivery and drive quality across the application lifecycle. Performs continuous testing for security, API, and regression suite. Creates automation strategy, automated scripts and supports data and environment configuration. Participates in code reviews, monitors, and reports defects to support continuous improvement activities for the end-to-end testing process.

Must have skills : Data Engineering

Good to have skills : NA

Minimum 3 Year(s) Of Experience Is Required

Educational Qualification : 15 years full time education

Purpose of the Job - The Data & Analytics Data Engineer plays a pivotal role in advancing the organization's data strategy and delivering robust analytical solutions across critical business functions. This position is responsible for designing, developing, and deploying scalable data pipelines and infrastructure. Working within a cross-functional team, the Data Engineer will identify analytical opportunities, contribute to solution architecture, and ensure the secure and compliant handling of confidential data.

Responsibilities:

Retrieve, integrate, and prepare diverse datasets for analysis by the Data & Analytics team and the broader business user community.

Collaborate as a key technical interface with local IT landscape teams and various business stakeholders to understand data requirements and deliver effective solutions.

Key Responsibilities: The Data & Analytics Data Engineer is responsible for designing, developing, and deploying robust data products and solutions within a Microsoft Azure-based data ecosystem. This role involves:

Data Product Development: Leading the retrieval, migration, integration, and construction of data products, applying industry-proven disciplines in Data Acquisition, Integration, Curation, and Distributed Data Processing to support diverse Data & Analytics use cases.

Pipeline Architecture & Automation: Designing, deploying, and automating scalable data pipelines in an Azure Databricks-based Lakehouse, leveraging Azure Data Factory and Databricks best practices. This includes transforming and curating datasets for rapid consumption and discovery by analytical and business users.

Data Modernization: Contributing to the migration of complex on-premise data solutions to Azure, ensuring efficiency and scalability.

Technical Leadership: Providing technical expertise and mentoring to foster best practices within the data engineering space and coordinating development and support activities.

Required Skills & Experience:

Azure Data Platform Expertise:

Solid experience in building modular, extensible, and highly resilient ETL/ELT data pipelines using Azure Data Lake Storage (ADLS), Azure Data Factory, Azure Databricks, and Azure Integration Runtime.

Proven ability to enable and support BI/reporting application development using Microsoft Power BI.

Solid experience with Microsoft Azure-based Enterprise Data Warehouse implementations, including DWH/Datamart design, development, and Data Engineering for supporting enterprise-scale BI/Reporting applications.

In-depth knowledge of the Azure data ecosystem, including Azure SQL Data Warehouse (Synapse), MS SQL Server, and advanced Databricks features (Lakehouse, Delta Lake, Unity Catalog), as well as connectivity solutions like Express Route.

Data Migration & Modernization:

Solid experience migrating on-premise data pipelines built using technologies like Informatica to Azure Data Factory/Databricks.

Knowledge of migrating large, complex on-premise Data Warehouses (e.g., IBM DB2 LUW, z/OS) to MS Azure-based data warehouses/lakehouses.

DevOps & Automation:

Extensive experience with data pipeline automation, orchestration, and administration using a comprehensive DevOps toolchain (e.g., Artifactory, GitHub, Confluence, Jira, ServiceNow, Azure DevOps).

Proficiency in supporting large, complex daily batch ingestion processes with a strong emphasis on Delta processing techniques.

Data Governance & Quality:

Knowledge of Data Modeling principles and Metadata Management techniques (e.g., data lineage) using tools like Databricks, Sybase Power Designer, or Alation Data Catalog.

Work directly with Data & Analytics, IT, Global Cloud Solutions teams, Source Systems as well as representatives from vendors to identify and resolve issues and remove obstacles in achieving the Data & analytics goals.

Expertise in designing and building efficient data pipelines and workflows to support custom workloads while ensuring data quality using Microsoft Azure technologies (Data Factory, ADLS, Databricks, Integration Runtime, ExpressRoute, Power BI, DevOps, MLflow, Artifactory, Azure SQL Data Warehouse, Synapse).

Strong experience in assembling and managing large, complex datasets to meet the functional requirements of Data Warehouse and BI applications.

Skilled in transforming raw and disparate data into structured, clean, and accessible datasets for Business Intelligence and analytics applications.

Proficient in automating data processing workflows and pipelines using CA Autosys R11, Python, shell scripting, and Azure DevOps CI/CD processes, with experience integrating data products through APIs using JSON/XML.

Experience working in distributed cluster environments with Unix/Linux command-line interfaces and Bash scripting.

Query, retrieve, integrate, and prepare data to support profiling, analysis, and the development of BI and reporting applications.

Analyze and profile datasets to identify sensitive fields and assist in defining rules for data anonymization and tokenization.

Generate descriptive statistics (e.g., min, max, count, sum), identify data types and field lengths, and detect recurring patterns while tagging data with relevant keywords, descriptions, or categories.

Perform data quality assessments, evaluate join feasibility and risks, and discover and validate metadata.

Analyze data distributions, identify potential keys and functional dependencies, detect embedded value dependencies, and conduct inter-table relationship analysis.

Prepare datasets optimized for consumption by data visualization tools such as Power BI.

Integrate Data Warehouse/Lakehouse pipelines and workflows with visualization platforms (e.g., Power BI) to deliver multiple data streams and enable rich business insights.

Administer and monitor Data Lake/Lakehouse environments supporting batch and near real-time data processing.

Establish connectivity and data interfaces between Azure platforms and diverse source systems, including application databases, data warehouses, data marts, and other analytics systems.

5+ Years experience designing and building data pipelines in Microsoft Azure (Data Factory, Databricks (Lakehouse, Delta Lake, Unity catalog), ADLS, Integration Runtime, Express route & MS SQL Server) supporting Datawarehouses.

Proficient in creating data pipelines and in hybrid data movement from on-prem systems to MS Azure.

Experience with object-oriented programing languages such as Python, R, Scala, and Java.

Experience using wide variety of file formats and compression techniques.

Experience with CI/CD processes.

Knowledge of emerging Data Engineering tools & technologies.

Broad understanding and experience with BI and real time analytics platforms.

Ability to collect data, establish facts, and identify trends and variances.

Ability to Integrate information from a variety of sources with varied levels of complexity

Ability to Review and interpret and evaluate information.

Ability to Formulate and test hypotheses for the purpose of forecasting outcomes.

, 15 years full time education