Job Summary
The Data Engineering Specialist is responsible for designing, implementing, and maintaining scalable data solutions on the Microsoft Azure Data Platform and SQL Server environment. The role involves collaborating with cross-functional stakeholders to ensure efficient processing, storage, governance, and retrieval of large volumes of structured and unstructured data. The candidate will play a key role in optimizing data architecture, improving performance, and enabling reliable data-driven decision-making.
Roles and Responsibilities
Technical Responsibilities
- Design, build, and maintain scalable and reliable data pipelines.
- Develop and implement ETL solutions using Azure Data Factory and Azure Databricks to extract, transform, and load data across multiple source and target systems.
- Analyze existing data landscapes and recommend re-architecture, optimization, and streamlining strategies to enhance scalability and efficiency.
- Collaborate effectively with onshore counterparts to address technical gaps, requirement challenges, and complex business scenarios.
- Monitor, troubleshoot, and optimize data systems to ensure high performance, reliability, and availability.
- Apply strong database management principles to maintain data integrity and consistency.
- Optimize data processes for performance, scalability, and cost efficiency.
- Ensure data architecture aligns with business requirements and data governance standards.
- Define and execute data engineering strategies aligned with organizational objectives.
- Integrate data from multiple internal and external systems while ensuring quality and consistency.
- Perform data validation, testing, and verification to ensure accuracy of transformations and support analytical or machine learning models.
- Define and implement data mapping in collaboration with business, digital, and data teams.
- Maintain, test, and validate data pipelines to ensure optimal performance.
- Assemble large and complex datasets that meet both functional and non-functional business requirements.
- Identify data gaps and work with business and IT stakeholders to align on data needs.
- Troubleshoot and resolve technical data issues promptly and efficiently.
- Optimize data flow and collection processes for cross-functional teams.
- Collaborate with IT and DevOps teams to align data infrastructure with overall enterprise architecture.
- Implement best practices for data security, privacy, and compliance.
- Drive continuous improvement initiatives within the data engineering function.
- Understand the impact of data conversions on servicing operations and manage complex cases accurately and efficiently.
Domain & Additional Responsibilities
- Familiarity with ACH transaction lifecycle (debit, credit, returns, NOC, settlement).
- Experience in financial data reconciliation and ledger-based systems.
- Understanding of bank integrations and multi-bank processing workflows.
- Support and execute data migrations across multiple databases and servers.
- Stay updated with emerging technologies and industry trends.
Key Skills & Competencies
- Strong expertise in Azure Data Factory, Azure Databricks, Azure SQL, ADLS, Azure Functions, and Power BI.
- Advanced proficiency in SQL, complex query development, and performance tuning.
- Strong understanding of data warehousing concepts, ETL processes, and data lifecycle management (ingestion, transformation, loading, validation, and optimization).
- Experience handling large volumes of structured and unstructured data.
- Knowledge of DevOps practices and CI/CD frameworks for data pipelines.
- Experience with Infrastructure as Code (IaC) tools such as Terraform.
- Strong understanding of data structures and algorithms.
- Excellent analytical, problem-solving, and troubleshooting skills.
- Ability to work in onshore-offshore models and manage challenging technical scenarios.
- Strong written and verbal communication skills with the ability to work independently and collaboratively.
- Ability to handle ambiguity and convert vague requirements into structured deliverables.
Expectations from the Current Role/Profile
- Proven experience in designing and developing data warehouse solutions using Azure SQL, ADLS, ADF, Azure Functions, and Power BI.
- Demonstrated expertise in SQL optimization and relational database management.
- Experience building and managing ETL pipelines in Azure Data Factory.
- Hands-on experience with Azure cloud services and DevOps/CI/CD frameworks.
- Strong capability in data ingestion, transformation, validation, and performance tuning.
- Ability to contribute to architecture frameworks, best practices, and technical design discussions.
- Drive automation initiatives across the data analytics team.
- Maintain a balance of technical depth and strong interpersonal skills to effectively collaborate with stakeholders.
If you intersted please share your cv @[Confidential Information]
