Key Responsibilities:
Data Engineering & Pipeline Development
- Design, build, and maintain scalable ETL/ELT pipelines using SSIS, Azure Data Factory, and Azure Databricks.
- Develop efficient data ingestion, transformation, and processing frameworks for large-scale enterprise systems.
- Ensure reliability, scalability, and performance of data pipelines.
Data Architecture & Warehousing
- Design and implement enterprise data warehouse and data mart solutions using Kimball methodology.
- Develop large-scale database architectures with partitioning, indexing, compression, and encryption.
- Support implementation of scalable MPP systems such as Azure Synapse and Snowflake.
Azure Cloud & Platform Engineering
- Work with Azure services including AKS, Event Hub, Azure Functions, Cosmos DB, and Azure DevOps.
- Build and deploy cloud-native data engineering solutions.
- Support cloud migration and modernization of legacy data systems.
- Ensure scalability, availability, and resilience of cloud-based data platforms.
Data Modeling & Database Management
- Perform advanced data modeling and design relational and dimensional data structures.
- Manage SQL Server, MySQL, and Oracle databases with strong T-SQL expertise.
- Optimize database performance through query tuning, indexing, and partitioning strategies.
Big Data & Advanced Data Processing
- Work with big data technologies including Spark and NoSQL databases.
- Implement micro-batch and streaming data processing solutions.
- Handle large datasets with focus on performance optimization and scalability.
Data Quality & Governance
- Implement data quality frameworks, metadata management, and Master Data Management (MDM).
- Ensure data integrity, consistency, and compliance with organizational standards.
- Maintain documentation and enforce data engineering best practices.
BI & Reporting Support
- Collaborate with BI teams to support dashboards, cubes, and reporting solutions.
- Identify and resolve data issues impacting reporting and analytics systems.
- Contribute to definition and implementation of business metrics and KPIs.
DevOps & CI/CD Integration
- Implement CI/CD pipelines for data engineering workflows using Azure DevOps.
- Automate deployment, monitoring, and maintenance of data pipelines.
- Support continuous improvement of engineering and deployment processes.
Collaboration & Leadership
- Provide technical leadership and mentoring to data engineering teams.
- Work with business stakeholders to define data requirements and reporting needs.
- Participate in architecture reviews, SDLC processes, and agile ceremonies.
- Troubleshoot complex system and data issues across environments.
