Design and develop ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) pipelines to process and curate data sets using technologies such as SQL Server, Azure Data Factory, and Databricks ensuring data quality, accuracy, and consistency.
Data Pipelines: Design, build, and maintain robust, large-scale, fault-tolerant and automated data pipelines for batch and streaming data ingestion of data and processing and storing large datasets.
Orchestration: Utilize workflow orchestration tools to automate data pipeline execution and dependency management.
Develop and maintain data infrastructure, including data warehouses, data lakes, and data marts.
Data Architecture and Modeling: Design and implement robust data models and architectures that align with business requirements and support efficient data processing, analysis, and reporting.
Monitoring and Alerting: Integrate monitoring and alerting mechanisms to track pipeline health, identify performance bottlenecks, and proactively address issues. Improve platform resilience, data quality, and observability with advanced monitoring, alerting, and automated data governance.
Build and manage streaming data architectures using tools such as Apache Kafka, and Spark/Flink Streaming.
Deep knowledge of modern lake-house/table formats: Apache Iceberg, Delta Lake, or Apache Hudi.
Enforce standards and automation across the entire data lifecycle, including schema evolution, dataset migration, and deprecation strategies.
Experience managing data lakes, catalogs, and metadata governance in large-scale environments.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide data solutions.
