Key Responsibilities
- Design and implement scalable data architectures including data lakehouse and real-time streaming solutions.
- Develop and maintain robust data pipelines using Spark, EMR, Glue, and other big data technologies.
- Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data requirements.
- Ensure data quality, integrity, and security across all data platforms.
- Lead data modeling efforts using Kimball, Inmon, Data Vault, and Medallion architectures.
- Establish and enforce best practices for data governance, metadata management, and data lineage.
- Mentor junior engineers and contribute to team development and knowledge sharing.
- Implement CI/CD pipelines and infrastructure-as-code for data workflows using tools like Terraform and CloudFormation.
- Monitor and optimize data workflows and implement alerting mechanisms for pipeline failures.
- Ensure compliance with data privacy regulations such as GDPR and HIPAA.
Technical Skills and Core Competencies
- Expertise in cloud platforms (AWS & Azure) and services such as S3, Kinesis, Redshift, and Athena.
- Azure Blob Storage, Azure Databricks, Azure Data Factory, - Azure Synapse/ Stream Analytics.
- Proficiency in big data technologies including Spark, Hadoop, Hive, and Presto.
- Strong experience with ETL tools and frameworks.
- Knowledge of data lakehouse formats like Delta, Iceberg, and Hudi.
- Experience with data visualization tools such as Tableau and Power BI.
- Familiarity with enterprise architecture frameworks like TOGAF.
- Understanding of microservices architecture and API design for data services.
- Experience with data testing frameworks such as Great Expectations.
- Strong communication and documentation skills.
