Key Responsibilities
Data Ingestion & Pipeline Management
- Configure and execute data ingestion pipelines using a reusable GCP-based framework (ODP).
- Manage job configurations, runtime parameters, and metadata for ingestion processes.
- Monitor daily job executions, ensure SLA compliance, and troubleshoot failures.
- Debug and resolve runtime data issues independently.
- Escalate framework-level issues to the core development team when required.
GCP Services & Cloud Implementation
- Work extensively with GCP services including GCS, BigQuery, Cloud Composer, Data Fusion, Dataproc, and GKE.
- Design and optimize ELT/ETL workflows and data processing jobs.
- Ensure secure and scalable data storage and processing solutions.
Infrastructure & DevOps
- Maintain infrastructure and deployment scripts using Terraform (Infrastructure as Code).
- Follow version control best practices using Git.
- Contribute to minor framework enhancements and continuous improvements.
Collaboration & Governance
- Collaborate with cross-functional teams for project delivery.
- Suggest process and framework improvements based on hands-on experience.
- Maintain documentation and ensure operational best practices.
