Gcp Data Engineer

Key Responsibilities:

• Data Pipeline Development: Design, develop, and maintain scalable batch and streaming data pipelines on Google Cloud Platform (GCP).
• Big Data Tools: Work extensively with BigQuery, Dataflow, Pub/Sub, Cloud Composer (Airflow), and Dataplex for data integration and orchestration.
• Programming: Write optimized SQL and Python code for data processing and transformations.
• Streaming & Batch Processing: Build and manage pipelines using Apache Beam for both batch and streaming workloads.
• CI/CD & Infrastructure: Implement CI/CD workflows and Infrastructure as Code using GitHub Actions.
• Governance & Security: Ensure data governance, security, observability, and monitoring of data pipelines.
• Performance & Cost Optimization: Optimize pipelines for performance, scalability, and cost efficiency.
• AI/ML Integration: Integrate data pipelines with Vertex AI to support AI/ML use cases.
• Collaboration: Work closely with cross-functional engineering and data teams to ensure reliable delivery of data solutions.