I
Technical Lead
I
- Posted 15 hours ago
- Be among the first 10 applicants
Job Description
Job Description
Skillset: (Senior Data Engineer)
- . Hands-on experience with GCP (BigQuery, Composer, Dataproc, Dataflow, Pub/Sub)
- . Working knowledge for data processing and pipelines
- . Strong programming skills in SQL and PySpark
- . Experience with Oracle and SQL Server databases
- . Understanding of data modelling (dimensional / medallion basics)
- . Experience in batch pipelines and exposure to streaming
- . Familiarity with data ingestion, ETL/ELT processes
- . Basic understanding of data quality and validation
- . Exposure to Airflow/Composer and scheduling
- . Working knowledge of Git and CI/CD
Detailed Responsibilities:
- . Develop and maintain batch and streaming data pipelines
- . Build SQL and PySpark transformations for large datasets
- . Ingest data from multiple sources into BigQuery/cloud platforms
- . Implement data transformation logic as per requirements
- . Work with teams to consume curated datasets
- . Develop workflows using Cloud Composer (Airflow)
- . Perform data validation and reconciliation
- . Optimise queries and pipelines for performance
- . Support migration of legacy ETL jobs to cloud
- . Troubleshoot pipeline and performance issues
- . Participate in testing, UAT, and deployment
- . Document pipelines, transformations, and sources
- . Collaborate with analytics and reporting teams.




