Job Description : Sr. GCP Data Engineer
Experience : 7 to 12 Years
Work location: Chennai, Bangalore, Hyderabad, Pune-Hybrid
Shift Timing : 2 to 11 PM
I
nterview process : L1 and L2 round
Job description:
- 5+ years experience
- Should have experience in GCP BigQuery, DataProc(PySpark)
- Good to have experience on Informatica
Requirements
- How do you upload files to GCS
- Using Google Cloud Console (UI) drag and drop files into a bucket.
- Using gsutil CLI:
- How do you query data in BigQuery
Write SQL queries in:
- BigQuery Console (UI)
- bq CLI bq query SELECT ...
APIs or client libraries (Python, Java, etc.)Cluster monitoring with Cloud Monitoring & alerts.Upload the script to the DAGs folder in Composer's GCS bucket Composer automatically deploys it.
- What is the purpose of a Dataproc job Dataproc job is a task you submit to a cluster to process data, such as:
- Spark job (Scala/Python/Java)
- Hive query
- Pig script
- Hadoop MapReduce job
- How do you handle errors in Dataproc jobs
- Job retry policies (set retries in workflow templates).
- Error logs in Cloud Logging (examine stdout/stderr).
- Graceful failure handling with workflow DAG (e.g., skip/stop).
- How do you create an Airflow DAG in Cloud Composer
- Write a Python script defining the DAG (dag_id, schedule_interval, default_args).
- Define tasks using operators (e.g., PythonOperator, BashOperator, BigQueryOperator)
