GCP Data Engineer

Job Title: GCP Data Engineer

Role: Engineer

Responsibilities

We are looking for a motivated GCP Data Engineer with good analytical skills. As a GCP Data Engineer, you will be responsible for assisting in various projects related to the development and maintenance of GCP Data warehouses. You will work closely with a team of experienced professionals to design and maintain warehouse components and data pipelines.

Key Responsibilities

• Collaborate with senior team members to develop and maintain various data components like databases, data pipelines, etc.
• Assist in the development and implementation of Google Big Query components, Airflow DAGs and GCP components for batch and stream processing.
• Work with other team members to ensure project deadlines are met.
• Conduct research and development to improve performance of data pipelines maintained in Airflow DAGs and Google Big Query SQL scripts
• Participate in team meetings to discuss project updates and progress.
  
  

Skill Requirements

• 1 to 8 years of IT development experience required
• Knowledge of Databases, SQL is a must
• Experience with development of ETL pipelines in Google Cloud Platform (GCP) is a must - Experience in BigQuery. Cloud Functions, Composer is a must
• Experience with Git, JIRA and any DevOps tool for CI/CD automation is preferred. - Understanding of basic programming languages or relevant skills.
• Strong analytical/problem solving skills
• Excellent communication skills and ability to work in a team environment.
  
  

TCS Confidential

Must Have Skills

Strong SQL Understanding with Hands-on experience

Data Warehousing concepts

GCP BigQuery, Cloud Functions, Composer/Airflow

Good To Have Skills

Python

Experience working on a project with Azure DevOps pipelines and CI/CD automation Experience with Github or any other Version control Tool

GCP – Dataflow, Pub/Sub

Knowledge on Change Management Process and Agile Events

Experience with JIRA or any other ticketing tool in Agile project management

Project Specific Requirements

Duration of interview will be 60 minutes. Types of questions should test on the topics listed below. Based on the experience of the candidate / associate, difficulty level of the questions should vary. For every 3 simple questions, 1 medium or advanced questions are to be asked.

Must-Have Skills

• Strong SQL Understanding with Hands-on experience
• Data Warehousing concepts
• GCP - BigQuery, Cloud Functions, Composer/Airflow, Cloud Scheduler Good to have Skills:
• Python Basics
• Experience working on a project with Azure DevOps pipelines and CI/CD automation
• Experience with Github or any other Version control Tool
• GCP – Dataflow, Pub/Sub
  
  

Sample Questions – Difficulty level: Medium

• What is the difference between full outer join and cross join
• Given an employee table with emp_id, dept_id, salary, write a query to find the details of the employees who are earning the highest salary in each department of the company.
  
  

TCS Confidential

• What are Slowly Changing Dimensions Type 2 Can you give a practical example. 4. What is the difference between a list and tuple
• What is the use of a composer in an ETL pipeline
• What are slots in GCP BigQuery
• What are the best practices while running queries in BigQuery
• What are the common operators used in composer
  
  

Sample Questions – Difficulty level: Advanced

• Given a customer transaction table with cust_id, txn_date, txn_id, txn_amount, Write a query to show the running totals of the customer spending over the last 3 months. 2. We need to store details about vendors in our system - name, address, contact no and email. Any of these fields might change over a period. The system needs to store the history of changes made to the vendor details. Design tables to store the main data as well as history of changes 3. What are the different partition granularities allowed on a timestamp column in BigQuery 4. In Composer/airflow, how will you give handshake between 2 DAGs
• In BigQuery or SQL, how will you avoid full table scan during MERGE statement 6. A task C in a DAG has 2 predecessor tasks – A and B. I want to trigger C even if either one of the tasks – A or B is successful. How will you achieve this
  
  

Skills: composer,pipelines,gcp,cloud,components