Project Role : Custom Software Engineer
Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.
Must have skills : Google Cloud Platform Architecture
Good to have skills : Google BigQuery
Minimum 3 Year(s) Of Experience Is Required
Educational Qualification : 15 years full time education
Key Responsibilities
Data Pipeline Development
Design and implement robust, scalable batch and streaming data pipelines using Python and GCP services.
Build and maintain data transformation workflows in Dataform (SQLX) integrated with BigQuery.
Develop event-driven and real-time data processing solutions using Pub/Sub and Cloud Functions / Cloud Run.
Manage large-scale structured and unstructured data stored in GCS, BigQuery, and Cloud Spanner.
Python & Google SDK
Write clean, well-structured Python code leveraging GCP client libraries (google-cloud-bigquery, google-cloud-storage, google-cloud-pubsub, google-cloud-spanner, etc.).
Develop reusable Python modules, utilities, and SDKs for internal data platform tooling.
Ensure all code meets quality standards through thorough unit testing with pytest and code reviews.
Infrastructure & DevOps
Provision and manage GCP infrastructure using Terraform, following infrastructure-as-code (IaC) best practices.
Maintain CI/CD pipelines in GitLab for automated testing, linting, and deployment of data workloads.
Understanding of monitoring, alerting, and observability for data pipelines and GCP resources.
Manage containerised workloads on Cloud Run, including Dockerfiles and container registries.
Skills
5+ years of experience in data engineering with at least 2 years on GCP.
Strong Python programming skills — clean code, OOP, packaging, and performance optimization.
Hands-on experience with BigQuery: query optimization, partitioning, clustering, and data modelling.
Experience with Pub/Sub for event-driven architectures and streaming ingestion.
Proficiency in Cloud Run and/or Cloud Functions for serverless and containerized workloads.
Experience writing unit tests with pytest — mocking, fixtures, coverage reporting.
Hands-on Terraform experience for GCP resource provisioning (projects, IAM, networking, data services).
Experience with GitLab CI/CD — pipeline authoring, environment management, secrets handling.
Solid understanding of GCS for data lake patterns — file formats (Parquet, Avro, JSON), lifecycle policies.
Nice to Have
Experience with Dataform for SQL-based transformation pipelines in BigQuery.
Familiarity with Cloud Spanner for globally distributed relational workloads.
Knowledge of data mesh or data lakehouse architecture patterns.
Knowledge/Experience with Apache Beam / Dataflow for large-scale streaming or batch processing.
GCP Professional Data Engineer or Cloud Architect certification.
Familiarity with Apache Airflow / Cloud Composer, or similar orchestration tools.
Additional Information:
- The candidate should have minimum 3 years of experience in Google Cloud Platform Architecture.
- This position is based at our Bengaluru office.
- A 15 years full time education is required.
, 15 years full time education
