Senior Data Engineer

Big Data Developer / Data Engineer

Job Summary

We are seeking a highly skilled Big Data Developer / Data Engineer with strong expertise in Spark, PySpark, DataProc, and Airflow to design, build, and optimize scalable data processing pipelines on Google Cloud Platform (GCP). The ideal candidate should have hands-on experience with big data technologies, cloud-native architectures, and distributed data processing frameworks.

Key Responsibilities

• Design, develop, and maintain scalable big data solutions using Spark and PySpark.
• Build and optimize ETL/ELT pipelines for large-scale data processing.
• Work with Google Cloud Platform (GCP) services including DataProc, BigQuery, Cloud Storage, Compute Engine, and Cloud Functions.
• Develop workflow orchestration pipelines using Apache Airflow.
• Create and optimize complex SQL queries for data transformation and analytics.
• Process structured and unstructured datasets using Hadoop ecosystem technologies.
• Collaborate with cross-functional teams including Data Analysts, Architects, and Business stakeholders.
• Ensure data quality, reliability, performance, and scalability of data platforms.
• Implement best practices for cloud infrastructure, security, monitoring, and deployment.
• Support CI/CD and containerized deployments using Docker and Kubernetes.

Must-Have Skills

• Strong hands-on experience with:
• Spark
• PySpark
• Google Cloud DataProc
• Apache Airflow
• Expertise in SQL and database optimization.
• Experience with Hadoop and Hive.
• Strong knowledge of Python for scripting and data engineering tasks.
• Hands-on experience with Google Cloud Platform (GCP).
• Experience building scalable distributed data processing systems.

Nice to Have

• Experience with real-time data processing.
• Certifications in Google Cloud Platform (GCP).
• Exposure to Agile/Scrum methodologies.