- Design, develop, and maintain scalable data pipelines using Scala/Python (PySpark).
- Work with big data technologies to process and manage large datasets efficiently.
- Develop and optimize ETL workflows using Apache Airflow.
- Implement real-time and batch data processing solutions using Kafka and other streaming technologies.
- Build and maintain distributed databases (SQL & NoSQL) for efficient data storage and retrieval.
- Develop and manage CI/CD pipelines for data engineering workflows.
- Work with Google Cloud Platform (GCP) Data Tech Stack, including BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Storage.
- Implement data governance, monitoring, and security best practices.
- Collaborate with data scientists, analysts, and software engineers to provide reliable data solutions.
- Optimize query performance and troubleshoot issues related to data pipelines and infrastructure.
Required Skills & Qualifications:
- Strong programming skills in Scala and/or Python (PySpark).
- Hands-on experience with big data technologies such as Hadoop, Spark, or Databricks.
- Experience in scripting (Shell/Python) for automation.
- Good understanding of GCP Data Tech Stack (BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage).
- Experience with Apache Airflow for orchestrating data pipelines.
- Familiarity with CI/CD pipelines and DevOps practices.
- Experience with Kafka for real-time data streaming.
