Data Engineering

• Design, develop, and optimize large-scale data processing pipelines using PySpark.
• Utilize Apache tools and frameworks (e.g., Hadoop, Hive, HDFS) for data ingestion, transformation, and management.
• Ensure high performance and reliability of ETL jobs in production environments.
• Collaborate with Data Scientists, Analysts, and stakeholders to deliver robust data solutions.
• Implement data quality checks and maintain data lineage for transparency and auditability.
• Handle ingestion, transformation, and integration of structured and unstructured data sources.
• (If applicable) Leverage Apache NiFi for automated, repeatable data flow management.
• Write clean, efficient, and maintainable code in Python and Java.
• Contribute to architecture, performance tuning, and scalability strategies.

Required Skills:

• 57 years of experience in data engineering.
• Strong hands-on experience with PySpark and distributed data processing.
• Deep knowledge of Apache ecosystem: Hadoop, Hive, Spark, HDFS.
• Solid understanding of ETL principles, data warehousing, and data modeling.
• Experience with large-scale datasets and performance tuning.
• Familiarity with SQL and NoSQL databases.
• Proficient in Python and intermediate knowledge of Java.
• Experience with Git and CI/CD pipelines.

Nice-to-Have Skills:

• Hands-on experience with Apache NiFi.
• Real-time streaming pipeline development experience.
• Exposure to cloud platforms like AWS, Azure, or GCP.
• Familiarity with Docker or Kubernetes.

Soft Skills:

• Strong analytical and problem-solving capabilities.
• Excellent communication and collaboration skills.
• Self-motivated with the ability to work both independently and in teams.