Data Engineer

Key Skills: Pyspark, AWS, Airflow, Terraform, Python, Kafka, Fast Api

Roles and Responsibilities:

• Own end-to-end ETL and feature engineering pipelines using Python, PySpark, and SQL.
• Develop and maintain lightweight FastAPI endpoints for feature serving, validation, health checks, and internal data services.
• Extend and integrate with existing codebases and frameworks, delivering incremental improvements.
• Containerize applications and coordinate deployments on AWS using Terraform and CI/CD pipelines.
• Ensure secure handling of secrets, basic observability, and production reliability.
• Collaborate with stakeholders to understand requirements, clarify ambiguities, and propose practical solutions.
• Communicate trade-offs, assumptions, and technical decisions effectively.

Skills Required:

• Strong proficiency in Python for data engineering and backend services.
• Hands-on experience with PySpark for large-scale data processing.
• Experience working with Kafka for real-time data streaming.
• Solid understanding of AWS cloud services for data platform deployment.
• Experience using Airflow for workflow orchestration and scheduling.
• Proficiency in Infrastructure as Code using Terraform.
• Ability to build and maintain RESTful APIs using FastAPI is an added advantage.
• Good understanding of ETL processes, data modeling, and feature engineering.
• Experience with containerization and CI/CD pipelines for deployment automation.
• Strong problem-solving skills and ability to work with evolving requirements.

Education: Bachelor's degree in Computer Science, Engineering, or a related field.