Big Data Engineer(AWS, Python,Airflow)

Job Description – AWS / Python / Airflow / Big Data Engineer

Role: AWS / Python / Airflow / Big Data Engineer

Experience: 3–5 Years

Job Overview

We are seeking two skilled AWS/Python/Airflow/Big Data Engineers with 3–5 years of hands-on experience to join our data engineering team. The ideal candidates will be responsible for designing, developing, and maintaining scalable data pipelines and workflows in a cloud-based AWS environment. This role requires strong expertise in Python, Apache Airflow, AWS services, and Big Data technologies.

Key Responsibilities

• Design, develop, and maintain ETL and data pipelines using Apache Airflow.
• Build and manage workflow orchestration (DAGs) for reliable data processing.
• Implement scalable and cost-efficient solutions on AWS Cloud, leveraging:
• S3, EC2, Lambda, EMR, Glue, Redshift
• Write efficient, reusable, and maintainable Python code for data processing and automation.
• Work with Big Data frameworks such as Apache Spark and Hadoop for large-scale data processing.
• Optimize data workflows for performance, scalability, and cost efficiency.
• Collaborate with cross-functional teams (data analysts, product, DevOps) to understand data requirements and deliver solutions.
• Ensure data quality, reliability, security, and monitoring across all pipelines and workflows.
• Troubleshoot and resolve data pipeline failures and performance bottlenecks.
  

Required Qualifications

Technical Skills

• AWS Cloud Services: Hands-on experience with core AWS services used in data engineering.
• Python Programming: Strong coding skills with experience using libraries such as Pandas, PySpark, and automation scripts.
• Apache Airflow: Proven experience building and managing Airflow DAGs and workflows.
• Big Data Technologies: Familiarity with Spark, Hadoop, or similar distributed data processing frameworks.
• Data Modeling & ETL: Solid understanding of ETL processes, data transformations, and pipeline design.
• Databases & SQL: Working knowledge of SQL and relational databases.
  
  

Soft Skills

• Strong problem-solving and debugging abilities.
• Excellent communication and collaboration skills.
• Ability to work independently and in a fast-paced team environment.
  
  

Preferred Qualifications

• Experience with CI/CD pipelines and DevOps practices.
• Familiarity with containerization technologies such as Docker and Kubernetes.
• Exposure to cloud monitoring, logging, and performance optimization tools.
  
  

Skills: python,pipelines,big data,cloud,aws,airflow