Python + Data Engineer

Key Responsibilities:

• Data Pipeline Design & Development:
• Design, develop, and optimize data pipelines using Python and core AWS services such as Glue, Lambda, S3, EMR, Redshift, Athena, and Kinesis.
• Implement ETL/ELT processes to extract, transform, and load data from multiple sources into centralized repositories (e.g., data lakes or data warehouses).
• Collaboration & Requirement Gathering:
• Collaborate with cross-functional teams to understand business requirements and translate them into scalable data solutions.
• Monitoring & Troubleshooting:
• Monitor, troubleshoot, and enhance data workflows for performance and cost optimization.
• Data Quality & Governance:
• Ensure data quality and consistency by implementing validation and governance practices.
• Security & Compliance:
• Work on data security best practices in compliance with organizational policies and regulations.
• Automation:
• Automate repetitive data engineering tasks using Python scripts and frameworks.
• CI/CD Pipelines:
• Leverage CI/CD pipelines for deployment of data workflows on AWS.

Required Skills & Qualifications:

• Professional Experience:
• 5+ years of experience in data engineering or a related field.
• Programming Skills:
• Strong proficiency in Python, with experience in libraries such as pandas, pyspark, and boto3.
• AWS Expertise:
• Hands-on experience with core AWS services for data engineering:
• AWS Glue for ETL/ELT processes.
• S3 for storage.
• Redshift or Athena for data warehousing and querying.
• Lambda for serverless compute.
• Kinesis or SNS/SQS for data streaming.
• IAM Roles for security.
• Database Skills:
• Proficiency in SQL and experience with relational (e.g., PostgreSQL, MySQL) and NoSQL (e.g., DynamoDB) databases.
• Data Processing Frameworks:
• Knowledge of big data frameworks such as Hadoop and Spark is a plus.
• DevOps & CI/CD:
• Familiarity with CI/CD pipelines and tools like Jenkins, Git, and CodePipeline.
• Version Control:
• Proficient with Git-based workflows.
• Problem-Solving:
• Excellent analytical and debugging skills.