Company Description
ThreatXIntel is a startup cyber security company specializing in protecting businesses and organizations from cyber threats. Our services include cloud security, web and mobile security testing, cloud security assessment, DevSecOps, and more. We offer customized and affordable solutions to meet the specific needs of our clients, regardless of their size, so they can focus on growing their business.
Role Description
We are looking for a freelance Python Data Engineer with strong experience in building and managing data pipelines on AWS. The ideal candidate will have hands-on expertise in AWS services like Glue, Lambda, Step Functions, S3, and a solid foundation in CI/CD using Jenkins. Experience with testing strategies, especially regression testing and managing live dependencies, is essential.
Key Responsibilities:
- Design, develop, and deploy scalable data pipelines using AWS Glue, Lambda, and Step Functions.
- Integrate and manage data flows across S3, RDS, and other AWS data sources.
- Implement Python-based ETL logic for data transformation and aggregation.
- Set up and maintain CI/CD pipelines using Jenkins for automated deployment.
- Conduct regression testing to ensure stability of pipelines and handle live data dependencies.
- Optimize data workflows for performance and cost-efficiency.
- Monitor production jobs, troubleshoot issues, and ensure high availability.
Required Skills:
- Strong proficiency in Python (especially for data engineering and automation).
- Proven experience with AWS Glue, Lambda, Step Functions, and S3.
- Hands-on with CI/CD tools, especially Jenkins.
- Experience in writing test cases, performing regression testing, and managing live production dependencies.
- Familiarity with data formats like Parquet, JSON, CSV and data cataloging.
- Good understanding of cloud cost management and resource optimization.
Preferred Qualifications:
- Experience in working with AWS CDK or CloudFormation is a plus.
- Knowledge of data lake or lakehouse architecture.
- Familiar with agile development and DevOps practices.
