I
AWS Data Engineer
I
- Posted 11 hours ago
- Be among the first 20 applicants
Job Description
Responsibilities and JD in brief along with additional criteria to be considered:
- Design and develop scalable AWS Data Engineering solutions and cloud-native data platforms.
- Build batch and real-time data pipelines using AWS Glue, PySpark, Databricks, S3, Kinesis , Redshift and Kafka/MSK.
- Develop and optimize Databricks, Delta Lake and Spark workloads for large-scale data processing with Pyspark , Databricks transformations.
- Implement streaming solutions using Amazon Kinesis Data Streams, Firehose and Analytics for near real-time data ingestion and processing.
- Handle high-volume (TB/PB scale) data pipelines with a focus on performance, scalability and cost optimization.
- Develop complex SQL transformations, data models and analytics-ready datasets.
- Implement CI/CD, Infrastructure as Code (Terraform/CloudFormation) and DevOps best practices.
- Ensure data quality, governance, security, monitoring and production support.
- Collaborate with architects, stakeholders and cross-functional teams in Agile delivery environments.
- Troubleshoot and optimize end-to-end AWS data platforms, streaming pipelines and cloud infrastructure
- Strong hands-on AWS Data Engineering experience across large-scale cloud data platforms.
- Expertise in AWS S3, AWS Glue, AWS Lambda, AWS Redshift, AWS Lake Formation, AWS CloudWatch and AWS IAM.
- Strong experience in real-time and streaming data processing using Amazon Kinesis Data Streams, Kinesis Firehose, Kinesis Analytics, Amazon Kafka or equivalent streaming technologies.
- Hands-on coding proficiency in Python, PySpark and SQL with strong optimization skills.
- Experience building high-volume, low-latency data ingestion and transformation pipelines.
- Strong experience in Databricks, Delta Lake, Structured Streaming and Spark Performance Tuning.
- Experience handling TB/PB-scale datasets, partitioning strategies, file optimization, schema evolution and data lifecycle management.
- Strong understanding of Data Lake, Lakehouse and Modern Data Platform architectures.
- Practical data modeling experience for analytical, operational and reporting workloads.
- Experience implementing CI/CD pipelines using AWS CodePipeline, CodeBuild, CodeDeploy, GitHub Actions or Jenkins.
- Strong troubleshooting skills for production issues, performance bottlenecks and cost optimization.
- Excellent client communication, stakeholder management and problem-solving skills.
More Info
Key Skills
AWS Lake Formation
CodeBuild
GitHub Actions
Amazon Kinesis Data Streams
Kafka MSK
CodeDeploy
Databricks Delta Lake
Firehose
About Company
Skills:
data engineering , Metadata Management, Aws Redshift, AWS Glue, Data Governance, Etl, ELT, Data Cataloging
Skills:
cloud platform , AWS - Database Technologies, Cloud, AWS, Technology, AWS Core services, AWS Data Analytics
Skills:
AWS, Apache Spark, Machine Learning, Node.js, Natural Language Processing, Data mining, Python, Technical computing tools, Container orchestration, Managed cloud DB
Skills:
Lambda, S3, Pyspark, Python, AWS, Glue