T
Data Engineer
T
Data Engineer
Tata Consultancy Services- Posted 2 hours ago
- Be among the first 10 applicants
Job Description
Job Description: AWS PySpark Data Engineer
Job Title: AWS PySpark Data Engineer
Experience:5+ Years
Location:Indore
Job Summary
We are looking for an experienced AWS PySpark Data Engineer with strong hands-on expertise in building scalable data engineering solutions on AWS. The role will focus on developing and optimizing ETL/ELT pipelines, processing large-scale datasets using PySpark/Spark, and building cloud-native data solutions using AWS services.
Mandatory Skills
- 5+ years of overall experience
- PySpark / Apache Spark
- Python
- SQL
- AWS Glue
- Amazon S3
- AWS EMR
- AWS Lambda
- Amazon Redshift
- ETL/ELT development
- Data Lake/Data Warehouse concepts
- Large-scale/distributed data processing
Key Responsibilities
- Design, develop and maintain scalable ETL/ELT pipelines using PySpark.
- Build data ingestion and transformation frameworks on AWS.
- Process large volumes of structured, semi-structured and unstructured data.
- Develop and optimize Spark/PySpark workloads.
- Build cloud-native Data Lake and Data Warehouse solutions.
- Implement data quality, validation, monitoring and governance.
- Monitor and troubleshoot production data pipelines.
- Participate in CI/CD, deployments and Agile delivery.
- Work closely with data architects, analytics teams and business stakeholders.
AWS Technology Stack
Core: S3, Glue, EMR, Lambda, Redshift
Additional: Athena, IAM, CloudWatch, Step Functions
Good-to-Have
- Databricks
- Apache Kafka
- Apache Airflow
- Snowflake
- Delta Lake
- Terraform
- Docker/Kubernetes
- Streaming data pipelines
More Info
Key Skills
AWS EMR
Large-scale distributed data processing
About Company
Skills:
Hadoop, Power Bi, Pyspark, Sql, Big Data Technologies, Numpy, Pandas, Gcp, Spark, Databricks, Azure, Python, Etl, AWS
Skills:
BigQuery, Gcp, Pyspark, Data Quality, Databricks, Advanced Sql, Etl, ELT, Delta Lake, Performance optimization, Data pipeline design
Skills:
composer , Google Cloud Platform (GCP), BigQuery, Csv, Schema Design, Data Modeling, Dataproc, Json, Data Warehousing, Avro, Sql, ELT, Cloud Storage, DataFlow, Python, Etl, Parquet, Pub Sub
Skills:
data reconciliation , data deduplication , snowflake , Databricks, Sql, Data Warehousing, Pyspark, Kafka, Etl, Gcp, Terraform, ELT, Apache Spark, AWS, Kubernetes, Python, Azure, Data Transformation, Docker, Data Modelling, Data Quality Validation, Airflow, dbt, Medallion Architecture, Data Accuracy, Data Consistency, Data Completeness
RE : Openings for Data Engineer – AI Implementation - Contract to Hire Position - RemoteCareer Path Solutions Private Limited
C
Skills:
data engineering , Python, Sql, Etl, ELT, Data Cleaning, Data Transformation, Api Integration, Json, MongoDB, Neo4j, Ocr, Data Quality, Data Pipelines, Data Validation, RAG, Retrieval Augmented Generation, Vector Database, Qdrant, Embeddings, semantic search, Knowledge Graph, Cypher, Document Processing, Document Chunking, Metadata Extraction, Pipeline Monitoring