Search by job, company or skills

Virtusa

AWS data engineer

6-12 Years
new job description bg glownew job description bg glownew job description bg svg
  • Posted a day ago
  • Over 50 applicants
Quick Apply

Job Description

  • Develop and implement efficient data pipelines using Apache Spark (PySpark preferred) to process and analyze large-scale data.
  • Design, build, and optimize complex SQL queries to extract, transform, and load (ETL) data from multiple sources.
  • Orchestrate data workflows using Apache Airflow, ensuring smooth execution and error-free pipelines.
  • Design, implement, and maintain scalable and cost-effective data storage and processing solutions on AWS using S3, Glue, EMR, and Athena.
  • Leverage AWS Lambda and Step Functions for serverless compute and task orchestration in data pipelines.
  • Work with AWS databases like RDS and DynamoDB to ensure efficient data storage and retrieval.
  • Monitor data processing and pipeline health using AWS CloudWatch and ensure smooth operation in production environments.
  • Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
  • Perform performance tuning, optimize distributed data processing tasks, and handle scalability issues.
  • Provide troubleshooting and support for data pipeline failures and ensure high availability and reliability.
  • Contribute to the setup and maintenance of CI/CD pipelines for automated deployment and testing of data workflows.

Required Skills & Experience:

  • Experience: Minimum of 6+ years of hands-on experience in data engineering or big data development roles, with a focus on designing and building data pipelines and processing systems.
  • Technical Skills:
  • Strong programming skills in Python with hands-on experience in Apache Spark (PySpark preferred).
  • Proficient in writing and optimizing complex SQL queries for data extraction, transformation, and loading.
  • Hands-on experience with Apache Airflow for orchestration of data workflows and pipeline management.
  • In-depth understanding and practical experience with AWS services:
  • Data Storage & Processing: S3, Glue, EMR, Athena
  • Compute & Execution: Lambda, Step Functions
  • Databases: RDS, DynamoDB
  • Monitoring: CloudWatch
  • Experience with distributed data processing, parallel computing, and performance tuning techniques.
  • Strong analytical and problem-solving skills to troubleshoot and optimize data workflows and pipelines.
  • Familiarity with CI/CD pipelines and DevOps practices for continuous integration and automated deployments is a plus.

Preferred Qualifications:

  • Familiarity with other cloud platforms (Azure, Google Cloud) and services related to data engineering.
  • Experience in handling unstructured and semi-structured data and working with data lakes.
  • Knowledge of containerization technologies such as Docker or orchestration systems like Kubernetes.
  • Experience with NoSQL databases or data warehouses like Redshift or BigQuery is a plus.

Qualifications:

  • Education: Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
  • Experience: Minimum of 6+ years in a data engineering role with strong expertise in AWS and big data processing frameworks.

More Info

Job Type:
Employment Type:
Open to candidates from:
Indian

About Company

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.
Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.
Virtusa is an Equal Opportunity Employer. All applicants will receive fair and impartial treatment without regard to race, color, religion, sex, national origin, ancestry, age, legally protected physical or mental disability, protected veteran status, status in the U.S. uniformed services, sexual orientation, gender identity or expression, marital status, genetic information or on any other basis which is protected under applicable federal, state or local law.
Applicants may be required to attend interviews in person or by video conference. In addition, candidates may be required to present their current state or government-issued ID during each interview. All candidates must be authorized to work in the USA.

Job ID: 118076125

Similar Jobs

Early Applicant
Early Applicant