Big Data Engineer
Purview India Consulting And Services Llp- Posted a month ago
- Over 200 applicants have applied
Job Description
We are seeking a skilled Big Data Engineer to join our team in India. The ideal candidate will have extensive experience in designing and implementing robust big data solutions that support our data-driven decision-making processes.
Job description
- Design, build, and maintain scalable ETL/ELT pipelines using AWS Glue, Lambda, Step Functions, and Apache Spark.
- Develop and optimize data lake and data warehouse architectures using AWS S3, Redshift, and Athena.
- Implement data ingestion, transformation, and storage solutions using AWS Glue, Kinesis, and Kafka.
- Work with structured and unstructured data to develop efficient data pipelines.
- Optimize query performance in SQL-based and NoSQL databases (Redshift, DynamoDB, Aurora, PostgreSQL, etc.).
- Ensure data security, compliance, and governance using IAM roles, KMS encryption, and AWS Lake Formation.
- Automate data workflows using Terraform, CloudFormation, or CDK.
- Monitor and troubleshoot data pipelines, ensuring high availability and performance.
- Collaborate with Data Scientists, Analysts, and Engineers to provide robust data solutions.
Required Skills:
- Strong experience with AWS Data Services (Glue, Redshift, S3, Lambda, Kinesis, Athena).
- Proficiency in Python, SQL, and Scala
- Experience with ETL/ELT pipeline design and data warehousing concepts.
- Strong knowledge of relational and NoSQL databases (PostgreSQL, MySQL, DynamoDB).
- Familiarity with DevOps practices and CI/CD tools (Jenkins, GitHub Actions).
- Experience with Kafka or Kinesis for real-time data streaming.
- Knowledge of Terraform, CloudFormation, or CDK for infrastructure automation.
- Hands-on experience with data security and governance.
- Strong analytical and problem-solving skills
More Info
About Company
Purview is a leading Digital Cloud & Data Engineering company headquartered in Edinburgh, United Kingdom having a presence in 14 countries India (Hyderabad, Bangalore, Chennai and Pune), Poland, Germany, Finland, Netherlands, Ireland, USA, UAE, Oman, Singapore, Hong Kong, Malaysia and Australia.
We have a strong presence in UK, Europe and APEC, providing services to Captive Clients (HSBC, NatWest, Northern Trust, IDFC First Bank, Nordia Bank etc) in fully managed solutions and co-managed capacity models. Also, we support various top IT tier 1 organisations (Capgemini, Deloitte, Wipro, Virtusa, L&T, CoForge, TechM and more) to deliver solutions and workforce/resources.
