Big Data Engineer

We are seeking a skilled Big Data Engineer to join our team in India. The ideal candidate will have extensive experience in designing and implementing robust big data solutions that support our data-driven decision-making processes.

Job description

• Design, build, and maintain scalableETL/ELTpipelines usingAWS Glue, Lambda, Step Functions, and Apache Spark.
• Develop and optimizedata lakeanddata warehousearchitectures usingAWS S3, Redshift, and Athena.
• Implementdata ingestion, transformation, and storage solutionsusingAWS Glue, Kinesis, and Kafka.
• Work with structured and unstructured data to develop efficientdata pipelines.
• Optimize query performance inSQL-basedandNoSQLdatabases (Redshift, DynamoDB, Aurora, PostgreSQL, etc.).
• Ensure data security, compliance, and governance usingIAM roles, KMS encryption, and AWS Lake Formation.
• Automate data workflows usingTerraform, CloudFormation, or CDK.
• Monitor and troubleshoot data pipelines, ensuring high availability and performance.
• Collaborate withData Scientists, Analysts, and Engineersto provide robust data solutions.

Required Skills:

• Strong experience withAWS Data Services(Glue, Redshift, S3, Lambda, Kinesis, Athena).
• Proficiency inPython, SQL, and Scala
• Experience withETL/ELT pipeline designanddata warehousingconcepts.
• Strong knowledge ofrelational and NoSQL databases(PostgreSQL, MySQL, DynamoDB).
• Familiarity withDevOps practicesandCI/CD tools(Jenkins, GitHub Actions).
• Experience withKafka or Kinesisfor real-time data streaming.
• Knowledge ofTerraform, CloudFormation, or CDKfor infrastructure automation.
• Hands-on experience withdata security and governance.
• Strong analytical and problem-solving skills