Search by job, company or skills

go digital technology consulting llp

Lead Data Engineer

go digital technology consulting llp
Mumbai, India
7-9 Years
Save
  • Posted 2 months ago
  • Be among the first 10 applicants
Early Applicant

Job Description

Technologies / Skills

Advanced SQL, Python and associated libraries like Pandas, Numpy etc., Pyspark , Shell scripting, DataModelling, Big data, Hadoop , Hive, ETL pipelines and IaC tools like Terraform etc.

Responsibilities

  • Proven success in communicating with users, other technical teams, and senior management to
  • Collect requirements, describe data modeling decisions and develop data engineering strategy.
  • Ability to work with business owners to define key business requirements and convert to user stories with required technical specifications.
  • Communicate results and business impacts of insight initiatives to key stakeholders to collaboratively solve business problems.
  • Working closely with the overall Enterprise Data & Analytics Architect and Engineering practice leads to ensure adherence with the best practices and design principles.
  • Assures quality, security and compliance requirements are met for supported area.
  • Design and create fault-tolerance data pipelines running on cluster
  • Excellent communication skills with the ability to influence client business and IT teams
  • Should have design data engineering solutions end to end. Ability to come up with scalable and modular solutions

Required Qualification

  • 7 years of relevant IT experience
  • 4+ years of hands-on experience Designing and developing Data Pipelines for Data Ingestion or Transformation using Python (PySpark)/Spark SQL in AWS cloud
  • Experience in design and development of data pipelines and processing of data at scale using technologies like EMR, Kinesis, Kafka, Lambda, Glue, Athena, Redshift, Step Functions.
  • Advanced experience in writing and optimizing efficient SQL queries with Python and Hive handling Large Data Sets in Big-Data Environments
  • Experience in debugging, tunning and optimizing PySpark data pipelines
  • Should have implemented concepts and have good knowledge of Pyspark dataframes, joins, caching, memory management, partitioning, parallelism etc.
  • Understanding of Spark UI, Event Timelines, DAG, Spark config parameters, in order to tune the long running data pipelines.
  • Experience working in Agile implementations
  • Experience with building data pipelines in streaming and batch mode.
  • Experience with Git and CI/CD pipelines to deploy cloud applications
  • Good knowledge of designing Hive tables with partitioning for performance

Desired Qualification

  • Experience in data modelling
  • Hands on creating workflows on any Scheduling Tool like Autosys, CA Workload Automation
  • Proficiency in using SDKs for interacting with native AWS services
  • Strong understanding of concepts of ETL, ELT and data modeling

More Info

Job Type:
Permanent Job
Industry:
Other
Function:
Data Engineering
Employment Type:
Full time

Job ID: 146667243

Jobs by Skill - IT
Jobs by Skill - Non IT
International Jobs
Last Updated: 23-06-2026 02:04:37 AM
Homejobs in MumbaiLead Data Engineer

Similar Jobs

Lead Data Engineer

LearningMate
Mumbai, India

Skills:

Aws Rds, Aws Redshift, Ms Sql Server, Pyspark, PostgreSQL, AWS Glue, Dataproc, DataFlow, Sql, GCP BigQuery, Google Cloud Storage

Lead Data Engineer- Vice President - Data Engineering

morgan stanley
Mumbai, India

Skills:

snowflake , Automated Testing, Data Modeling, Sql, Testing Frameworks, Agile Methodologies, Etl, GenAI, building scalable data pipelines, AI-based solutions, modern data engineering tools, cloud data platforms

Lead Data Engineer

Media.net
Mumbai, India

Skills:

Python, Sql, Java, Presto, Hive, Cassandra, Nosql, Big Data, Apache Kafka, Scala, MongoDB, Spark, HDFS

Lead Software Engineer - Big Data

Jpmorgan & Co
Mumbai, India

Skills:

Oracle Sql, Maven, Kafka, Spring Boot, Artifactory, Avro, Nosql, Swagger, Cicd, Big Data Technologies, Java, Yaml, Scala, Dynamodb, Impala, Pig, Junits, Jenkins, Git, Hive, Spark, Data Lake, Api, Data Warehousing, orc, Sequence files, AI-assisted software development tools, Parquet

Lead Data Engineer(Snowflake/DBT)

Tide
Remote

Skills:

Git, Sql, Aws, dbt