Search by job, company or skills

Talentiser

Lead Big Data Engineer

Talentiser
Bengaluru, India
9-11 Years
new job description bg glownew job description bg glownew job description bg svg
  • Posted 17 hours ago
  • Be among the first 10 applicants
Early Applicant

Job Description

  • BE/MTech in Computer Science or an equivalent professional experience
  • 9+ years of design, architecture, and development experience, tackling complex problems in largescale data pipelines
  • Solid foundation in Data Structures, Algorithms, Object-Oriented Programming, and Software Design
  • Architectural expertise in data modeling for productiongrade batch and streaming processing systems
  • Deep understanding of Spark-based processing with focus on resource optimization
  • Practical understanding of Airflow for orchestration and Kafka for streaming
  • Solid foundation in distributed systems: consistency, reliability, fault tolerance, retries, circuit breakers, and timeouts
  • Production experience with CI/CD (e.g., GitHub Actions/Jenkins), containers (Docker), Kubernetes, and infrastructure-as-code (Helm/Terraform)
  • Hands-on experience integrating LLM calls in data pipelines: prompt orchestration, batching, rate limiting, guardrails, output validation
  • Exposure to embedding generation and vector indexing as part of data processing pipelines.
  • Programming experience in Python (Spark). Strong SQL and exposure to at least one cloud
  • Develop batch and streaming ETL/ELT pipelines across APIs, databases, files, and event streams
  • Use SQL and optimized Spark pipelines to transform raw data into clean, standardized, query-ready datasets
  • Build reusable data marts and feature sets for downstream teams (analytics, ML, product)
  • Tune queries, partitioning, clustering, indexing, and storage formats (Parquet/ORC)
  • Optimize compute and storage costs; manage scaling strategies and right-size resources
  • Implement CI/CD for data code and pipelines; manage environments and releases
  • Translate business needs into technical specifications; document datasets, SLAs, and usage guidelines
  • Support incident response and root-cause analysis for data quality issues
  • Partner with analytics, ML, engineering, and product teams to define data requirements
  • Mentor junior engineers and contribute to engineering best practices
  • Drive architectural decisions and influence long-term data strategy

More Info

Job Type:
Permanent Job
Industry:
Other
Function:
Data Engineering
Employment Type:
Full time

About Company

Talentiser

Job ID: 144183041

Jobs by Skill - IT
Jobs by Skill - Non IT
International Jobs
Last Updated: 12-03-2026 06:52:37 PM
Homejobs in Bengaluru / BangaloreLead Big Data Engineer