Senior Data Engineer

Senior Data Engineer to build the pipelines that turn raw multimodal capture — audio, egocentric video and sensor streams from 20+ countries — into versioned, quality-checked, client-ready datasets for frontier AI and robotics teams.

Location: Bengaluru, Karnataka, India · Employment: full-time, on-site · Experience: 5+ years

WHAT YOU WILL DO

Design and build batch and streaming pipelines that ingest raw audio, video and sensor (IMU) data from the global capture network into cloud storage. • Build processing stages for unstructured media: transcoding, segmentation, metadata extraction, deduplication and joins with annotation and contributor data. • Own data quality: automated validation, QC gates and anomaly detection so that bad captures are caught before they reach a client. • Build dataset versioning, lineage and manifests so any delivered dataset can be reproduced exactly. • Package and deliver datasets in the formats client teams ask for (Parquet, WebDataset, Hugging Face datasets, MCAP and similar), on time and to specification.

WHAT YOU NEED

5+ years of data engineering, building production pipelines end to end

Strong Python and SQ

• Distributed processing at scale — Spark, Ray, Beam or Dask

• Workflow orchestration (Airflow, Dagster or Prefect) and cloud object storage on AWS or GCP

• Has worked with large unstructured or media data (audio, video, images, sensor logs), not only tabular warehouse data