Software Engineer

JD :-

Experience: 5+ years of software development experience with at least 2–3 years building Python-based data processing pipelines and production-grade workers (serverless or containerized). Experience with media/document processing and embedding/vector workflows is strongly preferred.

The Python Worker Developer will design, build, and operate a serverless ingestion/processing worker triggered by AWS SQS to enrich incoming media metadata. The worker will fetch job details from Supabase Postgres, perform video analysis via Thalamus, extract structured data from documents using Docling, generate embeddings using Google Gemini, and persist canonical metadata and vector embeddings (pgvector) back into Supabase. This role requires strong engineering fundamentals, reliability mindset (idempotency/retries), and hands-on experience with Cloud Run

Responsibilities

Develop and maintain Python Lambda worker(s)

Implement job lifecycle handling by fetching job payloads from Postgres SQL using JobId and updating job status (QUEUED/RUNNING/COMPLETED/FAILED).

Implement embedding generation using Google Gemini (text and/or multimodal embeddings) and manage model/version configuration.

Design chunking strategies for transcripts/credits/metadata (timecode-aware, page-aware, source-aware) to optimize semantic search quality.

Persist results into Supabase Postgres canonical tables and pgvector tables for semantic retrieval.

Implement robust idempotency and deduplication controls to avoid duplicate inserts (e.g., chunk hash + model version).

Build resilient error handling with retries/backoff, DLQ-safe behavior, and partial-failure recovery patterns.

Optimize runtime performance within Lambda constraints (cold start mitigation, batching, connection pooling).

Implement structured logging, metrics, and tracing to support observability and incident response.

Write unit and integration tests for core extraction/embedding logic and DB persistence behavior.

Participate in code reviews, sprint planning, estimation, and release activities.

Maintain clear documentation for worker workflows, configuration, and operational runbooks.

Technical Skills

Strong Python proficiency (3.10+) with production coding standards and async/concurrency fundamentals.

Solid experience with PostgreSQL and writing reliable ingestion/upsert logic; familiarity

Exper