Data Engineer

We're seeking an experienced Data Engineer to help build and scale our next-generation AI-powered marketing personalization platform (V2.0). You'll design and implement a robust multi-database infrastructure that enables real-time personalization, vector search, graph analytics, and large-scale data processing.

This is a greenfield opportunity to architect data pipelines from the ground up using vector and graph databases and LLM-based systems. You'll play a key role in migrating our existing platform while creating a scalable foundation powering AI agents across thousands of marketing campaigns.

Data Architecture & Infrastructure (40%)

• Design and implement multi-database systems (MongoDB, Redis, Milvus, Neo4j, BigQuery)
• Build scalable real-time pipelines and ETL/ELT workflows
• Implement data governance, quality, and high-throughput optimization

Vector & Graph Systems (25%)

• Optimize Milvus collections for semantic search
• Design Neo4j schemas for customer journeys and relationships
• Develop hybrid search (vector + graph + text) with performance tuning

ML Data Infrastructure (20%)

• Build data pipelines for LLM fine-tuning and GNN training
• Manage data versioning, lineage, and A/B testing systems
• Implement real-time feature computation for contextual models

Analytics & Monitoring (15%)

• Create BigQuery schemas and real-time dashboards
• Implement observability (Prometheus, Grafana) and anomaly detection
• Optimize data costs across cloud environments

Databases: MongoDB, Redis, Milvus, Neo4j, BigQuery

Processing: Airflow/Prefect, Pandas/Polars, dbt, Spark

ML Pipeline: vLLM, MLflow, Sentence Transformers, PyTorch, TensorFlow

Cloud & Infra: GCP, Docker, Kubernetes, Terraform, GitHub Actions

Languages: Python (3.10+), SQL, Bash

Must-Have

• 5+ years of data engineering experience in production systems
• Advanced Python and SQL expertise
• Experience with 3+ database types (SQL, NoSQL, Vector, Graph)
• Proven ability to build high-scale data pipelines (>1M records/day)
• Strong data modeling, validation, and optimization skills
• Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake)

Preferred

• Experience with Milvus, Pinecone, or Weaviate
• Graph databases (Neo4j, Neptune) and embedding-based search
• ML/Feature store experience
• Background in marketing technology or CDPs

Phase 1 – Foundation: Migrate 10M+ vectors, implement MongoDB schemas, Neo4j models, and BigQuery warehouse

Phase 2 – Optimization: Build data quality monitoring, caching (Redis), and <50ms vector query latency

Phase 3 – ML Infrastructure: Create LLM fine-tuning pipelines, GNN feature stores, and A/B testing systems

• Collaborate with ML engineers and data scientists on cutting-edge AI systems
• High ownership and real product impact
• Modern tools and a flexible environment
• Clear growth path to Senior, Lead, or Principal roles
• Shape the future of AI-driven marketing personalization