Senior Data Engineer

Responsibilities

• Design, develop, and maintain data pipelines, platform components, and data-intensive services.
• Own data engineering work from requirements and technical design through implementation, deployment, and production support.
• Contribute to data acquisition strategies, dataset contracts, acceptance criteria, quality standards, lineage, and lifecycle policies.
• Build scalable batch and real-time data pipelines, lakehouse architectures, and distributed processing solutions for high-volume workloads.
• Help evolve the data ecosystem to support growth in data volume, velocity, variety, and product complexity.
• Apply and improve reusable patterns for data ingestion, transformation, storage, orchestration, testing, observability, and delivery.
• Build production systems that meet expectations for reliability, security, performance, scalability, cost efficiency, and SLA-driven delivery.
• Contribute to operational practices for monitoring, incident response, failure recovery, replay, backfills, schema evolution, and disaster recovery.
• Partner with Data Science and ML teams to productionize ML, AI, and LLM-based workloads and integrate them into customer-facing products.
• Collaborate with Cloud, DevOps, Application, Security, Product, and client teams to deliver robust solutions for business and customer needs.
• Evaluate technologies and architectural patterns, contribute to technical proofs of concept, and recommend adoption based on measurable value.
• Document technical decisions, trade-offs, risks, and implementation approaches.
• Participate in technical reviews and help improve design, code quality, testing, maintainability, and production ownership.
• Support engineering roadmaps, hiring, and capability development across the data engineering discipline.

• 7+ years of hands-on software or data engineering experience, including experience designing, building, and operating production-grade data systems.
• Experience contributing to complex data initiatives across teams.
• B.Sc. or M.Sc. in Computer Science, Computer Engineering, or an equivalent technical field, or comparable practical experience.
• Strong understanding of data-intensive and distributed systems, including scalability, reliability, consistency, and performance trade-offs.
• Strong proficiency in Python and working proficiency in Scala.
• Strong expertise with at least one major cloud platform: AWS, Microsoft Azure, or Google Cloud Platform.
• Experience with large-scale data processing technologies such as Apache Spark and Databricks.
• Experience designing and operating event-driven or streaming systems using technologies such as Apache Kafka.
• Experience with workflow orchestration and platform tooling such as Apache Airflow, including maintainable DAGs and operational business-critical workflows.
• Advanced SQL skills and strong experience with data modeling, query optimization, and data storage systems.
• Hands-on experience with at least one of the following: data lake or lakehouse architectures; columnar or analytical databases; or NoSQL or distributed database systems.
• Experience owning production-grade batch and/or streaming pipelines, including data quality, observability, schema evolution, backfills, recovery procedures, and SLA management.
• Hands-on experience designing, developing, deploying, and operating production-grade microservices.
• Hands-on experience with containers, Kubernetes, and modern CI/CD practices; experience with Helm is an advantage.
• Familiarity with LLM application and workflow frameworks such as LangChain or LangGraph.
• Ability to evaluate technologies, make sound technical decisions, and communicate trade-offs to engineering and non-engineering stakeholders.
• Strong collaboration, mentoring, and technical communication skills.
• Ability to work effectively in a dynamic, fast-paced environment while balancing platform improvements with near-term product delivery.
• Pragmatic, results-driven approach and commitment to strong engineering fundamentals.
• Strong written and verbal communication skills in English.

Nice to Have

• Experience with Delta Lake and/or Apache Iceberg; ML lifecycle tools such as MLflow.
• Experience with Pandas/Polars and building data services/APIs (e.g., FastAPI).
• Experience building LLM-powered agents or chat assistants (RAG, tool/function calling, workflow or multi-agent orchestration) using modern frameworks and platforms.
• Infrastructure as Code (Terraform/Pulumi/CloudFormation) and cloud security fundamentals (IAM, secrets, encryption).
• Experience with observability tooling (metrics/logging/tracing) and cost/performance optimization for distributed workloads.
• Experience building applications with React and Node.js.