Senior Specialist - System Management

Elasticsearch Lead CDP GCP Agile

Experience

814 years overall with 5 years handson Elasticsearch architecture operations and 2 years leading delivery teams

We are looking for an Elasticsearch Lead to design build and run search and analytics capabilities for a Consumer Data Platform CDP on Google Cloud Platform GCP This role will lead the Elasticsearch engineering and operational strategycovering cluster architecture indexing strategy ingestion pipelines performance tuning observability security and reliabilitywhile working closely with product owners data engineering platformSRE and application teams in an Agile ScrumKanban delivery model

The ideal candidate has strong experience implementing Elasticsearch for highvolume customerevent data enabling near realtime search segmentation identity resolution support and analytics use cases typical of CDP ecosystems

Key Responsibilities

1 Architecture Solution Design

Own endtoend Elasticsearch architecture for CDP use cases search analytics segmentation customer 360 support behavioralevent search

Define index strategy mapping shardreplica strategy ILM policies rollover patterns data tiering and lifecycle design

Create reference architectures for Elasticsearch on GCP compute storage networking security DR

Lead design reviews and ensure alignment with enterprise standards security privacy scalability resiliency

2 Delivery Leadership Agile

Lead Elasticsearch workstream in Agile squads planning refinement estimation sprint execution demos

Translate product requirements into technical stories acceptance criteria and measurable outcomes latency throughput relevancy

Mentor engineers and review codeconfigurations pipelines mappings queries templates dashboards

Coordinate crossteam dependencies Data Engineering PlatformSRE API teams IAMSecurity Governance

3 Implementation Engineering

Implement ingestion patterns into Elasticsearch using one or more

oKafka Connect Elasticsearch

oDataflowBeam

oLogstashBeats

oPubSub pipelines

obatch ingestion via GCS Dataflow

Build search experiences and APIs leveraging

oquery DSL aggregations filters scoring synonyms analyzers multilanguage search if needed

Design and implement reindexing strategies zerodowntime migrations and schema evolution patterns

4 Performance Scale Reliability

Drive performance engineering query latency indexing throughput JVMheap tuning caching shard allocation refresh intervals

Implement high availability and scalability patterns multizone clusters autoscaling approach hotwarmcold tiers

Own operational readiness runbooks SLIsSLOs capacity planning cost optimization load testing

5 Security Governance Compliance

Implement RBAC indexlevel security encryption in transitat rest and secrets management

Ensure compliance with customer data governance retention deletion auditability and access controls

Support privacy and regulatory requirements relevant to CDP data eg consent PII handling righttoerasure workflows

6 Observability Operations

Setup monitoring and ing for cluster health node stats indexingsearch rates thread pools GC disk watermarks

Integrate with observability tools ElasticKibana PrometheusGrafana Cloud Monitoring etc

Partner with SREPlatform to define incident response DR drills upgrade strategy patching cadence

Core Elasticsearch Expertise

Strong knowledge of

ocluster architecture and distributed systems fundamentals

oindexmapping design analyzers tokenizers relevance tuning

oaggregations query optimization search templates

oILM rollover snapshots shard allocation strategies

Experience with largescale clusters TB data high QPS high ingestion rates

Expertise in troubleshooting redyellow clusters unassigned shards circuit breakers GC pressure slow logs

Cloud Platform GCP

Handson experience deployingrunning Elasticsearch on GCP using

oGCE GKE managed instance groups autoscaling

oVPC design firewalling private connectivity

oCloud Storage for snapshots and backup

oCloud MonitoringLogging integration

Understanding of infrastructure automation

oTerraform IaC

oHelmKubernetes manifests if on GKE

oCICD pipelines for configuration and deployment

Data Integration

Strong understanding of CDPstyle data

oconsumer profiles identities eventsclickstream attributes segments

Integration patterns with

oKafka PubSub DataflowBeam Spark BigQuery

Experience modeling eventtime and profile data for search analytics

Agile Leadership

Proven ability to lead teams in Agile delivery sprint ceremonies story slicing estimation incremental delivery

Strong communication and stakeholder management ability to drive decisions