The Principal Engineer is the most senior individual contributor in the System Platform Team. This is not an architect-in-an-ivory-tower role - it is a hands-on technical leadership position where roughly 60% of time is spent writing production code and 40% on cross-cutting technical responsibilities: architecture, technology evaluation, design reviews, and mentoring. The expectation is that when the team runs into its hardest technical problem, the Principal is the one who unblocks it - not by delegating, but by pairing or by writing the solution.
Over the course of the year, the Principal intentionally rotates across different engineering domains - developer experience, release engineering, data and messaging, incident and compliance tooling. This breadth is deliberate: by the end of the year, the Principal has deep firsthand knowledge of the full platform codebase, which is what makes their architecture reviews, technology decisions, and mentorship credible across all areas.
The Stack You Will Work On
- A Java/Spring Boot microservices platform (GreyMatter) running on GKE, backed by Kafka, PostgreSQL, Redis, and MinIO/S3
- An Erlang/OTP real-time robot-control system (Butler Core) running on GCP VMs, using Mnesia, EMQX (MQTT broker), and Cowboy HTTP
- A hybrid CI/CD setup spanning GitHub (mono-repo for 19 platform services) and Bitbucket (legacy/firmware repositories), with Jenkins, Spinnaker, ArgoCD, Nexus, Terraform, and Ansible
- Kafka-based event streaming with Avro schema contracts and Redis for payload management
- An internal context-retrieval server that surfaces runbooks, tickets, deployment history, and architecture docs as LLM-queryable tools
- An incident management platform (evaluating commercial tools vs. in-house) with a custom compliance evidence layer for SOC2/ISO audits
- A home-grown Just-In-Time access portal covering GKE, GCP IAM, Kafka ACLs, SQL, Mnesia, and config management, integrated with Google Workspace SSO.
Responsibilites
Architecture & Technical Direction
- Lead cross-team architecture reviews - help the team make large technical decisions, document trade-offs, and land on durable solutions
- Own build-vs-buy decisions: evaluate commercial tools against in-house alternatives with documented analysis, and live with the consequences
- Define and enforce engineering standards across the whole team: schema conventions, observability patterns, security postures, secrets hygiene, API design
- Review system designs across every engineering area the team works in - not just the area you are currently contributing to
- Partner with the Associate Director on technical hiring: contribute to interview design, loop calibration, and leveling decisions
Hands-On Engineering
- Rotate across different engineering domains throughout the year, contributing meaningfully to each - shipping production code, not just reviewing it
- Write and maintain design documents and architecture decision records (ADRs) - these are required deliverables, not optional write-ups
- Unblock engineers on hard problems by pairing directly, not by delegating to someone else
- Build and own cross-cutting libraries, shared tooling, and foundational components that multiple services depend on
- Mentorship & Culture
- Technically mentor the Staff Engineers - paired design sessions, deep code review, escalation target for hard problems
- Raise the bar in code review: model the level of feedback and depth of analysis that sets the team standard
- Drive a blameless engineering culture - incidents and mistakes are learning opportunities, not blame events
- Treat documentation as first-class engineering output: design docs and runbooks ship with features, not after them
Requirements
- 14+ years of software engineering with at least 5 years at Staff Engineer level or equivalent senior IC level
- Has shipped at least one platform, developer-experience, or infrastructure product at scale that other engineering teams measurably adopted - not just delivered, but adopted and continued to use
- Production experience across at least four of: Kafka, Kubernetes, GitOps and ArgoCD, Grafana/InfluxDB observability stacks, OAuth2/OIDC identity systems, LLM orchestration or RAG systems
- Demonstrated build-vs-buy judgment: has made significant make-or-buy decisions, documented the trade-offs, and lived with the outcome
- Deep system design instincts - can hold a 10+ service distributed system in mind and identify coupling problems and failure modes before they become production incidents
- Excellent written communication - drives technical influence through documents, not just verbal authority in meetings
- Comfortable operating as the senior-most IC on a small team where pragmatism and real constraints shape decisions
Nice to Have
- Familiarity with Erlang/OTP, Mnesia, or BEAM VM tooling - for context on the Butler Core platformExperience with hybrid deployment environments spanning Kubernetes and traditional VMs in the same production system
- Has led a technical governance body - architecture review board, RFC process, or equivalent cross-team decision-making forum
- Background evaluating and procuring engineering tooling at a technical (not just commercial) level
- Familiarity with VDA5050 or AGV/robotics communication protocols
