About WhiteCrow
We are global talent research, insight, and sourcing specialists with offices in the UK, USA, Singapore, Malaysia, Hong Kong, Dubai, and India. Our international reach has helped us to understand and penetrate specialist markets at a global level. In addition to this, our service is also extended to complement our client's in-house talent acquisition teams.
About our client
Our client operates in the consumer financial services space, focusing on making everyday purchases and essential needs more accessible through flexible financing solutions. They support individuals across their financial journey—from obtaining their first line of credit to managing long-term financial flexibility—by enabling more informed and responsible credit decisions.
Our client has built a vast network that connects consumers with a wide range of small and mid-sized businesses, as well as providers in the health and wellness sector. Through this ecosystem, they play a meaningful role in supporting both customer financial well-being and the growth of businesses that form a critical part of the broader economy.
As a AVP , Principal Engineer - AI Agent Engineering, you will be responsible for...
Building (hands-on, 60%)
- Designing and implementing end-to-end agents on AWS Strands SDK and Bedrock AgentCore, including tool orchestration, multi-step reasoning, memory, HITL gates, and structured streaming. Use CrewAI or LangChain/LangGraph only where Strands does not fit, with explicit justification.
- Writing clean, idiomatic, testable Python: typed (mypy/pyright), modular, async-aware, with strong package boundaries. Following SOLID and apply the right design pattern for the job (Strategy, Adapter, Repository, Factory, Mediator, Circuit Breaker, Saga) — and applying the right agentic pattern (ReAct, Plan-and-Execute, Reflection, Router, Hierarchical Multi-Agent, Tool-Use, RAG-Fusion) deliberately rather than by default.
- Building agent backends as proper services (FastAPI), with clear domain models, dependency injection, hexagonal/ports-and-adapters separation between LLM/tool calls and business logic, and contract tests against platform APIs and MCP tools.
- Implementing RAG correctly: chunking strategy, embedding choice, hybrid retrieval, grounding/citations, and eval-driven iteration. Use only platform-provided vector stores and follow data-classification and residency rules.
- Integrating with the model gateway, MCP/tool registry, identity (OIDC/OAuth2/SCIM), and observability SDK rather than calling model APIs or building auth/logging directly.
Leading and guiding (40%)
- Setting and enforcing the engineering bar across 1–2 agile teams: code reviews, designing reviews, ADR (architecture decision record) discipline, definition of done, and PR standards. Block merges that skip evals, observability, or guardrails.
- Decomposing roadmap items into well-shaped backlogs; pair-program with mid/junior engineers; mentor on Python, cloud, and agentic concepts; grow the next tech leads.
- Owning the team's technical roadmap in partnership with PM and the AI Platform architects; identifying reusable primitives and push them upstream into the platform instead of forking.
- Driving incident response for agent workflows: triage, RCA, postmortems, and reliability follow-ups. Carrying primary on-call rotation alongside the team.
- Writing designs, runbooks, and ADRs that both engineers and non-technical stakeholders can read.
DevOps / LLMOps
- Owning CI/CD for agent services end-to-end (Jenkins or GitHub Actions): unit + integration + contract tests, SAST/secret scanning, image build, IaC plan/apply, and gated promotions across dev/QA/prod AWS accounts.
- Treating evals as a first-class CI gate: golden datasets, rule-based and LLM-as-judge scoring, replaying harness for deterministic reproduction of production sessions, and regression checks on every model or prompt change.
- Instrumenting every agent through the platform observability SDK: structured logs, OTEL traces with token/cost stamping, per-tool spans, and dashboards in CloudWatch / Splunk / New Relic. Defining and meeting SLOs (p95 latency, success rate, RAG groundedness).
- Running online evals and drift detection on production traffic; wire kill-switch and cost circuit breakers; responding to guardrail and content-safety incidents.
- Contributing Terraform/CDK modules for agent services and following least-privilege IAM, private networking, and secrets-via-vault patterns by default.
What you already have...
- Bachelor's degree in Computer Science / Engineering or equivalent practical experience.
- 8+ years of software engineering experience, with 3+ years as a tech lead / staff / principal engineer on cloud-based production systems.
- Expert-level Python: typing, async, packaging, testing (PyTest, hypothesis), API design (FastAPI), and clean architecture. Able to read and improve a teammate's code on sight.
- Demonstrated mastery of software design patterns and the judgment to know when *not* to use them. Comfortable leading design reviews and ADR discussions.
- Hands-on experience building agentic / LLM applications in production or advanced pilots, including tool use, multi-step reasoning, memory, and HITL.
- Working knowledge of AWS Strands SDK and/or Bedrock AgentCore SDK, plus core AWS services (Bedrock, IAM, VPC/networking, S3, ECR/EKS or ECS, Secrets Manager, CloudWatch).
- Solid RAG fundamentals: embeddings, vector stores, hybrid retrieval, grounding, eval-driven iteration.
- Strong DevOps / LLMOps: CI/CD pipelines, IaC (Terraform or CDK), containerization, observability (logs/traces/metrics), and incident response. Has carried on-call.
- Experience integrating through API / model / MCP gateways with proper authn/z, rate limiting, retries, idempotency, and error semantics.
- Track record of mentoring and raising the bar for a team — not just shipping personal code.
- Strong written and verbal communication across engineering, product, security, and risk audiences.
