Job Title: Lead Prompt Engineer
Employment Type: Full-time
Work Type: Hybrid
Duration: 12 months (Annual Renewal)
Location: Gurgaon or Hyderabad
ABOUT THE ENGAGEMENT:
Aceolution is hiring for the work that supports the development, evaluation, and safety calibration of frontier generative AI systems used by hundreds of millions of users worldwide.
KEY RESPONSIBILITIES:
As Per the engagement scope: architecture understanding, complex prompt design, and defining QA frameworks. You are the technical owner of the pod and the primary technical voice in client calibration sessions.
- Own the architectural direction of the client tools API integration. Define how prompts, RAG, and evaluation infrastructure fit together for each workflow the pod supports.
- Design complex prompt patterns. Set the standards for few-shot construction, negative constraints, and structured output that the team builds against.
- Define and own the QA framework. Specify the Golden Dataset structure, regression-test methodology, and the rubric for good vs bad output.
- Lead the adversarial testing approach. Ensure prompts are robust to malicious input, prompt injection, and edge-case content.
- Review and approve all production prompt deployments. Last line of defense before a change reaches the live orchestration layer.
- Conduct weekly architecture reviews with Prompt Engineers — direct, technical feedback.
- Represent the team in client technical reviews. Defend architecture decisions and translate client product feedback into engineering work.
- Drive the team's 95% alignment success metric, plus secondary metrics: false-positive rate, average handle time reduction, engineering-feedback acceptance rate.
MUST-HAVE SKILLS AND EXPERIENCE:
- 5–12 years total experience, with at least 3 years working hands-on with LLMs in production — not just inference, not just chat; production deployments with real users.
- Demonstrable production experience with at least one major LLM API — Gemini, OpenAI, or Anthropic. Gemini-specific experience is strong-to-have but not mandatory; a strong candidate from another LLM platform will ramp in 2–3 weeks. Must be able to discuss context window optimization, token economics, and model-specific behavior quirks fluently.
- Experience designing RAG systems end-to-end: chunking strategy, embedding model selection, vector DB choice, retrieval evaluation.
- Experience designing or running adversarial-testing frameworks. You have stress-tested LLM systems against malicious or edge-case input.
- Strong Python proficiency. You read and review code daily; you do not hand it off to others.
- Experience translating ambiguous client requirements — where guidelines are 50% indicative, not 100% deterministic — into structured prompts and evaluation criteria.
- Demonstrated stakeholder management at the Director / Senior PM level on the client side. You can defend an architecture decision in a room of senior client engineers.
STRONG-TO-HAVE SKILLS:
- Experience with multi-agent frameworks: LangGraph, CrewAI, AutoGen, or custom orchestration.
- Familiarity with eval frameworks: DeepEval, Ragas, OpenAI evals, or in-house equivalents.
- Experience with model fine-tuning approaches — LoRA, QLoRA, full fine-tuning.
- Background in content moderation, ad-tech evaluation, or trust-and-safety systems.
- Experience working in a vendor or consulting model with a single high-stakes client.
Important notice:
Aceolution Inc. will never request a monetary deposit for any role or project with the company, and our recruitment and sourcing teams only use @aceolution.com address when emailing candidates. Ignore aceolutions.com which is a spammer email ID doing rounds over the past few months.
