Evaluating GenAI Agents in Production: Lessons from Booking.com

Shipping an LLM-based agent is the easy part. Knowing whether it actually works once it is live - breaking down problems, calling external tools, and iterating through multi-step reasoning against real user traffic - is where most teams get stuck. Evaluation that works for a single-model prompt falls apart the moment the system can take actions, and public benchmark scores tell you almost nothing about how the agent behaves on your product. This is a talk about what we actually do at Booking.com to evaluate AI agents in production, with the real numbers and trade-offs behind the framework.

We start with the foundation that everything else rests on: LLM-as-a-Judge. We cover how we build golden datasets that represent the true production distribution - including the edge cases - with annotation protocols rigorous enough to trust, and how we develop judge-LLMs through iterative prompt engineering with chain-of-thought reasoning, then deploy them for continuous automated evaluation at scale. The practical detail matters here: a strong model to set the upper bound in development, a cheaper model to run in production without breaking the cost budget.

The core of the talk is our dual approach to agent evaluation. Black box evaluation measures task completion - the fraction of expected tasks correctly completed per conversation - using a judge LLM that receives the full dialogue, the predefined task list, and product-specific constraints. Glass box evaluation looks inside the agent to understand why it behaved the way it did. We assess tool proficiency through tool call validity (checking calls against JSON schemas for correct names, required parameters, types, and value formats) and tool correctness (whether the agent picks the right tools, including correctly declining when no relevant tool exists). We dig into tool argument hallucination - the agent calls the right tool but invents wrong parameter values, "Paris, France" versus "Paris, Texas" - and tool reliability: a set of judge-LLM checks ensuring tool names and descriptions are clear, concise, and machine-interpretable, because obscure or verbose specifications quietly degrade agent performance.

We then get concrete about benchmarking. Before an agent earns its added complexity, it has to beat simpler baselines - zero-shot LLM, few-shot LLM, deterministic LLM chains - on performance, cost, and latency, so the decision to deploy is data-driven rather than hype-driven. Finally we walk through the frontier problems we are still wrestling with: consistency, where pass^k across paraphrased queries drops below 20% after just 8 trials; reasoning efficiency as task complexity grows; memory quality and its measurable impact on task completion; and trajectory optimality assessed with preference-based judges.

Attendees leave with a concrete, reusable playbook for evaluating AI agents in production - the metrics, the judge prompts, the baselines, and the failure modes - grounded in real-world experience running agents at one of the world's largest travel platforms.