BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//LatentShift//latentshift.ai schedule//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:LatentShift · Evaluating GenAI Agents in Production: Lessons 
 from Booking.com
X-WR-TIMEZONE:Europe/Istanbul
REFRESH-INTERVAL;VALUE=DURATION:PT12H
X-PUBLISHED-TTL:PT12H
BEGIN:VEVENT
UID:ed01-evaluating-genai-agents-in-production@latentshift.ai
DTSTAMP:20261007T125617Z
DTSTART:20261017T114000Z
DTEND:20261017T120000Z
SUMMARY:LatentShift · Evaluating GenAI Agents in Production: Lessons from 
 Booking.com · Ilkcan Keles
DESCRIPTION:Short talk · 20 min · English\n\nIlkcan Keles · Senior Machi
 ne Learning Engineer\, Booking.com\n\nShipping an LLM-based agent is the e
 asy part. Knowing whether it actually works once it is live - breaking dow
 n problems\, calling external tools\, and iterating through multi-step rea
 soning against real user traffic - is where most teams get stuck. Evaluati
 on that works for a single-model prompt falls apart the moment the system 
 can take actions\, and public benchmark scores tell you almost nothing abo
 ut how the agent behaves on your product. This is a talk about what we act
 ually do at Booking.com to evaluate AI agents in production\, with the rea
 l numbers and trade-offs behind the framework.\n\nWe start with the founda
 tion that everything else rests on: LLM-as-a-Judge. We cover how we build 
 golden datasets that represent the true production distribution - includin
 g the edge cases - with annotation protocols rigorous enough to trust\, an
 d how we develop judge-LLMs through iterative prompt engineering with chai
 n-of-thought reasoning\, then deploy them for continuous automated evaluat
 ion at scale. The practical detail matters here: a strong model to set the
  upper bound in development\, a cheaper model to run in production without
  breaking the cost budget.\n\nThe core of the talk is our dual approach to
  agent evaluation. Black box evaluation measures task completion - the fra
 ction of expected tasks correctly completed per conversation - using a jud
 ge LLM that receives the full dialogue\, the predefined task list\, and pr
 oduct-specific constraints. Glass box evaluation looks inside the agent to
  understand why it behaved the way it did. We assess tool proficiency thro
 ugh tool call validity (checking calls against JSON schemas for correct na
 mes\, required parameters\, types\, and value formats) and tool correctnes
 s (whether the agent picks the right tools\, including correctly declining
  when no relevant tool exists). We dig into tool argument hallucination - 
 the agent calls the right tool but invents wrong parameter values\, "Paris
 \, France" versus "Paris\, Texas" - and tool reliability: a set of judge-L
 LM checks ensuring tool names and descriptions are clear\, concise\, and m
 achine-interpretable\, because obscure or verbose specifications quietly d
 egrade agent performance.\n\nWe then get concrete about benchmarking. Befo
 re an agent earns its added complexity\, it has to beat simpler baselines 
 - zero-shot LLM\, few-shot LLM\, deterministic LLM chains - on performance
 \, cost\, and latency\, so the decision to deploy is data-driven rather th
 an hype-driven. Finally we walk through the frontier problems we are still
  wrestling with: consistency\, where pass^k across paraphrased queries dro
 ps below 20% after just 8 trials\; reasoning efficiency as task complexity
  grows\; memory quality and its measurable impact on task completion\; and
  trajectory optimality assessed with preference-based judges.\n\nAttendees
  leave with a concrete\, reusable playbook for evaluating AI agents in pro
 duction - the metrics\, the judge prompts\, the baselines\, and the failur
 e modes - grounded in real-world experience running agents at one of the w
 orld's largest travel platforms.\n\nhttps://latentshift.ai/schedule/evalua
 ting-genai-agents-in-production/
LOCATION:Bağlarbaşı Kongre ve Kültür Merkezi\, Boğaziçi Salonu\, Sel
 ami Ali Mah.\, Gazi Cad. No:22\, 34664 Üsküdar\, İstanbul
GEO:41.024366;29.033571
URL:https://latentshift.ai/schedule/evaluating-genai-agents-in-production/
CATEGORIES:LatentShift,Short talk
STATUS:CONFIRMED
TRANSP:OPAQUE
BEGIN:VALARM
ACTION:DISPLAY
TRIGGER:-PT10M
DESCRIPTION:Starts in 10 minutes · Evaluating GenAI Agents in Production: 
 Lessons from Booking.com
END:VALARM
END:VEVENT
END:VCALENDAR