ECS 2027

31/05/2027, Congress Center Düsseldorf
Düsseldorf, Germany

Stop guessing: How to write effective evals for AI coding agents

You built an MCP server, wrote an agent skill, or improved your documentation. How do you know whether it actually helps AI coding agents complete tasks better? You run an eval of course! But a poorly designed eval can give you confidence in the wrong thing. Too many evaluations reward agents for invoking specific tools, following prescribed steps, or producing expected snippets of code. They measure compliance with our assumptions rather than success from the user's perspective. In this session, we'll walk through designing effective evaluations for AI coding agents, starting with realistic user scenarios and defining what successful outcomes look like. You'll learn how to write meaningful evaluation criteria, distinguish must-haves from nice-to-haves, and assess both implementation quality and task cost. We'll review common pitfalls, discuss alternative approaches and their trade-offs, and explore how to deal with model variability, baseline comparisons, and misleading results. We'll also look at how to use agent trajectories to understand failures and turn evaluation results into concrete improvements. Throughout the session, we'll draw on real-world examples and lessons learned from our ongoing work at Microsoft evaluating how AI coding agents interact with developer tools and technologies. You'll leave with a practical approach to writing evals that answer the question that matters: does your technology help agents deliver better outcomes at an acceptable cost?