The first agent demo usually feels better than it is. You give it a clean task, the tools behave, the API returns what you expected, and the transcript looks like progress. Then someone asks what happens when the customer gives partial information, the tool times out, or the agent takes an action that looks plausible but is wrong.
I have learned to treat that moment as the beginning of evaluation. A useful agent is not just a model with tools. It is a system that can make bounded decisions, leave evidence behind, and fail in ways the product can survive.
What work is the agent actually allowed to own?
Start by naming the job in plain operational terms. "Resolve support tickets" is too broad. "Classify an inbound request, gather missing fields, and draft a routing decision for review" is something you can test. The boundary matters because an agent with a vague mandate will quietly become a product policy engine.
I like writing down the actions the agent can take without approval, the actions that require review, and the actions it must never take. That document does not need to be long. It needs to be specific enough that an engineer, product manager, and operator would make the same call on a weird case.
What should be in the eval set?
Your eval set should look like the work, not like a prompt contest. Pull real examples if you can, scrub sensitive data, and keep the annoying cases. Empty attachments. Duplicate requests. A customer who changes their mind halfway through. A record that has two possible owners. Those are the cases that tell you whether the agent understands the workflow or only the demo path.
For each case, define what good looks like before you run the agent. Sometimes the right answer is a final action. Sometimes it is asking a clarifying question. Sometimes it is refusing to continue because the tool result is incomplete. If the expected behavior is fuzzy, the score will become whatever the last reviewer felt like that afternoon.
How do you test tool use?
Do not only grade the final answer. Grade the path. Which tool did the agent call? Did it pass the right fields? Did it notice a bad response? Did it retry safely or stop? A transcript can look confident while hiding a tool call that should never have happened.
This is where simple fixtures help. Mock the tool returning a timeout, a 403, stale data, or a response with one missing field. The question is whether the agent keeps its promises when the world stops being polite. Production will run that test for you eventually. I prefer running it first.
When is the agent ready to ship?
I do not look for a magic score. I look for a pattern: strong performance on normal cases, predictable behavior on edge cases, clean handoff when confidence is low, and logs that explain what happened after the fact. If an operator cannot reconstruct the decision, the system is not ready for unsupervised work.
Rollout should be staged. Start in shadow mode, compare the agent's decisions against humans, then move to assisted mode where people approve the action. Only after that should the agent own a narrow path by itself. Even then, keep sampling. Agents drift because products drift, tools change, and users find new ways to be users.
What should leaders ask before approving it?
Ask to see five passes and five failures. The failures are usually more useful. They show whether the team has built a product around the agent or merely wrapped a chat transcript in a nicer screen.
The standard I trust is boring to demo and very helpful in production: bounded authority, real eval cases, tested tool failures, visible handoffs, and a rollout plan that can slow down without causing drama. If this is the kind of problem your team is working through, I am always glad to compare notes.