Skip to main content

​
Why this is a common failure point

Agents backed by a knowledge base fail in two opposite ways. They make things up: a price, a policy, a menu item that isn’t there, stated with total confidence. Or they play it too safe and say “I don’t know” to questions the knowledge base answers plainly. Both look like reasonable conversation, and both cost you: one misleads callers, the other sends them to a human.

​
How Roark tests it

Knowledge base grounding runs callers who ask questions your knowledge base can answer, then checks every claim and offer the agent made against the source. It scores both failure directions separately, so an agent that never hallucinates but never answers either can’t hide behind a good grounding score.

​
Setting it up

Pick the knowledge bases to check against (leave it empty to use every knowledge base in the project), then attach the flows that ask the questions.

​
What it measures

The template seeds these, and you can add or remove metrics in Advanced before running.

​
What to look for

  • Read both scores together. High grounding with many missed answers means the agent is too cautious, not accurate.
  • Ungrounded offers cost the most. A made-up discount or a menu item you don’t sell becomes a promise the business has to handle.
  • Write flows around real questions. Pull them from your support queue so the test reflects what callers actually ask.

​
Over the API

Run it with POST /v1/simulation/run and "template": "knowledge-base-grounding". GET /v1/simulation/template lists what each template accepts.