Evaluation note
How Dialogo Evaluates Agent Tool Use: The MCP-Atlas Internal Test
A transparent explanation of Dialogo’s internal 500-task MCP-Atlas evaluation, its 371 completed tasks, and the limitations of the result.
Result and scope
Dialogo completed 371 of 500 tasks in an internal MCP-Atlas evaluation, a raw completion rate of 74.2%. This is an internal engineering result, not an independently audited customer-performance claim.
The test was used to identify failure modes in tool selection, parameter construction, state handling, recovery, and final-answer verification. It should not be interpreted as a guarantee for a customer workflow, because real deployments differ in systems, permissions, data quality, policies, and evaluation criteria.
What a task measures
Each task requires an agent to understand an objective, select available tools, construct valid inputs, handle returned information, and produce the expected outcome. A task is counted as completed only when the evaluation criteria are satisfied.
What the score does not prove
- It does not prove universal reliability.
- It does not represent a specific customer environment.
- It does not measure every security or governance requirement.
- It does not replace workflow-specific acceptance testing.
- It is not an independent certification.
How the result is used
Failures are grouped by cause and turned into engineering work: better tool descriptions, stronger argument validation, clearer state transitions, retry boundaries, approval policies, and outcome verification. The value of the evaluation is the repeatable feedback loop, not the headline number alone.
Evaluation standard for customer pilots
A customer pilot receives its own test set, expected outcomes, permitted actions, approval rules, and success measures. Results are reported with the sample, period, baseline, and method so the customer can verify what changed.
Start with one workflow and prove the boundary.
Map the event, systems, permissions, approvals, exceptions, and evidence before granting access. The result is a pilot scope your operations and technical teams can evaluate together.
Book a workflow mapping