Pass-rate trend

The trend feed is not currently being published

This page was built around a nightly scheduled job. That scheduler has been retired, so no new points are being appended. This page preserves the historical method, not a live results feed. No downloadable results are currently published.

This page was never the release gate, and an earlier version of it said otherwise. What blocks a release is described in full on Evaluation methodology: the versioned corpora and their hashes, the grading tolerance, the ratchet policy, and the two-armed refusal metric.

We publish measurement at all because customers ask "how accurate is your AI on data we don't see in the demo?" and a sales slide is not an honest answer. On why we publish a method and a 2×2 rather than a single accuracy percentage, see Why there is no headline number.

Methodology

What the trend measured. An evaluation suite ran questions through the chat UI against a seeded Salesforce test tenant on a nightly schedule. It ran curated fixtures and graded each on whether the question reached the expected outcome. It did not grade the rows in the answer. The suites that grade result sets and refusals are the ones described on Evaluation methodology.

What was never in the trend. Two further suites were scheduled alongside it and never contributed a point: a Salesforce natural-language-to-SOQL suite, whose runner had not landed, and an MCP tool-selection suite whose grading step was a placeholder. Both remain unbuilt.

How the numbers were produced

1. A browser test run reported the outcome of each fixture.
2. A publish step merged it with the previous run into a trend file.
3. The page read that trend file. That publishing process has been retired; this historical page no longer requests a feed or offers a download.

accuracy_pct = passing / total * 100, where passing counted the fixtures that finished in the expected outcome (the expected outcome was reached and the chat UI showed it without an error).

Caveats

• Fixture set is curated, not exhaustive. A pass-rate on a curated suite is a statement about the questions we know to test, not about your data. Real-world tenant data has join paths and column semantics this suite has not seen. This is one of the reasons we no longer publish a headline percentage anywhere; see Why there is no headline number.
• Flaky runs counted as failures. A test is flaky when it passes only on a retry. Flaky counted as failing, because a customer-facing answer is only useful if it is right the first time.
• The seeded tenant was small. The suite ran against a sandbox Salesforce org, not a production replica. Answer latency is measured separately, from real answers.

To evaluate your own question set against your own data, contact your account team and read Running it on your own questions.