Pass-rate trend

The chart below is updated nightly from the same eval suite that gates every release of Nexatron's natural-language pipeline. We publish it because customers ask "how accurate is your AI on data we don't see in the demo?" and a sales slide is not an honest answer.

Loading trend data…

Methodology

What is in the trend today. The planner-eval-live suite runs the structured-planner pipeline through the chat UI against a seeded Salesforce test tenant, every night at 07:00 UTC. The suite executes a curated set of fixtures and grades each on whether the produced plan matches the expected plan. Fixtures live in backend/tests/services/conversation/eval_fixtures.py.

What is not in the trend yet. Two other eval suites are scheduled but their runner modules are still ahead of us:

• salesforce-evals (nightly 06:00 UTC) targets the NL-to-SOQL path against a live Salesforce test org. The workflow at .github/workflows/salesforce-evals.yml references evals.salesforce_runner, which has not landed yet.
• mcp-evals (weekly Mondays 02:00 UTC) targets MCP tool selection accuracy. The LLM-judge step is a placeholder pending backend/evals/mcp_tool_runner.py.

These will join the trend as their runners ship. The cadence on mcp-evals will flip to nightly at the same time.

How the numbers are produced

1. The Playwright live job at .github/workflows/playwright-eval.yml emits a JSON report with {stats: {expected, unexpected, flaky, skipped}, suites: [...]}.
2. A separate publish job at .github/workflows/eval-trend-publish.yml fires every night at 08:00 UTC. It downloads the previous run's artifact, runs the aggregator at backend/scripts/aggregate_eval_trend.py, and pushes the merged trend file to S3.
3. The trend file at /eval-trend/trend.json is served by the same CloudFront distribution as this docs site. The page fetches it client-side on every load — no rebuild required for new data to appear.

accuracy_pct = passing / total * 100, where passing is the count of Playwright fixtures that finished in the expected outcome (i.e., the planner produced the right plan and the chat UI rendered it without surfacing an error).

Caveats

• Fixture set is curated, not exhaustive. A 90% pass-rate on the suite does not mean Nexatron is 90% correct on your data — it means it is 90% correct on the questions we know to test. Real-world tenant data has join paths and column semantics this suite has not seen.
• Flaky runs count as failures. Playwright marks an outcome flaky when it retries and eventually passes. We count flaky as failing for the public number because a customer-facing answer is only useful if it is right the first time.
• The seeded tenant is small. The suite runs against a sandbox Salesforce org, not a production replica. Performance on real-tenant scale is measured separately (see release-time latency benchmarks).

If you want to run Nexatron against your own question set under your own data, contact the team — the per-tenant eval framework lives at backend/app/services/tenant_evals.py.