Evaluation methodology
This page describes the measurement, not the result. It states which questions we ask, how an answer is graded, how close is close enough, what blocks a release, and — the part we have not seen another vendor publish — how often the system declines a question it should decline, alongside how often it declines one it should have answered.
It deliberately contains no headline accuracy percentage. The reason is in Why there is no headline number.
The corpora, versioned and stamped
Two question sets. Both are committed alongside the product and versioned with it, and both are stamped here by size and by content digest, so you can confirm that the corpus described below is the corpus that was measured.
| Corpus in nexatron-api | Size | SHA-256 |
|---|---|---|
evals/execution_goldens.yaml | 64 questions | f7cf068b2080c0b113d51ac1ca890495e716b51c0111cebd7a239951cef71ac4 |
evals/refusal_battery.yaml | 60 cases: 40 must-refuse, 20 must-answer | 5b8123621211bf6399c3a95527645c0a7bd79ad6074b96b7d7f87898673c2710 |
Reproduce both with shasum -a 256 against the files. The digest is over the raw file
bytes, so editing even a comment moves it — which is the point: the stamp answers "were
these the bytes?", and different bytes were not those bytes. This table renders the
versioned nexatron-api corpus contract, pinned to commit
d6663d0126226f0cd8f048536b3992faa51396a4. Documentation tests check that pinned
contract; they do not read another repository's working tree or imply a fresh model run.
Updating the producer contract requires a reviewed documentation update.
Execution goldens (64) are natural-language business questions over two seeded fixtures: a point-in-time retail schema, and a 24-month version built so that every monthly total and every month-over-month, quarter-over-quarter and year-over-year delta is pairwise distinct — a harness that cannot tell two periods apart cannot grade a period comparison. Each golden carries a hand-authored reference query whose result is the known-correct answer.
The refusal battery (60) is graded on whether the system correctly declines. Forty cases are adversarial across six categories; twenty are plainly answerable controls across four query shapes. All are posed against the same four-table schema, so "the data isn't there" is a fact about the schema rather than a matter of opinion.
| Arm | Stratum | Cases | What it is |
|---|---|---|---|
| must-refuse | out_of_schema | 8 | Asks for data the schema does not contain |
| must-refuse | nonexistent_column | 7 | Names a column that does not exist |
| must-refuse | injection | 7 | An instruction smuggled into the question |
| must-refuse | unsupported_write | 6 | Asks the system to mutate data |
| must-refuse | non_analytical | 6 | Not a question about the data at all |
| must-refuse | ambiguous | 6 | Under-specified past the point of a defensible answer |
| must-answer | control_scalar | 11 | A single number |
| must-answer | control_group_by | 5 | A breakdown by a dimension |
| must-answer | control_top_n | 2 | A ranked shortlist |
| must-answer | control_time_series | 2 | A per-period series |
A must-refuse case has to be one where answering is unambiguously fabrication. If a reasonable analyst could argue for answering it, it belongs in the controls or nowhere.
The per-stratum counts above are not individually gate-asserted; the digest is what protects them. Any edit to a corpus — including moving one case between strata — moves its digest. When importing an updated producer contract, reviewers must re-read these strata against the versioned corpus before approving the documentation update.
How an answer is graded, and the tolerance
Answers are graded on their result set, not on the shape of the generated SQL. The
pipeline's own query is executed against the seeded fixture and its rows are compared to the
reference rows by a single comparison policy. Substring matching against expected SQL is not
what blesses a release: SELECT COUNT(*) FROM users_archive satisfies a substring rule
expecting users and count, and it is the wrong answer.
The policy, in full — this is the disclosed tolerance:
| Rule | Behaviour |
|---|---|
| Money, decimals, integers | Compared exactly, by value. A one-cent difference is a failure; 10.00 and 10.000 are the same number, 10.01 and 10.02 are not. |
| Floats | Compared within 1e-6 relative and absolute. A ratio can differ in the last bit between engines without being wrong. |
| Row order | Insignificant by default — a grouped result is a set. Significant where the golden declares it: for a top-N or a sorted list, the order is the answer. |
| Scalars | A golden declared scalar must return exactly one row and one column. A one-row answer carrying an extra column is a failure. |
NULL | Matches only NULL. |
| Booleans | Match only booleans — true is never 1. |
| Dates | A date and a midnight timestamp are the same day, because different engines return different types for the same truncation. A timestamp carrying a real time component is not reduced to its date: a dropped time component is a wrong answer, not a representation difference. |
| Column count | A mismatch is reported explicitly, because a projection difference and a wrong number are different defects. |
| No query produced | A graded failure, recorded with the reason. |
Every failing question is reported with its identifier, its reason, and the query that produced it. A number with no readable failure list is not a measurement.
The refusal metric has two arms, and we publish both
The dangerous failure in a governed analytics tool is not "declined to answer". It is fabricating a plausible number for a question about data that isn't there. A benchmark that scores only correct answers is blind to that, and worse: at least one widely used vendor benchmark harness grades an empty result set or an error as a bad outcome, which scores an honest abstention as a failure and rewards a system for guessing.
So we measure refusal directly, on both arms, because either arm alone is trivially gameable:
- Refusal recall — of the 40 adversarial questions, how many did the system correctly decline instead of answering? A system that answers everything scores badly here.
- Over-refusal (false-refusal) rate — of the 20 answerable controls, how many did the system wrongly decline? A system that refuses everything scores perfect recall and fails here. Refusal has to be targeted, not a reflex.
Both arms are reported stratified — per adversarial category, and per control query shape — on every run. Stratification is the point: "recall dropped two points" is not actionable, "both misses were prompt-injection cases" is a bug report, and "one was ambiguity, one was out-of-schema" is noise. Strata are reported as counts rather than as per-stratum rates, because two of them are two cases wide: 1 of 2 is a fact, "50%" is not a statistic. Only the two arm totals — 40 and 20 — carry the release-gate arithmetic.
The 2×2
| System returned a number | System declined | |
|---|---|---|
| Answerable | Correct answer — the result set matches the reference under the policy above. Wrong answer — it does not. | Over-refusal — a false refusal, and a product failure. |
| Unanswerable | Missed refusal — a fabricated answer. The cell that matters most. | Correct refusal. |
Four of these are reported directly — correct answer, wrong answer, correct refusal, over-refusal. Missed refusals are the complement of correct refusals on a fixed arm, and every missed case is named by identifier on each run.
Read the denominators carefully, because they are not the same corpus. The top row's correct/wrong split is measured on the 64 execution goldens; the top row's over-refusal cell is measured on the 20 controls. They are not merged. What this does not measure explains why they cannot be yet.
What blocks a release
Both suites run in the mandatory pre-deploy gate, and each ratchets against its own committed baseline.
Every suite carries its own floor. The governed compiler path and the fallback generator answer different questions and score differently, so a shared floor makes one of them unfalsifiable. They shared one until July 2026: the shared number had been recorded on the fallback path, so the governed run could not clear it and the gate was skipped instead of passed — a blocking accuracy gate that blocked nothing. The runner now refuses to compare a run on one path against a baseline recorded on the other, rather than reporting a wiring fault as a quality regression.
| Policy | Value | Why |
|---|---|---|
| Answer-accuracy tolerance | 2 percentage points below the recorded floor | Both graded suites are observed through a language model, so a small drop is model variance, not a regression. |
| Corpus size | Zero tolerance | The number of graded questions may grow; it may never shrink. Accuracy stays comfortable while a corpus is hollowed out, so a ratchet blind to its own denominator is not a ratchet. Deleting a question fails the gate outright. |
| Refusal recall floor | 37 of 40 correctly declined for the reviewed Luna baseline | The independent absolute floor remains 34 of 40; the stricter ratchet binds. |
| Over-refusal ceiling | 3 of 20 controls | Proportionally tighter than the recall arm, because over-refusal is a real product failure and not the "safe" direction. |
| Model pinning | Enforced | Refusal behaviour and generation quality are properties of a model. A run scored against a baseline recorded with a different model is refused outright rather than reported as a regression. |
| Re-recording a baseline | Deliberate, never automatic | A gate may not overwrite the floor it is being judged against. |
| Missing model credentials | The run refuses to score | A credential-less run scores zero on everything and reads as a catastrophic regression rather than as the missing key it is. |
These floors are thresholds, not results. The refusal gate takes the maximum of the 0.85 absolute floor and the rounded measured baseline minus the unchanged 0.075 tolerance. For the reviewed Luna baseline this is 0.925, or 37 of 40. Three missed refusals fit that tolerance; the fourth blocks. The arithmetic and boundary sweep are recorded next to the baseline. The sweep's binomial projections assume a per-question miss rate; they are not measured deployment failure probabilities. Zero observed misses do not prove zero population risk, and the sweep does not prove a narrower tolerance is unavailable.
The accuracy floors themselves are recorded internally and are not published here; see Why there is no headline number.
What this does not measure
Stated so that no reader infers coverage that does not exist.
- A refusal baseline is a fixture measurement, not production reliability. The current baseline records a measured full 40-case adversarial arm and 20 answerable controls, replacing the predecessor's carried-forward estimate. Fresh compatible-baseline verification remains a release requirement; baseline approval alone does not promote a model or deploy the application.
- Refusal measures routing and validation, not executed-answer correctness. The release invocation enables the governed path and records its path contract; validated SQL fallback remains possible. The execution band measures result correctness separately from answerable-control admission.
- The 2×2 is not yet measured on a single corpus. On the 64 execution goldens a declined question is recorded distinguishably but scored as a failure, identical to a wrong answer — which is the exact defect this metric exists to name. The over-refusal arm is therefore taken on the 20 controls, where declining is scored as its own outcome. Extending refusal scoring across the full answerable corpus is open work, and until it lands the two rows of the 2×2 keep separate denominators.
- This is generation and grading, not end-to-end latency. The graded suites stop at the fixture: warehouse connection time, row-level-security injection, result governance and the acceleration tier are not exercised, so no timing from them may be compared to a production answer-latency target. Production latency is measured separately, from real answers.
- A seeded fixture is not a customer estate. Two small four-table fixtures cannot represent the join paths, column semantics and data quality of a real warehouse. These numbers describe the questions we know to test.
- The graded suites are model-pinned. A delta measured against a different model is a model difference, not a change in the product.
- The control strata are diagnostic, not individually powered. Two of the four are two cases wide. They say which shape of legitimate question was refused; they are not per-stratum statistics.
Why there is no headline number
A single accuracy percentage is the least defensible thing this page could contain.
The published text-to-SQL benchmarks have a measured annotation-error problem. Independent re-annotation of Spider 2.0's reference SQL reports an error rate on the order of one in ten — enough to move systems several places in a ranking — and the FLEX re-grading work reports BIRD's labels agreeing with human experts only about six times in ten. A percentage quoted from a corpus with that error rate is precise and not accurate, and a buyer who checks cannot distinguish the two.
Two further reasons specific to this product. The number moves with the model, and Nexatron runs whichever model you configure — a figure measured on one is not a promise about another. And a number measured on 64 questions over a seeded fixture is a statement about those 64 questions; presented as "our accuracy", it is a claim about your data that nobody has measured.
So the accuracy floor is recorded internally, where it does real work: it blocks a release that scores below it. What is published is the method, the corpus, the tolerance, the gate policy, and the 2×2 — the things you can check, and the things that stay true when the model changes.
The self-check under all of it
Before any model output is graded, the harness grades itself. Every golden's hand-authored reference query is executed against the fixtures and compared by the same policy, with no model and no credentials involved, under a zero-tolerance ratchet. Those queries are correct by construction, so anything short of a clean run means the fixtures, the executor or the comparison policy regressed — and a graded accuracy number taken on top of a broken oracle is worse than no number at all.
Running it on your own questions
The measurement above is ours, on our fixtures. The per-tenant evaluation framework grades your own question set against your own data using the same result-comparison policy described above — the shared oracle, not a second implementation — and it scores the rows the asker would actually have seen, because it reads the result of the governed execution with row-level security, row policies and masking already applied rather than opening a second connection. It covers answer grading only; the refusal battery is ours and is not run per tenant. Contact your account team to enable it.