Evals

Evals holds question sets: questions your team actually asks, with the answers they should get. Re-run them after a definition changes to see that answers still come out right. Workspace admins manage and run them.

This is not the same as the public eval methodology, which describes how Nexatron itself is measured.

Create a question set

1. Select New question set, give it a name and an optional description, and select Create. Your sets are listed under Question sets.
2. Open the set and select Add question. Write the question as people would ask it.
3. Say what The answer engine should do: Answer it, Say it can't answer it (a good answer for a question nobody should get a number for), or Ask which one was meant.
4. Optionally set what to check:
   • Expected answer (optional): a number, date, value, or rows as JSON.
   • Tolerance for numbers, % (optional): blank means amounts must match to the cent.
   • Definition it should use (optional).
   • Least trusted definition allowed (optional): for example, fail if the answer used only Nexatron's own interpretation.

A question with none of these set is reported as ungraded: it is asked, but nothing checks its answer, and it is left out of the pass rate.

Run and read the results

Select Run eval. Each question is asked as you, through the Nexatron answer core, so every answer is the one you would get on Ask. Answers are checked on their values and the definition that answered them.

Pass rate over runs shows the trend, and Recent runs lists each run. Open a run to see Run detail: each question's checks (answered as expected, value, definition, trust), the expected answer, the query the answer ran, and what the answer core said.

Add at least one enabled question before running; a run over zero questions records no pass rate.