Test sets & accuracy

Laya answers any question you write, but how you word it changes the answers a lot. A test set tells you how well a question works on your own texts before you put it in production.

Test sets live in the console under Test sets. A set is a list of texts, each optionally with the answer you expect. You run a question over the whole set and get the accuracy, where it goes wrong, and every answer as a CSV.

Create a set

Upload or paste up to 2,000 rows of up to 8,000 characters:

CSV with a header row: a text column and, to measure accuracy, a label column. expected, answer, category, class, intent and outcome work as the label column too. Other columns are ignored.
Tab-separated text: text, a tab, then the expected answer.
JSON Lines: one {"text": "...", "label": "..."} per line.

support.csv

text,label
"I was charged twice this month, please refund one payment.",refund
"The dashboard shows a blank page since this morning.",bug_report
"Could you add an integration with Slack?",feature_request

Labels are compared without regard to case. For a choice question they are option names, for a score the level names, for a yes/no question yes or no (true/false and 1/0 work too), and for a flow the outcome values. Rows without a label are answered but not scored, so a set without labels is also a way to classify a backlog and download the answers.

Two samples are one click away in the console: 40 support tickets labelled with the customer's intent, and 30 comments labelled safe or toxic.

Run a question or a flow

Pick a template, write your own question (choice, yes/no or score) or pick one of your flows, choose one or more models, and run. Each row is billed like a /v1/decide call; a flow bills only the questions on each row's path. The console shows the cost before you start. Runs appear in Logs with the endpoint test-run, without their text: the text is already in the set.

A run keeps going if you close the page. If the balance runs out, it stops and keeps what was answered.

Compare models

Pick up to six models before you run, for example laya-english, clef-flash and clef. Each gets its own run over the same rows with the same question, and once two or more runs are finished the page shows them side by side: accuracy, AUC, median latency per row and cost per 1,000 rows. Click a run to see its mistakes.

It is the quickest way to choose: the cheapest model that is accurate enough on your texts. On our own held-out sets the winner changed with the task.

What you get

Accuracy on the labelled rows.
Precision, recall and F1 per answer, so you see which answers the question mixes up.
A confusion matrix: expected answers down, given answers across.
For questions with two answers (yes/no, spam/legit): AUC, and the probability threshold that best separates them, with how many positives it catches and how many negatives it flags. Use it as a yes/no step's threshold, or its “unsure below” value, in a flow. It is chosen on the same rows it is measured on, so expect slightly less on new text.
The rows themselves, filtered to mistakes, unsure answers (probability under 0.6) or errors, and a CSV of every answer.

Getting a question right

Label 50 to 200 real examples, with every answer you care about represented.
Run the closest template, then your own wording. Keep the runs: the list shows each run's accuracy side by side.
Read the mistakes. Most fixes are in the option descriptions: say what belongs in each option, and what doesn't (“harsh criticism of a product, without attacking a person”).
Change one thing per run. Wording that wins by one or two rows on a small set is noise; look for clear gains.
Keep a part of your examples out of the tuning, and check the final wording on them once.

If no wording gets good enough, a model fine-tuned on your examples usually does better. See fine-tuned models.