# Test sets & accuracy

Laya answers any question you write, but how you word it changes the answers a lot. A test set tells you how well a question works on your own texts before you put it in production.

Test sets live in the console under [Test sets](https://layahost.com/tests). A set is a list of texts, each optionally with the answer you expect. You run a question over the whole set and get the accuracy, where it goes wrong, and every answer as a CSV.

## Create a set

Upload or paste up to 2,000 rows of up to 8,000 characters:

- **CSV** with a header row: a `text` column and, to measure accuracy, a `label` column. `expected`, `answer`, `category`, `class`, `intent` and `outcome` work as the label column too. Other columns are ignored.
- **Tab-separated** text: text, a tab, then the expected answer.
- **JSON Lines**: one `{"text": "...", "label": "..."}` per line.

support.csv

```text
text,label
"I was charged twice this month, please refund one payment.",refund
"The dashboard shows a blank page since this morning.",bug_report
"Could you add an integration with Slack?",feature_request
```

Labels are compared without regard to case. For a choice question they are option names, for a score the level names, for a yes/no question `yes` or `no` (`true`/`false` and `1`/`0` work too), and for a flow the outcome values. Rows without a label are answered but not scored, so a set without labels is also a way to classify a backlog and download the answers.

Two samples are one click away in the console: 40 support tickets labelled with the customer's intent, and 30 comments labelled safe or toxic.

## Run a question or a flow

Pick a template, write your own question (choice, yes/no or score) or pick one of your [flows](https://layahost.com/docs/flows), choose a model, and run. Each row is billed like a `/v1/decide` call; a flow bills only the questions on each row's path. The console shows the cost before you start. Runs appear in Logs with the endpoint `test-run`, without their text: the text is already in the set.

A run keeps going if you close the page. If the balance runs out, it stops and keeps what was answered.

## What you get

- **Accuracy** on the labelled rows.
- **Precision, recall and F1 per answer**, so you see which answers the question mixes up.
- **A confusion matrix**: expected answers down, given answers across.
- For questions with two answers (yes/no, spam/legit): **AUC**, and the **probability threshold** that best separates them, with how many positives it catches and how many negatives it flags. Use it as a yes/no step's threshold, or its “unsure below” value, in a flow. It is chosen on the same rows it is measured on, so expect slightly less on new text.
- The rows themselves, filtered to *mistakes*, *unsure* answers (probability under 0.6) or errors, and a CSV of every answer.

## Getting a question right

1. Label 50 to 200 real examples, with every answer you care about represented.
2. Run the closest template, then your own wording. Keep the runs: the list shows each run's accuracy side by side.
3. Read the mistakes. Most fixes are in the option descriptions: say what belongs in each option, and what doesn't (“harsh criticism of a product, without attacking a person”).
4. Change one thing per run. Wording that wins by one or two rows on a small set is noise; look for clear gains.
5. Keep a part of your examples out of the tuning, and check the final wording on them once.

If no wording gets good enough, a model fine-tuned on your examples usually does better. See [fine-tuned models](https://layahost.com/docs/models#fine-tuned).