Judging quality: blind tests and rubrics

Two tools, one shop

Sofia has an online shop selling cheese and regional sweets. She gets about twenty customer emails a week: delays, mixed-up orders, requests for an invoice. She uses an AI tool for the draft replies. A friend swears another one is "much better". An advert guarantees it is a third.

Who is right? Nobody knows, because nobody has tested with Sofia's emails. This lesson teaches you to run that test in an afternoon, without knowing anything technical.

The concept: same task, no names, with criteria

A fair test has four pieces:

1. The same task for both tools, or for two different requests in the same tool.
2. Answers with no names. You call them A and B. Whoever scores does not know which is which.
3. A rubric: criteria written before you read the answers.
4. A third person scoring. You already have preferences; they do not.

This is called a blind test. It is the principle of blind wine tasting: the label stays hidden so that it does not rule over the taste.

The rubric: 3 to 5 criteria, scores from 1 to 5

For Sofia's emails, four criteria are enough:

• Correct: does not make up deadlines, prices or rules that the shop does not have.
• Solves it: answers what the customer asked and says what the next step is.
• Tone: polite and warm, without exaggeration or stock phrases.
• Portuguese: European Portuguese, with no mistakes (it is the language of Sofia's customers).

For each criterion, write what a 1, a 3 and a 5 are worth. For example, in "Correct": 1 = makes up a rule of the shop; 3 = vague, but no mistakes; 5 = everything matches the shop's rules.

How to set up the test, step by step

1. Choose 5 real emails, varied: one easy, three medium, one hard (the angry customer). Remove names, addresses and order numbers, as you learned in Level 1.
2. Use the same request in both tools, word for word, each time in a new conversation.
3. Copy the answers into a document. For each email, shuffle the order: sometimes tool 1 is A, sometimes it is B. Keep the key on a separate piece of paper.
4. Remove whatever gives away the source: special formatting, openings like "Sure! Here is…".
5. Give the document and the rubric to a third person. Ask for a score from 1 to 5 on each criterion and one sentence to justify each 1 or 2.
6. Close the scores: from here on nobody touches them. Only now do you open the key and, for each email, mark which answer (A or B) came from which tool. Add up per tool, not per letter: the letter changes from email to email.

The test is not only for choosing tools. It also compares two requests in the same tool. Often, improving the request is worth more than changing tool.

The AI as critic, not as judge

You can ask the AI itself to critique an answer with your rubric:

Assess the answer below with this rubric. Be demanding.
Criteria (score from 1 to 5): Correct, Solves it, Tone, Portuguese (European).
For each criterion: the score, one sentence to justify it and one concrete suggestion to improve.
At the end, rewrite the answer with the improvements.
Shop rules: [paste here]
Answer to assess: [paste here]

It is useful for improving a draft. But do not use it as the final judge: it tends to be kind to its own work and sometimes gives a high score to an answer with mistakes that it cannot see itself. The score that decides is a person's.

And what about public tests?

When a new model comes out, tables appear with names of tests and scores. They are called benchmarks. They measure things like maths exams, programming problems or general knowledge questions, almost always in English. They are there for technicians to compare models with each other.

But they do not measure your work. No benchmark knows whether the answer to the man who ordered cheese and got blackberry jam sounds good to a customer in Terceira. And whoever advertises picks the tests in which they look good in the photo. Your test, with your emails, is worth more to you than any table.

Exercise: your first blind test