Let the machine explore too

Exploratory testing with coding agents

Exploratory testing is a 40-year-old idea. With coding agents, the machine can explore too, and what it finds turns into tests.

The idea: a person explores, a machine checks

Many of the bugs we chase in a live AI product don’t sit inside one function. They sit between the parts. At Fermi we build a voice tutor: it talks, writes on a board, and asks the student questions. Its bugs look like this: the tutor says “pick A, B or C” while the screen shows 1, 2, 3. Or the student asks to learn Ampère’s law, and the tutor receives “teach me Teach me Ampère’s law”.

Each part’s own tests can pass while this happens, because each part does its job. A person using the product notices some of it. “That felt off.” But a report from memory rarely says what happened first, so the developer starts by trying to make it happen again.

So we split the job in two.

• A person explores. They use the product the way a student would, and try to break it.
• A machine checks. It records every message passing through one point between the parts, and checks each one against a few rules we wrote down before we started. When a rule breaks, it writes a line, with the message that broke it.

The person is good at “this feels wrong”. The machine is good at “this message broke rule 3 at 17:50”. Together, the bug report comes with its evidence.

A coding agent can use the product too. Once it can, the machine explores as well, and what it finds, we keep as tests.

One session, start to finish

Here is a real session from 13 July 2026: one tester, one lesson on Ampère’s law.

1. Write down what you will try. Before starting, the tester listed six things:

1. Run a normal lesson.
2. Pause to think for a long time while the tutor is mid-explanation.
3. Answer a quick check-your-understanding question wrong, then right.
4. Ask something beyond the topic.
5. Interrupt the tutor while it is speaking.
6. Steer it in the middle of an explanation.

This list is called a charter. It keeps a free session pointed at the parts most likely to break.

2. Write down what must always be true. These are rules the machine can check without understanding the lesson. Three from our sessions:

• Math on screen is drawn as a formula, never left as code like $1/r$.
• The student’s first message reaches the tutor as they sent it.
• While the tutor is teaching, no more than 30 seconds pass without a message. Not while it waits for the student to answer.

These are invariants: things that hold whatever the tester does.

3. Listen at one place. Most of what the tutor does passes through one point: the server sending messages to the browser. We copy every message there into a log. That listening point is the tap. Before exploring, we check that messages are actually arriving in the log. When a rule needs something that never passes this point, we add a second tap.

4. Explore, while a script checks. The tester works through the list. A small script reads the log as it grows and checks the rules. When one breaks, it adds a line to a file called ISSUES.md. The tester adds what they notice to the same file. From that session, shortened:

1
2
3

[machine] first message arrived as "teach me Teach me Ampère's law"
[person]  tutor says "Pick A, B, or C", the screen shows numbered badges
[machine] math left as code in a note on the board: "B falls off as $1/r$"

5. Hand over the file. ISSUES.md plus the log is the bug report. For the doubled “teach me”, the developer doesn’t need the tester to describe what they did. The log shows the first message as it arrived, and the cause sits one step back: the page added “teach me” to text that already started with it.

One session: a person, or the agent, uses the product while a script checks the log.

That is the whole loop: a person explores, a machine checks, and the evidence travels with the bug.

What each side caught

In that session, the script flagged math left as code in the text the server sent, 10 times. The tester noticed it on screen twice. They were repeats of a few bugs, not 10 different ones, but nobody had to spot every one.

The tester caught what the rules didn’t cover. In one diagram, a circle that should have been centred on a wire wasn’t. The script had no rule for “this picture is wrong”. An agent that looks at the screen could be asked to look for that; in this session, the person did.

That is why we keep both: the person notices what looks wrong, and the script checks every message.

The log also explains bugs the person finds. On 15 July 2026 a tester reported that a question appeared on the board and was never spoken. Several fixes had gone in over the previous 16 hours, each right on its own. The log showed the order of events:

1. The tutor’s spoken lines wait in a line to be played, the audio queue. The spoken question was in it, waiting its turn.
2. A safety net added a matching question card, with no audio.
3. With no audio, the card didn’t wait in the queue. It showed at once.
4. Another fix says: when a question is on the board, pause the lesson until the student answers. So the queue paused, with the spoken question still in it.

Three correct changes, one silent lesson.

Three correct changes, one silent lesson. The repair put silent cards in the same queue as everything else, so they show in turn. Live runs then checked it.

The checker was wrong too

The script is code too, and early on it was wrong. Our first sessions filled ISSUES.md with things that weren’t bugs:

• A request to an outside tool took a while, and was flagged as dropped, because the time limit was shorter than the request.
• When the student interrupted, the stop signal arrived a moment before the tutor stopped speaking. The script expected the opposite order, and flagged a missed interruption.
• A “the tutor went silent” alarm turned out to be the student’s silence.
• On a new part of the product, the script reported nothing at all. It wasn’t receiving that part’s messages.

The last one is the dangerous one. An empty ISSUES.md can mean a healthy product, or a script that can’t see anything.

So now the checker gets tested before we trust it. We replay a recorded log with a bug planted in it, and the script must flag it. We replay a clean log, and it must stay quiet. That tests the script’s logic. It can’t tell you the live log is connected, which is why each session starts by checking that messages are arriving.

Test the script before you trust a quiet run.

One more rule: a problem that repeats is one issue. A stall that trips the rule every 10 seconds goes in once, not sixty times.

Let the machine drive

So far a person drives and the machine checks. A coding agent can drive too. It opens the product in a headless browser, talks to the tutor, answers questions (some wrong on purpose), and records its steps. The rules and the log checker stay the same.

Here is what that looked like on 17 July 2026. An agent ran ten lessons on ten topics, four turns each, with no one typing. In one of the ten, on entropy, the lesson outline showed ΔS_system as raw text, while the board drew it properly. An earlier fix had cleaned up math on the board, but the outline took a different path and never got it. We fixed the outline and wrote tests with those exact strings. Undo the fix, and the tests fail. The bug showed up in one topic out of ten.

Two runs, side by side:

The agent also re-runs the exact failing steps after a fix. On 16 July an agent re-ran the silent-question sequence twice to check its repair. In a session on grading answer sheets on 22 September 2026, each fix let the agent’s run get one step further before it stopped, until a final pass went all the way through. One fix landed at 21:11 IST; the agent had re-checked it by 21:27.

The person still matters. On 13 July a person corrected the numbers in an agent’s own report. Two screenshots once led an agent to say no loading spinner appeared; it had appeared, between the frames. The rule since then: record short-lived states, not only screenshots, and the person decides when the agent drives. What the agent adds is coverage and repetition.

Keep what it finds

Once the agent can run a session, the useful findings don’t have to stay in ISSUES.md. Notes from the grading sessions on 22 and 23 September became 30 test scenarios. Each one is an input, the result we expect, and the evidence to capture. One of them: upload the same answer sheet twice, grade the paper, and check that questions answered on both pages are still graded.

A list of scenarios like that, run against every new build, is a build verification test, or BVT. Ours started from exploration instead of being written up front. Most of the 30 are automated and set up to run as a report; none of them blocks a release. A smaller set of quick checks is set up to run on each build we consider shipping.

What a session finds becomes a test.

So the person explores, the machine checks, the agent re-runs, and what is worth keeping becomes a test.

Where this comes from

None of this starts with agents. The six things our tester tried on 13 July are a charter in exactly the sense below. Cem Kaner named exploratory testing in 1984: the tester designs and runs tests at the same time, learning as they go. In 2000 Jonathan and James Bach made it manageable. They split it into sessions of about 90 minutes, each with a charter, notes, and a review at the end. In 2006 Kaner was already writing about automated exploration with checks built in.

We kept the charter and the session.

What we added on top: rules agreed before the session starts, a listening point that records what passes through it, an agent that explores too, and findings that become a BVT.

Try it

Pick something you shipped this week. Write five things you will try on it, the way our tester did for the lesson. Write three rules that must always hold. Copy the messages at one point into a log, and run a small script over it while you explore. Test the script with one planted bug first.

For our tutor, one line of that would be: try “interrupt the tutor mid-sentence”; rule “no 30-second gap while it teaches”; planted bug: a recorded log with a 40-second gap, which must produce exactly one line in ISSUES.md.

A starter kit is here: https://gist.github.com/varun-ml/317f1cd80995bb1b12244125ceec4c3c. It has the list template, a small server that writes the log, and a script with three rules and a self-test. It is a starting point, not our full setup: it pauses the silence rule while the tutor waits for an answer, and it notices a gap when the next message arrives, not while it is happening.

A person explores, the machine checks, the agent re-runs, and what is worth keeping becomes a test.