05 · Things go wrong

Knowing when to stop

A confused model asks to read the same missing file fifty times running; then the provider falls over, and the run ends leaving no sign that anything happened.

~50 min · 1 lab · builds on 04 Tell the model what went wrong

This lesson builds on lesson 04. New here? Start at lesson 01, or carry on: every lab is self-contained.

A tool called average divides the total by how many numbers there were. The column is empty, so it divides by zero and a ZeroDivisionError comes out of it — a kind of failure nobody wrote a line about. Your lesson 4 loop runs it. What does the next request hold?

Nothing. A ZeroDivisionError is a bug, so the run ends and the traceback goes to whoever is watching

That sends the failure to the one reader who cannot act on it. The person watching did not choose that column; the model did, and the model is still mid-job.

The prompt, the call, and a toolResult for it flagged is_error, carrying the exception's own words

One call in, one result out, on every path. The boundary catches Exception, not one hand-picked type, which is why a failure nobody predicted still comes back as a result.

The prompt and the call. The failure is logged, and the loop asks the model to try again

That treats error text as a line for a log file. It is a message to the model: it is the only thing in the request that says why there is no average.

The model asked, the tool failed, the failure came back as its answer, and the model wrote a sentence a person can read. Nothing raised, and nothing was lost. Hold on to the shape of that: today two more things go wrong, and neither of them is a tool.

import harness, lab


def average(arguments):
    numbers = [int(word) for word in harness.str_arg(arguments, "numbers").split()]
    return str(sum(numbers) / len(numbers))     # nobody checked that there are any


mean = {"name": "average", "description": "Average a space-separated list of numbers.",
        "parameters": {"type": "object", "required": ["numbers"],
                       "properties": {"numbers": {"type": "string",
                                                  "description": "The numbers."}}},
        "execute": average}

model = lab.ScriptedModel([
    lab.reply(lab.call("average", {"numbers": ""})),
    lab.say("That column is empty, so there is no average to report."),
])
messages = []
harness.run_agent(model, harness.SYSTEM, messages, [mean], "Average the timings column.")

print("requests made:", len(model.calls))
print("request 1 held:", lab.shape(model.calls[0].messages))
print("request 2 held:", lab.shape(model.calls[1].messages))
arrived = model.calls[1].messages[-1]
print("the message that arrived: role", arrived["role"],
      "| tool_name", arrived["tool_name"], "| is_error", arrived["is_error"])
print("the model then said:", harness.text_of(messages[-1]))

requests made: 2
request 1 held: U
request 2 held: U A[c1] R(c1)
the message that arrived: role toolResult | tool_name average | is_error True
the model then said: That column is empty, so there is no average to report.

Two support chats, same sort of messages, same sort of replies. One has run 10 turns, the other 30. Counting only the input side, roughly how many times more has the 30-turn chat cost in total than the 10-turn one?

About nine. Every turn resends everything before it, so the running total grows with the square of the length: three times as long is about nine times the bill. That is lesson 1, still true and still compounding. Today you will watch the same arithmetic run fifty turns deep without anybody asking it to.

The model that will not stop

Your lesson 4 harness does not crash. A tool that raises comes back as a result the model can read; a tool name nobody ever defined comes back as a sentence. The loop keeps going until the model stops asking for tools, and that has been the whole of "when does a run end" since lesson 3.

Below is a model that does not stop asking. It wants to read missing.py. The file is not there, so read says so and lists the files that are. The model reads that, and asks for missing.py again. [general] Real models get stuck this way: one that has locked on to a wrong idea repeats it, and so does one being handed a tool result it cannot make sense of.

The lab's model has a guard on it, because this page has to load. A real one does not.

Your loop, unchanged, against that model. What stops this run?

The model gives up. After a few identical failures it changes tack and answers

That leaves the decision at the far end of the wire. The only thing the provider sends about stopping is a label, and lesson 3 took the label out of the loop's hands. Look at what every one of those replies was labelled.

The API refuses. No provider lets one conversation go round in a circle like this

There is no conversation at the far end, and no run. The provider sees separate requests, each one valid, each one longer than the last, each one billed. "The run" is a thing that exists only inside your while.

Nothing. It ends when the money runs out, or when somebody notices

There is exactly one way out of your loop, and the model controls it.

Nothing in the loop was ever going to stop it: the one exit is "the reply has no calls", and every one of those replies had one. Fifty requests answered, 87,125 input tokens, for a file that does not exist. [general] Nothing at the far end knows these fifty belong together: each arrives as its own valid request, and each is priced and answered on its own. Request 51 was refused by the lab, which told you what the run had cost — the one thing it does here that a real provider will not do for you.

Where this comes from: Tau pins the cure with a regression test. It scripts one reply that asks for a tool, runs the loop with max_turns=1, and asserts that exactly one request was made and that the run ends on an assistant message reading Agent stopped after max_turns=1 (tests/test_agent_loop.py:465-488).

import harness, lab

# The model: it asks to read missing.py, and it asks again, and again.
model = lab.ScriptedModel([lab.stuck(lab.call("read", {"path": "missing.py"}))])

ws = lab.Workspace({"config.py": "PORT = 9090\n"})
tools = [harness.make_read_tool(ws)]
messages = []

try:
    harness.run_agent(model, harness.SYSTEM, messages, tools, "Find the port.")
    print("the run ended by itself")
except lab.Runaway as stop:
    print(stop)

print("model calls:", len(model.calls))
print("messages on the record:", len(messages))
print("the first four:", lab.shape(messages[:4]))
labels = sorted({m["stop_reason"] for m in messages if m["role"] == "assistant"})
print("stop_reason on every one of those replies:", ", ".join(labels))
print("the tool said so every time, is_error:", messages[2]["is_error"])

model call 51 refused: max_calls=50. This run has already cost 87125 input tokens, and nothing in your loop was going to stop it.
model calls: 51
messages on the record: 101
the first four: U A[c1] R(c1) A[c2]
stop_reason on every one of those replies: toolUse
the tool said so every time, is_error: True

A number, and where you look at it

The fix is a number: at most N model calls in one run, passed by whoever started the run. That part is not interesting. Where you look at it is.

You give run_agent a max_turns=1, so: one model call. The model's one reply asks to read missing.py. Here are the four messages the run leaves on the caller's list when the limit is checked at the top of a turn, before the model is called. Put them in the order the loop appends them.

user · Find the port.
assistant · read(path="missing.py") · c1
toolResult · c1 · File not found: missing.py. Files here: config.py
assistant · (nothing) · error: Agent stopped after max_turns=1

The stop is last, after the result of the call the model had already asked for. That is not luck: by the top of a turn, every call of the turn before it has been answered, so a limit checked there cannot leave one hanging. Move the check one line earlier — look at it as soon as the reply arrives, which reads perfectly sensibly — and the run ends with a call nobody answered. The cell runs both and asks the strict checker what it makes of each record.

Two loops, identical except for one line: design A looks at the limit as soon as the model has replied, design B looks at it at the top of a turn. Same model, same max_turns=1. Read the two records, and the last line under each.

import harness, lab


def stop_message(text):
    """An assistant message with nothing in it, saying why the run stopped."""
    return {"role": "assistant", "content": [], "stop_reason": "error", "error_message": text}


def design_a(model, system, messages, tools, prompt, *, max_turns):
    """A: check the limit as soon as the model has replied."""
    turn = 0
    messages.append(harness.user_message(prompt))
    while True:
        reply = model.complete(system, messages, harness.tool_specs(tools))
        messages.append(reply)
        turn += 1
        if turn >= max_turns:
            stop = stop_message(f"Agent stopped after max_turns={max_turns}")
            messages.append(stop)
            return stop
        calls = harness.tool_calls(reply)
        if not calls:
            return reply
        for call in calls:
            messages.append(harness.run_tool(tools, call))


def design_b(model, system, messages, tools, prompt, *, max_turns):
    """B: check the limit at the top of a turn, before the model is called."""
    turn = 0
    messages.append(harness.user_message(prompt))
    while True:
        turn += 1
        if turn > max_turns:
            stop = stop_message(f"Agent stopped after max_turns={max_turns}")
            messages.append(stop)
            return stop
        reply = model.complete(system, messages, harness.tool_specs(tools))
        messages.append(reply)
        calls = harness.tool_calls(reply)
        if not calls:
            return reply
        for call in calls:
            messages.append(harness.run_tool(tools, call))


for name, run in (("A", design_a), ("B", design_b)):
    model = lab.ScriptedModel([lab.stuck(lab.call("read", {"path": "missing.py"}))])
    ws = lab.Workspace({"config.py": "PORT = 9090\n"})
    messages = []
    run(model, harness.SYSTEM, messages, [harness.make_read_tool(ws)],
        "Find the port.", max_turns=1)
    asked = sum(len(harness.tool_calls(m)) for m in messages if m["role"] == "assistant")
    answered = sum(1 for m in messages if m["role"] == "toolResult")
    print(f"design {name}:  model calls: {len(model.calls)}   record: {lab.shape(messages)}")
    print(f"           calls asked for: {asked}   results recorded: {answered}")
    print("           anything left open in the record:",
          lab.validate(messages, record=True) or "nothing")

design A:  model calls: 1   record: U A[c1] A(error)
           calls asked for: 1   results recorded: 0
           anything left open in the record: ['messages[1]: toolCall c1 has no toolResult']
design B:  model calls: 1   record: U A[c1] R(c1) A(error)
           calls asked for: 1   results recorded: 1
           anything left open in the record: nothing

Design A is not slower and not more expensive. It made exactly one model call, like B, and paid exactly the same. It just left a hole, and the checker names it: toolCall c1 has no toolResult. One call in, exactly one result out, right after it was lesson 4's rule, and it turns out not to care why you stopped. A record with a hole in it can never be sent again, so the user's next prompt is refused, and the one after that, and the session is over. Where a check sits is not a matter of taste.

So: the cap is max_turns, it counts the model calls of this run, and it is looked at once at the top of each turn, before the model is called. Counting this run matters: the caller's list may already hold ten earlier runs, and those turns were somebody else's budget.

Now the other half. The order cards put a message on the list when the limit was reached; this is where you find out what that choice buys, and what the two obvious alternatives cost. Three ways to end a run without an answer, all easy to write. Before you pick, hold three things in mind: what the caller is holding when it is over, what tomorrow's prompt would send, and what somebody reading a saved copy of this session next year would be able to tell.

Raise MaxTurnsExceeded. It is exactly what exceptions are for, and the caller decides what to do about it

An exception is addressed to whoever is on the stack at that instant. This failure has a second audience: the model, on the next prompt, and it is never on the stack. Lesson 4 made that argument about tools; the argument does not change because the thing that stopped is the loop.

Append an assistant message with nothing in it, labelled error, carrying the reason, and return it

The list is the only thing that outlives the function call, so anything that has to be known later has to be on it.

Return quietly. The caller can compare what it got back with what it asked for

That makes the return value the product of the run, which lesson 3 already found it is not. And it asks every caller, for ever, to remember a check that nothing enforces.

All three leave the caller the same five messages, because run_agent appends as it goes; only one leaves a sixth that says why there is no answer. The first two hand tomorrow's prompt a transcript in which nothing went wrong: it simply stops after a tool result, exactly as a transcript looks when the model has read the file and is about to say something useful, so nothing the model is sent tells it the job was cut short. The raise costs one thing more: every caller must now wrap the run, and a caller that forgets is killed by the very thing it was meant to survive. A stop written into the transcript rather than thrown is an in-band error, and the list it is written into is the record: what happened, in order, whether or not it reflects well on anybody.

import harness, lab


class MaxTurnsExceeded(Exception):
    """What the raising design raises."""


def on_limit_raise(messages, max_turns):
    raise MaxTurnsExceeded(f"Agent stopped after max_turns={max_turns}")


def on_limit_quiet(messages, max_turns):
    return None                                  # the run just ends


def on_limit_in_band(messages, max_turns):
    stop = {"role": "assistant", "content": [], "stop_reason": "error",
            "error_message": f"Agent stopped after max_turns={max_turns}"}
    messages.append(stop)
    return stop


def loop(model, system, messages, tools, prompt, *, max_turns, on_limit):
    """Lesson 3's loop, with the limit checked at the top of a turn."""
    turn = 0
    messages.append(harness.user_message(prompt))
    while True:
        turn += 1
        if turn > max_turns:
            return on_limit(messages, max_turns)
        reply = model.complete(system, messages, harness.tool_specs(tools))
        messages.append(reply)
        calls = harness.tool_calls(reply)
        if not calls:
            return reply
        for call in calls:
            messages.append(harness.run_tool(tools, call))


for label, on_limit in (("raise MaxTurnsExceeded", on_limit_raise),
                        ("return quietly", on_limit_quiet),
                        ("append an assistant message", on_limit_in_band)):
    model = lab.ScriptedModel([lab.stuck(lab.call("read", {"path": "missing.py"}))])
    ws = lab.Workspace({"config.py": "PORT = 9090\n"})
    messages = []
    print(label)
    try:
        returned = loop(model, harness.SYSTEM, messages, [harness.make_read_tool(ws)],
                        "Find the port.", max_turns=2, on_limit=on_limit)
        print("  run_agent returned:", "an assistant message" if returned else repr(returned))
    except MaxTurnsExceeded as exc:
        print(f"  run_agent raised: MaxTurnsExceeded({str(exc)!r})")
    print("  the caller's list:", lab.shape(messages))
    print("  it ends on:", lab.show(messages[-1:]).replace("\n", " "))
    print("  tomorrow's prompt would send:",
          lab.shape(messages + [harness.user_message("Look again.")]))

raise MaxTurnsExceeded
  run_agent raised: MaxTurnsExceeded('Agent stopped after max_turns=2')
  the caller's list: U A[c1] R(c1) A[c2] R(c2)
  it ends on: toolResult c2 -> "File not found: missing.py. Files here: config.py"  [is_error]
  tomorrow's prompt would send: U A[c1] R(c1) A[c2] R(c2) U
return quietly
  run_agent returned: None
  the caller's list: U A[c1] R(c1) A[c2] R(c2)
  it ends on: toolResult c2 -> "File not found: missing.py. Files here: config.py"  [is_error]
  tomorrow's prompt would send: U A[c1] R(c1) A[c2] R(c2) U
append an assistant message
  run_agent returned: an assistant message
  the caller's list: U A[c1] R(c1) A[c2] R(c2) A(error)
  it ends on: assistant -> (nothing)  [error: Agent stopped after max_turns=2]
  tomorrow's prompt would send: U A[c1] R(c1) A[c2] R(c2) A(error) U

Something else can end a run, and it is not your fault or the model's. Your request goes out; the provider answers 503 overloaded and nothing else. No text was generated, there is no reply to append, and the transcript stops mid-air. Same principle as the limit: what goes in the transcript, and what does the loop do next? Two or three sentences.

You have just decided what a stop should look like. Ask what is genuinely different about this one — and in particular, whether any of the difference is visible to the person who types the next prompt.

Nothing is different. It is a fact about this conversation that the conversation has to carry, so it is an assistant message with nothing in it, labelled error, holding the provider's own words, appended like any other message; and then the run is over, before any tool runs. Two ways to end badly, one door out. The reason the door is the same is worth saying plainly: whatever ends a run, the next prompt is going to be sent on top of whatever the run left behind.

Common answers, and what each one misses

"Retry it. A 503 usually clears in a second." Often true, and retrying belongs somewhere — just not here. Retries run out, and when they do something has to happen, and this is that something. [general] A provider adapter that retries hides the failures it recovers from; the loop only ever sees the final one.
"Raise it. The caller asked for an answer and there is no answer." The rung above priced that: the caller keeps the list either way, and nobody is left holding the reason.
"Put in a user message saying the provider failed." Then the transcript says you said it. A role tag is a claim about who spoke, and a model that takes it at face value may well apologise to you for its own provider's outage.

So you write it down, faithfully. The list is now U A(error): the prompt, and an assistant message with nothing in it holding 503 overloaded. The user, who has been watching a spinner, types "try again". Your loop appends it and sends the list. What comes back?

The answer. A message with nothing in it says nothing, so it costs nothing and changes nothing

That has what you store and what you send being one list, and the list being nobody's business but yours. It is going over a wire to something with rules.

A 400. The request is refused before the model reads a word of it — and so is every prompt after that one

The refusal is about a message that will still be there tomorrow.

A 400 the first time, and that is your own doing for recording a message the model never sent. Take it back out of the list and the problem is gone

It does work, and look at what it costs: the only line in the session that says the user asked and got nothing back. Deleting the evidence of a failure is how a session comes to look like it went fine.

A provider will not take an assistant turn with nothing in it, so one 503 poisons the list for good: the same 400, prompt after prompt, and the script's remaining answer never gets used. Deleting the empty message clears it and throws away the only record that the run was interrupted. The last block does neither. It builds a different list for the request, leaving that message out, and leaves the list alone — and the answer comes straight back.

That is the split, and it will not go away again: the record is what happened; the view is what you send, worked out from the record, fresh, every time.

Where this comes from: Tau's loop keeps failures in its durable history and builds a separate list for the provider, because "an empty failed or aborted turn is not model context and must not poison the next request" (src/tau_agent/loop.py:186-191). The behaviour is pinned by a test that asserts the failed message is still in the transcript and not in the next request (tests/test_agent_loop.py:341-380).

import harness, lab

model = lab.ScriptedModel([lab.fail("503 overloaded"), lab.say("The port is 9090.")])
messages = []


def answer(reply):
    """What came back: the model's words, or the reason the reply failed."""
    return reply.get("error_message") or harness.text_of(reply)


harness.run_agent(model, harness.SYSTEM, messages, [], "Find the port.")
print("prompt 1 'Find the port.'")
print("  the record now:", lab.shape(messages))
print("  it ends on:", lab.show(messages[-1:]))

for prompt in ("try again", "please, the port"):
    reply = harness.run_agent(model, harness.SYSTEM, messages, [], prompt)
    print(f"prompt {len(model.calls)} {prompt!r}")
    print("  sent:", lab.shape(model.calls[-1].messages))
    print("  back:", answer(reply))

print("script steps never used:", model.remaining)

# The same conversation, in a list built for the request: the message with
# nothing in it is left out. The record is not touched.
view = [m for m in messages if m["content"] or m["role"] != "assistant"]
print()
print("sending", lab.shape(view), "instead of", lab.shape(messages))
print("  back:", answer(model.complete(harness.SYSTEM, view, [])))
print("  the record still holds:", lab.shape(messages))

prompt 1 'Find the port.'
  the record now: U A(error)
  it ends on: assistant -> (nothing)  [error: 503 overloaded]
prompt 2 'try again'
  sent: U A(error) U
  back: 400 invalid_request: messages[1]: assistant message has empty content
prompt 3 'please, the port'
  sent: U A(error) U A(error) U
  back: 400 invalid_request: messages[1]: assistant message has empty content
script steps never used: 1

sending U U U instead of U A(error) U A(error) U A(error)
  back: The port is 9090.
  the record still holds: U A(error) U A(error) U A(error)

One opinion, and there is no right answer to it. You are shipping run_agent as a library. With max_turns=None as the default, an agent runs uncapped unless the caller asks for a cap. Tau ships that way. Tap every consequence that is actually true of that default, then, in the line below, say what you would ship.

A caller who never passes max_turns can be billed for a run that never ends.
A long job that legitimately needs forty turns is never cut short by a number somebody else guessed.
A caller who wants a cap has to edit the library to get one.
With a default cap in place, a runaway becomes impossible.

The first two are both true at once, which is why this is a judgement and not a bug. The third is not: the argument is right there in the signature. The fourth is the one worth arguing about — a cap bounds turns, not money, and one turn that reads a 5,000-line log can cost more than forty small ones. That is lesson 6.

[general] Harnesses that run unattended usually carry a cap, a spend limit, or both: the price of guessing the number too low is an annoyed user, and the price of not guessing is an invoice. Tau declares the argument and defaults it to None (src/tau_agent/harness.py:44), and no code in Tau's own source ever sets it, so the coding agent it ships is uncapped.

Build: three exits, one door

No new Python today. One thing worth naming: message.get("stop_reason") returns None instead of raising when a message has no such key, and some of the messages on the list — a toolResult, a user message — do not have one.

Three gaps in harness.py, about 20 lines in total. The signatures and docstrings are there; the bodies are yours.

error_message(text), in region 1: an assistant message with content empty, stop_reason "error", and text in error_message. The words go in that key, not in a content block: the model never said them.
run_agent has gained max_turns=None. Three changes to lesson 3's body. At the top of a turn, before the model is called: if this run has already made max_turns model calls, the reply is error_message(f"Agent stopped after max_turns={max_turns}") instead of a real one. A reply whose stop_reason is "error" is appended like any other and ends the run, before any tool runs. And the model is sent context_for_model(messages), never the list itself.
Count this run's model calls in a local counter. The caller's list may already hold three earlier runs, and those are not yours to count. max_turns=None means no limit at all.
context_for_model(messages), at the bottom of region 3: a new list holding everything except assistant messages that failed and are empty. A failed reply that got some text out stays in. An empty toolResult stays in: it is still the answer to its call. The list it was given comes back untouched.

The tests run two prompts on one list, let a thirteen-call job run to the end with no cap, cut a reply off half way through a sentence, and check the record after every exit. Nothing is allowed to raise.

Two of your three exits end in exactly the same two lines: a message goes on the list, and the run is over. Which lines of your loop already do that for an ordinary answer? And if the limit were looked at after the model replied rather than before, who would be left to answer the calls that reply asked for?
Give the run a counter of its own, starting at zero, and raise it at the top of the while, before anything else happens. There, decide where the reply comes from: the model, or error_message. Append it either way — and then the limit and the 503 are the same case, because both are a reply whose stop_reason is "error", and one if ends the run for both, above the line that looks for tool calls. context_for_model is one list comprehension over messages that returns a new list. Read stop_reason with .get, so a message that has no such key is kept. Call it on the line that calls the model, inside the loop, so the view is rebuilt for every request: the model has to see the result of the tool it just asked for.

Nearly the code:

run_agent(..., max_turns=None):
    turn = 0
    append the prompt to messages
    loop for ever:
        turn = turn + 1
        if there is a limit and turn is past it:
            reply = error_message("Agent stopped after max_turns=N")
        else:
            reply = the model, sent the view of messages
        append reply
        if reply's stop_reason is "error":
            return reply
        calls = the reply's tool calls
        if there are none: return reply
        for each call: append run_tool(tools, call)

context_for_model(messages):
    a new list of every message except the ones that are
    role assistant AND content empty AND stop_reason "error"

Three ways out, and all three leave a transcript the next prompt can be sent on top of. The limit and the 503 needed one exit between them, not two: the limit is a failed reply that never went over the wire, so the same two lines end the run for both. The last test stopped a run at the limit, took a new prompt on the same list, and got a real answer back.

What changed since lesson 04

The line-by-line diff needs JavaScript. The whole file this exercise starts from is printed at the end of it.

error_message(text) is an assistant message with no content, stop_reason "error", and the text in error_message.
max_turns=1 against a stuck model: one model call, its tool call answered, then an in-band stop.
max_turns=3 allows exactly three model calls, in every run, however long the list already is.
max_turns=None means no limit: twelve requests in a row are all served, and a limit of 20 changes nothing.
The view is worked out again for every model call: the model sees the result of the tool it just asked for.
The provider answers 503: the failed reply goes on the record and the run ends without raising.
After a 503 the user says "try again": the failed message stays on the record and is not sent.
A reply that failed after producing text is sent back to the model like any other message.
context_for_model drops assistant messages that failed and are empty, keeps the rest, and never edits the record.
A run stopped by max_turns leaves a list the next prompt is accepted on, and the model recovers.

# == 1 messages == # A conversation is a list of messages. Each message is a dict tagged with the role that spoke. SYSTEM = "You are a helpful assistant." # standing instructions, sent with every call def user_message(text): """A message from the user. Its content is a plain string.""" return {"role": "user", "content": text} def text_of(message): """The words of an assistant message. Its content is a list of blocks, not a string (lesson 2 explains why); the words are in the blocks of type "text".""" return "".join(block["text"] for block in message["content"] if block["type"] == "text") def tool_calls(message): """The tool calls an assistant message asks for, in the order it asked. Worked out from `content` on every call and stored nowhere else.""" return [block for block in message["content"] if block["type"] == "toolCall"] def error_message(text): """An assistant message that says why a run stopped: nothing in `content`, stop_reason "error", and the reason in `error_message`. A stop is written into the record, not raised.""" # --- your code --- # error_message("boom") == {"role": "assistant", "content": [], "stop_reason": "error", # "error_message": "boom"} # --- end --- # == 2 tools == # A tool is a dict. "name", "description" and "parameters" are advertised to the model. # "execute" is a function, execute(arguments) -> str, that only this program ever runs. def str_arg(arguments, name): """The string argument `name`, checked. The schema asked the model for a string and nobody made it send one, so every tool reads its arguments through here. Raises ValueError("<name> must be a string") when it is missing or not a string.""" value = arguments.get(name) if not isinstance(value, str): raise ValueError(f"{name} must be a string") return value def tool_specs(tools): """What the model is told about the tools: everything except the function.""" return [{"name": tool["name"], "description": tool["description"], "parameters": tool["parameters"]} for tool in tools] def run_tool(tools, call): """Run one tool call and return the toolResult message that answers it. One call in, exactly one result out, on every path. Tools just raise; this is the one place that turns a failure into a result the model can read. is_error means the tool could not do its job. It never raises Exception.""" tools_by_name = {tool["name"]: tool for tool in tools} tool = tools_by_name.get(call["name"]) if tool is None: content, is_error = f"Tool {call['name']} not found", True else: try: content, is_error = tool["execute"](call["arguments"]), False except Exception as exc: content, is_error = str(exc), True return {"role": "toolResult", "tool_call_id": call["id"], "tool_name": call["name"], "content": content, "is_error": is_error} def make_read_tool(ws): """The `read` tool, over the workspace `ws`.""" def execute(arguments): path = str_arg(arguments, "path") if not ws.exists(path): raise FileNotFoundError( f"File not found: {path}. Files here: {', '.join(ws.listdir('.'))}") return ws.read_text(path) return { "name": "read", "description": "Read a text file and return its contents.", "parameters": { "type": "object", "properties": {"path": {"type": "string", "description": "Path of the file to read."}}, "required": ["path"], }, "execute": execute, } def make_write_tool(ws): """The `write` tool, over the workspace `ws`: create a file, or replace all of it.""" def execute(arguments): path, content = str_arg(arguments, "path"), str_arg(arguments, "content") ws.write_text(path, content) return f"Successfully wrote to {path}." return { "name": "write", "description": "Write a text file. Creates the file, or replaces its whole contents.", "parameters": { "type": "object", "properties": {"path": {"type": "string", "description": "Path of the file to write."}, "content": {"type": "string", "description": "The new contents."}}, "required": ["path", "content"], }, "execute": execute, } def make_bash_tool(shell): """The `bash` tool: run one command line in `shell` and return what it printed. A command that fails has still been run: its exit code is news for the model, not a tool error, so it comes back as an ordinary result that ends with a status line.""" def execute(arguments): output, code = shell.run(str_arg(arguments, "command")) if code != 0: output += f"\nCommand exited with code {code}" return output return { "name": "bash", "description": "Run a shell command and return its output (stdout and stderr together).", "parameters": { "type": "object", "properties": {"command": {"type": "string", "description": "The command line."}}, "required": ["command"], }, "execute": execute, } # == 3 loop == # The code that talks to the model. The model keeps nothing between calls. def run_agent(model, system, messages, tools, prompt, *, max_turns=None): """Put `prompt` to the model and keep going until it stops asking for tools, a reply comes back failed (stop_reason "error"), or `max_turns` model calls have been made (None: no limit). Returns the final assistant message. `messages` is the caller's list: everything that happens is appended to it, so the caller holds the record even if the run dies. Every exit leaves a record that the next prompt can be sent on top of.""" # --- your code --- # This is lesson 3's body. Three changes: # - at the top of a turn, before calling the model: if max_turns model calls have already # been made in this run, append error_message(f"Agent stopped after max_turns={max_turns}") # and return it. Never check after the model's reply: its calls would go unanswered. # - a reply whose stop_reason is "error" is appended like any other, and ends the run # before any tool runs # - the model is sent context_for_model(messages), never the raw record # One of the hidden tests, against a model that asks for the same file for ever: # run_agent(model, SYSTEM, messages, tools, "Fix it.", max_turns=1) # assert len(model.calls) == 1 and lab.shape(messages) == "U A[c1] R(c1) A(error)" messages.append(user_message(prompt)) while True: reply = model.complete(system, messages, tool_specs(tools)) messages.append(reply) calls = tool_calls(reply) if not calls: return reply for call in calls: messages.append(run_tool(tools, call)) # --- end --- def context_for_model(messages): """What the model is sent: the record, minus assistant messages that failed and are empty. They are true, so the record keeps them; a provider refuses an empty message, so the view leaves them out. A failed reply that got some text out stays in. Returns a new list.""" # --- your code --- # record = [user_message("hi"), error_message("503 overloaded"), user_message("try again")] # context_for_model(record) == [record[0], record[2]], and `record` still holds all three # --- end ---Running code needs JavaScript. Above is the file this exercise starts from (harness.py). You can run its tests on your own machine: python lab/lab_check.py --harness harness.py --lesson 5 tests.py

Your harness, a workspace with three real files, and a model convinced the port is in settings.py, which does not exist. Three turns, then the limit. Then the user says the file is not there and to try something else — on the same list, which still holds the stop. Watch what the second run is allowed to send.

import harness, lab

FILES = {"config.py": "PORT = 9090\nDEBUG = False\n",
         "README.md": "Start it with python -m app.\n",
         "app/main.py": "from config import PORT\n"}


def confused(request):
    """Asks for settings.py for ever. Once a user message says to try something
    else, it looks at config.py instead, and answers from what it reads."""
    if "try something else" in request.user_text.lower():
        if request.last_result and "PORT" in request.last_result["content"]:
            return lab.say("The port is 9090. It is set in config.py, not settings.py.")
        return lab.reply(lab.call("read", {"path": "config.py"}))
    return lab.reply(lab.call("read", {"path": "settings.py"}))


ws = lab.Workspace(FILES)
model = lab.ScriptedModel([lab.forever(confused)])
tools = [harness.make_read_tool(ws)]
messages = []

harness.run_agent(model, harness.SYSTEM, messages, tools,
                  "What port does this app use?", max_turns=3)
print("run 1  model calls:", len(model.calls), " record:", lab.shape(messages))
print("       ends on:", lab.show(messages[-1:]))

reply = harness.run_agent(model, harness.SYSTEM, messages, tools,
                          "settings.py does not exist. Try something else.", max_turns=3)
print("run 2  model calls:", len(model.calls), " record:", lab.shape(messages))
print("       the agent says:", harness.text_of(reply))
print()
print("the request that got that answer:", lab.shape(model.calls[-1].messages))
print("the record it was built from: ", lab.shape(messages[:-1]))
print("anything wrong with the record:", lab.validate(messages, record=True))
print("paid for, in input tokens:", model.bill)

run 1  model calls: 3  record: U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3) A(error)
       ends on: assistant -> (nothing)  [error: Agent stopped after max_turns=3]
run 2  model calls: 5  record: U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3) A(error) U A[c4] R(c4) A
       the agent says: The port is 9090. It is set in config.py, not settings.py.

the request that got that answer: U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3) U A[c4] R(c4)
the record it was built from:  U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3) A(error) U A[c4] R(c4)
anything wrong with the record: []
paid for, in input tokens: 1170

The stop message is on the record and is not in the request. Nobody deleted anything, nobody mended anything, and the second run needed no special case for "the last one was cut off": it sent the view, and the view is a list a provider will take. Under twelve hundred input tokens in all, and the user got an answer.

You have two lists now where you had one. One is added to and never edited; the other is worked out from it afresh for every request, and is thrown away as soon as the request is sent. Today it leaves out one kind of message. It will learn more rules, and the first list will not change by a line: lesson 10 gives that split its name.

Figure 5.1 Lesson 3's machine with every way out drawn on it. Under each exit: the last message it leaves on the list.

Exit 1, the one you have had since lesson 3: the reply asks for no tools, so return it. The list ends on an ordinary assistant message.
Exit 2, the limit. It leaves from the top of the crank, before the model call, and the list ends on assistant · error, Agent stopped after max_turns=1.
Exit 3, the failed reply. It is appended, and then the same if that ends the run for the limit ends it here: the list ends on assistant · error, 503 overloaded.
Not an exit: a tool name that does not exist. That is a result with is_error set, the loop goes round again, and the model gets to fix it. Lesson 4's rule, unchanged.
The exit you must never build, crossed out: a limit checked between the reply and its tools. It leaves c1 open — the slot where toolResult · c1 belongs stays empty — and a record with a hole in it can never be sent again.

Hold a conversation: it knows what was said, and who said it.
Run a tool the model asks for and show it the result.
Keep going until the model stops asking.
Tell the model when a tool fails, and carry on.
Stop a runaway, and survive a provider failure.

In your own words, a sentence or two. Your run can now end three ways: the model stops asking, the limit is reached, the provider falls over. Say what the three have in common — and what it costs you when an exit breaks the pattern.

The first cell on this page ran an exit that broke it. Name what the caller could not do afterwards, and for how long.

However a run ends, the last thing it does is write down what happened, in a list the next prompt can be sent on top of. One rule, not three, which is why the limit and the 503 leave by one door, and why a token budget would leave by the same one. Break it — leave a call unanswered, or keep the reason in an exception nobody stores — and the damage is not to this run, which is over in any case. It is to every conversation that follows on that list, which is all of them.

Common answers, and what each one misses

"All three return the final message." True of your function, and it is the part that matters least. The caller held the list all along; lesson 3 settled that the return value is a convenience and not the product of a run.
"All three append an assistant message." Close, and it misses the test. The question is never "did you append something", it is "would a provider take this list tomorrow" — which is why the limit is looked at before the model call and not after. The appending was never the hard part.
"All three leave the caller to check what came back." That is the raise in a quieter coat. A rule that every caller has to remember is a rule some caller will not, and the one who forgets is the one who most needed it.

Stopping is not an accident to throw; it is a fact to write down.

Tau checks the limit at the top of a turn, writes the stop into the transcript as an assistant message with no content, and ends a failed reply with the same two lines. Word for word the same stop text as yours.

turn += 1
if max_turns is not None and turn > max_turns:
    reply = error_message(f"Agent stopped after max_turns={max_turns}")
else:
    reply = model.complete(system, context_for_model(messages), tool_specs(tools))
messages.append(reply)
if reply["stop_reason"] == "error":
    return reply

if max_turns is not None and turn > max_turns:
    error = _error_message(model, f"Agent stopped after max_turns={max_turns}")
    messages.append(error)
...
    yield TurnEndEvent(message=error)
    yield AgentEndEvent(messages=new_messages)
    return
...
messages.append(assistant)
if assistant.stop_reason in {"error", "aborted"}:
    yield TurnEndEvent(message=assistant)
    yield AgentEndEvent(messages=new_messages)
    return

The same shapes.

The stop message is built by one small function that puts the words in error_message and leaves content empty, exactly as yours does (src/tau_agent/loop.py:370-376).
The turn counter is local to the run and starts at 1, so a transcript that already holds ten turns does not eat this run's budget (src/tau_agent/loop.py:94).
The view is a separate function, called on every request, that drops assistant messages that failed and have no content (src/tau_agent/loop.py:185-201). Its docstring is the other half of this lesson in Tau's words: failures are kept for diagnostics, and an empty failed turn "is not model context".

Tau is async: until lesson 15, read async for as for and await f(x) as f(x).

What Tau adds. Two more ways for a run to end without an answer. A max_turns below 1 ends the run at once with a message about the argument, max_turns must be at least 1 (src/tau_agent/loop.py:83-91). Yours also makes no model call for max_turns=0 — it counts before it calls — but it writes down Agent stopped after max_turns=0, which is true and says nothing about the argument being nonsense; hand it -1 and it does the same. And a provider that streams nothing at all — no message, no error — gets an assistant message invented for it, Provider produced no assistant message, so that even a provider that simply stops talking leaves a transcript (src/tau_agent/loop.py:141-144).

Both of Tau's checks — the one that ends the run, and the one that filters the view — cover aborted beside error (src/tau_agent/loop.py:148, src/tau_agent/loop.py:192-199). Nothing in your code produces an aborted reply yet. In lesson 15 something will, and those same two lines of yours — the exit and the filter — will each gain a word. Tau's view does one more job than yours, on the same request: it repairs the pairing of calls and results before handing the list over (src/tau_agent/loop.py:201). That is lesson 10, and it is the second rule the view learns.

What Tau declares and does not use. max_turns is a configuration field that defaults to None (src/tau_agent/harness.py:44), and no code in Tau's own source ever sets it. The cap exists for whoever embeds the harness; the coding agent Tau ships is uncapped, which is the answer the opinion rung was really about.

Where yours is weaker. A 503 that reaches Tau's loop has already been given up on. Retries with backoff sit inside the provider adapter, below the loop (src/tau_ai/retry.py:15-20), and the Anthropic adapter retries a 429 and anything from 500 up until it runs out of attempts (src/tau_ai/anthropic.py:377-380). Yours has no retry and no cost budget; the Transfer question below is the shape the second one would take. And your record and your view are two plain lists holding the same dicts, so a caller who edits a message in one edits it in both. Tau's view shares its messages exactly the same way; what it does not share is the shape, because a Tau message is a declared model that refuses a field nobody declared, where a dict will take any key you hand it.

src/tau_agent/loop.py:112-120,146-151 · pinned to commit 9fe6a71 · view on GitHub

Transfer. A turn cap is a poor proxy for what you actually care about, which is money. So replace it: the run may spend at most BUDGET input tokens, and every reply tells you what its request cost, in reply["usage"]["input"]. Where does the check go, and what does it leave behind? Pick a placement, give your reason in one line, then run all three.

Right after the reply arrives, where its cost is: add it up, and stop as soon as the total is over budget

It is the one place the new number is already in your hand, which is exactly why it is tempting. Count the results against the calls in the record it leaves.

At the top of a turn, where the turn count was, using what the run has spent so far; append an assistant message saying what was spent and stop

The reason is not about tokens. The shape of the check has to fit the shape of a turn.

In a wrapper around the model call, which raises BudgetExceeded when the run has spent too much. One place, and no loop to change

The placement is fine — that wrapper runs at the same moment as (b). What it leaves behind is the problem, and it is the same problem the rung about MaxTurnsExceeded had.

Checked after the reply, the budget runs out on a reply that had just asked for c3, and c3 is never answered: three calls, two results, and the strict validator names the hole before the next prompt can even go out. Checked at the top of a turn, the record is balanced, and the next prompt is valid. The wrapper stops at the right moment and leaves nothing behind, so the transcript ends mid-conversation with no sign that a budget was ever involved. The number changed; where you look at it, and what you write down when you do, did not.

import harness, lab

BUDGET = 400        # input tokens this run may spend


class BudgetExceeded(Exception):
    """What the wrapper raises."""


def loop(model, system, messages, tools, prompt, *, placement):
    """Your lesson 5 loop with max_turns replaced by a token budget."""
    spent = 0
    messages.append(harness.user_message(prompt))
    while True:
        if placement == "wrapper" and spent > BUDGET:
            raise BudgetExceeded(f"spent {spent} of {BUDGET} input tokens")
        if placement == "top" and spent > BUDGET:
            stop = harness.error_message(
                f"Agent stopped after spending {spent} of {BUDGET} input tokens")
            messages.append(stop)
            return stop
        reply = model.complete(system, harness.context_for_model(messages),
                               harness.tool_specs(tools))
        messages.append(reply)
        spent += reply["usage"]["input"]
        if placement == "after" and spent > BUDGET:
            stop = harness.error_message(
                f"Agent stopped after spending {spent} of {BUDGET} input tokens")
            messages.append(stop)
            return stop
        calls = harness.tool_calls(reply)
        if not calls:
            return reply
        for call in calls:
            messages.append(harness.run_tool(tools, call))


for placement in ("after", "top", "wrapper"):
    model = lab.ScriptedModel([lab.stuck(lab.call("read", {"path": "missing.py"}))])
    ws = lab.Workspace({"config.py": "PORT = 9090\n"})
    messages = []
    print("checked", placement)
    try:
        loop(model, harness.SYSTEM, messages, [harness.make_read_tool(ws)],
             "Find the port.", placement=placement)
    except BudgetExceeded as exc:
        print(f"  raised BudgetExceeded({str(exc)!r})")
    asked = sum(len(harness.tool_calls(m)) for m in messages if m["role"] == "assistant")
    answered = sum(1 for m in messages if m["role"] == "toolResult")
    print("  record:", lab.shape(messages))
    print("  calls asked for:", asked, " results recorded:", answered)
    following = harness.context_for_model(messages) + [harness.user_message("Look again.")]
    print("  the next prompt would send:", lab.shape(following))
    print("  a strict provider says:", lab.validate(following) or "nothing; it is valid")

checked after
  record: U A[c1] R(c1) A[c2] R(c2) A[c3] A(error)
  calls asked for: 3  results recorded: 2
  the next prompt would send: U A[c1] R(c1) A[c2] R(c2) A[c3] U
  a strict provider says: ['messages[5]: toolCall c3 has no toolResult']
checked top
  record: U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3) A(error)
  calls asked for: 3  results recorded: 3
  the next prompt would send: U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3) U
  a strict provider says: nothing; it is valid
checked wrapper
  raised BudgetExceeded('spent 445 of 400 input tokens')
  record: U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3)
  calls asked for: 3  results recorded: 3
  the next prompt would send: U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3) U
  a strict provider says: nothing; it is valid

You hit

a model that asked for the same missing file fifty times running, and a provider whose one failure made every later prompt fail too

You built

error_message, the max_turns check at the top of a turn, one exit for failed replies, and context_for_model

The principle

every exit leaves a transcript you could send again, because a stop is a fact to write down, not an accident to throw

Your harness now

user_message
tool_calls
str_arg
run_tool
run_agent
error_message
context_for_model

Still open

Nothing stops one tool result being a 5,000-line file, and every turn after it pays for that file again. A cap on turns does not help. Lesson 6.

Previous 04 Tell the model what went wrong Next 06 The firehose Your agent reads a 5,000-line log. How many times do you pay for it?