Harness from First Principles [03/16]

03 · The model can only talk

Turn the crank

The bug is three files deep. Your chat makes two model calls, reads one file, and hands back an empty answer.

~40 min · 1 lab · builds on 02 Words are not deeds

This lesson builds on lesson 02. New here? Start at lesson 01, or carry on: every lab is self-contained.

A tool you build is an ordinary dict, so nothing stops you adding keys to it. Here is the count_words tool with a fifth key, timeout, that only your own code would ever read. The cell prints the keys of your dict, and then the keys of the tool as it arrived in the model's request. What does the second line say?

The same five. It is one dict, and the request carries it
Two of those five cannot travel. execute is a function, and there is no way to write a function down in a request; timeout is yours, and nothing outside your process has ever heard of it.
 Three: name, description, parameters
tool_specs decides what goes out, and it names those three. Everything else in the dict stays home, wanted or not.
 Four: everything except the function, since the rest is harmless text
Harmless, but not free: every key you send is text in every request, on every turn, and the model would have to guess what timeout means. What travels is what you chose to send, not what happens to be in the dict.

Your dict has five keys and the request carried three. The tool's spec is a thing you build for sending, not the tool itself. Today you find out how many times that spec goes out during one job.

import harness, lab

def count_words(arguments):
    return str(len(arguments["text"].split()))

tool = {"name": "count_words",
        "description": "Count the words in a text.",
        "parameters": {"type": "object",
                       "properties": {"text": {"type": "string"}},
                       "required": ["text"]},
        "execute": count_words,
        "timeout": 30}              # a fifth key, for your own use

model = lab.ScriptedModel([lab.say("Noted.")])
harness.chat(model, [], "How long is this sentence?", [tool])

print("your dict has:   ", list(tool))
print("the request held:", list(model.calls[0].tools[0]))
your dict has:    ['name', 'description', 'parameters', 'execute', 'timeout']
the request held: ['name', 'description', 'parameters']

Someone puts your lesson 1 chat behind a web server, with one messages list created when the process starts. User A types "My name is Ada." User B, in another browser, on another continent, asks "What is my name?" What is user B told?

"I don't know your name." The model has no idea who is asking, so it cannot claim to know
It does not know who is asking, and that is the problem: it answers the request in front of it. The request in front of it begins with Ada.
 "Your name is Ada."
One list, one process, and every request carries all of it.
 "Your name is Ada." — but only because the model remembered the earlier call from the same server
Right answer, wrong machinery. Restart the process and B is told nothing, although the model is the same model. The leak is in the list, not in the model.

Three messages went out, and Ada was in the first. Whoever holds the list holds the conversation, which is convenient until two conversations share one. Lesson 8 is about the ownership; today the list grows much faster than it did in lesson 1.

import harness, lab

model = lab.ScriptedModel([lab.forgetful()])
messages = []                    # one list for the whole server

# two people, two browser tabs, one process
harness.chat(model, messages, "My name is Ada.")          # user A
answer = harness.chat(model, messages, "What is my name?")  # user B

print("user B is told:", answer)
print("user B's request carried", len(model.calls[-1].messages),
      "messages")
user B is told: Your name is Ada.
user B's request carried 3 messages

Three files deep

main.py prints 0 where it should print 42. It imports scale from util.py, which imports FACTOR from const.py, and const.py is where the bug is. Nobody can see that from main.py: you have to read one file to learn the name of the next.

Your chat from lesson 2 has everything it needs. The three files are on the workspace, the read tool is in the tool list, and the model is scripted to work the way a debugger works: look at main.py, then util.py, then const.py, then say what is wrong.

You call chat(model, messages, "scale(21) prints 0. Find the bug.", tools) once. How many times does chat call the model?

Two. That is what you built: one call, and if the reply asks for tools, run them and call once more. It was one round trip when you wrote it and it is one round trip now, whatever the job needs. The cell below runs it and then turns the crank by hand, twice, so that you can see what the job actually costs.

Two calls, one file read, and the answer that comes back is the empty string. The job needs three files. Somebody has to decide that the work is finished. Who?

The model, by writing a reply that asks for nothing more
It is the only party that knows what it still needs, because it is the only party doing the job. Your code can see whether the reply asks for anything; it cannot see why.
 The API, through stop_reason: each reply is labelled, and the label says whether there is more to come
That treats the label as a message from the provider about your job. It is a note about how the model's own output ended. The section after next is about what happens when you take it for an instruction.
 You do, up front: a number of rounds, like max_rounds=5, which is the sort of setting a library would hand you
That is the belief that the going-round is somebody else's clever part, arriving as a setting. It is nine lines, and you write them this hour. A ceiling on the rounds is a real thing — it is a safety net for a model that will not stop, which is lesson 5 — but a safety net is not a way of knowing that a job is done.

The cell cranks the handle twice, and the model needed exactly that: util.py, then const.py, then the verdict. Nobody could have known in advance that three files was the number. The four requests cost 85, 162, 233 and 295 input tokens, because each one resends the whole list, as it did in lesson 1, with the tool spec riding along every time. And look at where chat left the record — U A[c1] R(c1) A[c2], ending on a call that nobody answered.

import harness, lab

ws = lab.Workspace({
    "main.py": "from util import scale\n\nprint(scale(21))\n",
    "util.py": "from const import FACTOR\n\n"
               "def scale(n):\n    return n * FACTOR\n",
    "const.py": "FACTOR = 0  # should be 2\n"})
tools = [harness.make_read_tool(ws)]

model = lab.ScriptedModel([          # one step per model call
    lab.reply(lab.text("Let me look."),
              lab.call("read", {"path": "main.py"})),
    lab.reply(lab.call("read", {"path": "util.py"})),
    lab.reply(lab.call("read", {"path": "const.py"})),
    lab.say("The bug is in const.py: FACTOR is 0.")])

messages = []
answer = harness.chat(model, messages,
                      "scale(21) prints 0. Find the bug.", tools)
print("chat returned", repr(answer), "after", len(model.calls),
      "model calls. Files read:", ws.reads)
print("the list ends on:", lab.show(messages[-1:]))

def crank():                   # one more round trip, by hand
    for call in harness.tool_calls(messages[-1]):
        messages.append(harness.run_tool(tools, call))
    messages.append(
        model.complete(harness.SYSTEM, messages,
                       harness.tool_specs(tools)))
    print("cranked by hand: ", lab.show(messages[-1:]))

crank()
crank()
print("model calls:", len(model.calls), "| input tokens per call:",
      [m["usage"]["input"] for m in messages if "usage" in m])
chat returned '' after 2 model calls. Files read: ['main.py']
the list ends on: assistant -> toolCall c2 read({"path": "util.py"})
cranked by hand:  assistant -> toolCall c3 read({"path": "const.py"})
cranked by hand:  assistant -> "The bug is in const.py: FACTOR is 0."
model calls: 4 | input tokens per call: [85, 162, 233, 295]

What makes it go round again

So the shape is not in doubt. Call the model; if the reply asks for tools, run them, append the results, and call again — and again, and again, until it stops asking. That is a while loop, and the only interesting line in it is the condition.

Write that condition as one expression over the last assistant message. You have reply, the message the model just sent. What goes after return?

You wrote a function in lesson 2 that answers exactly this question about a reply. What did it return when the model asked for nothing?

return bool(tool_calls(reply))

Go round again if, and only if, the reply contains tool calls. tool_calls works them out from content every time it is asked, which is why nothing can go stale between asking and running. Put your line into the cell below: it ships with a guess that always says no, and the last column tells you where the guess and the job disagree.

Common answers, and what each one misses

  • "reply["stop_reason"] == "toolUse"." The tempting one, because the label is right there and it is shorter to type. Whether it is true is the next two questions.
  • "len(messages) < 10", or a counter. That answers "have I gone round enough times", which is a question about your patience, not about the job. The model asked for three files here and could have asked for eleven.
  • "The reply has no text yet", or "the model has not said anything final". Run the cell: its very first reply is "Let me look." and a request for main.py. Replies talk and act in the same breath.

Three replies from the debugging script, and a go_on() that always says no. Put your one line in and run it: all three rows should agree.

import harness, lab

def go_on(reply):
    """True if the run must go round again. `reply` is the
    assistant message the model just sent. One expression."""
    return False        # <- your one line goes here

model = lab.ScriptedModel([
    lab.reply(lab.text("Let me look."),
              lab.call("read", {"path": "main.py"})),
    lab.reply(lab.call("read", {"path": "util.py"}),
              lab.call("read", {"path": "const.py"})),
    lab.say("The bug is in const.py: FACTOR is 0.")])

for needed in [True, True, False]:
    reply = model.complete(harness.SYSTEM, [lab.user("Find the bug.")])
    said = bool(go_on(reply))
    print(lab.show([reply], stop_reason=True))
    note = "" if said == needed else "   <- they differ"
    print(f"    go_on says {said}, the job needs {needed}{note}")
assistant -> "Let me look." + toolCall c1 read({"path": "main.py"})  [stop_reason=toolUse]
    go_on says False, the job needs True   <- they differ
assistant -> toolCall c2 read({"path": "util.py"}) + toolCall c3 read({"path": "const.py"})  [stop_reason=toolUse]
    go_on says False, the job needs True   <- they differ
assistant -> "The bug is in const.py: FACTOR is 0."  [stop_reason=stop]
    go_on says False, the job needs False

The label on the box

Every reply arrives with a stop reason: a short word saying why the model stopped writing. toolUse and stop went past in the last cell. It is right there in the message, it is one comparison to write, and it reads like an instruction meant for you.

The next two cells are replies whose label disagrees with what the reply contains — one lie in each direction. The scripts are ours; whether the disagreement matters is what you are about to decide.

The model asks to read const.py — and the reply is labelled stop. Your code trusts the label, so it does not run the tool and does not call again. A minute later the user asks "Well? What is it?". You append their question to the same list and send it. What comes back?

Nothing from the model: the request is refused, because the list holds a call with no result
The refusal names the message index and the call id. Nothing about the new question is wrong; the damage was already in the list.
 The answer. The reply said stop, so that call was never a live request — the model had already finished with it
That has the label describing the content. The block in the content is a request whatever the envelope says, and lesson 2's rule is about the block, not the label.
 The answer, one turn late. The model sees it asked for const.py and never got it, so it asks again
That treats a broken record as an untidy detail the model can work around. The model never sees this list: the shape is checked before the model is given anything to read, so nothing downstream gets the chance to be forgiving.

400 invalid_request: messages[1]: toolCall c1 has no toolResult. You did not break the record by sending the second prompt; you broke it when you stopped without running the tool, and every later prompt on that list gets the same answer. Lesson 2's rule — one call in, exactly one result out — turns out to be a rule about when your loop is allowed to stop.

Where this refusal comes from: [general] hosted providers reject a request in which a call has no result. Tau keeps a function whose whole job is to guarantee that "every tool call has exactly one adjacent result" before a provider sees the list (src/tau_agent/tool_history.py:40-47).

import harness, lab

ws = lab.Workspace({"const.py": "FACTOR = 0  # should be 2\n"})
tools = [harness.make_read_tool(ws)]
specs = harness.tool_specs(tools)

model = lab.ScriptedModel([
    lab.reply(lab.call("read", {"path": "const.py"}),
              stop_reason="stop"),          # the label lies
    lab.say("FACTOR is 0.")])

messages = [lab.user("What is FACTOR in const.py?")]
reply = model.complete(harness.SYSTEM, messages, specs)
messages.append(reply)
print(lab.show([reply], stop_reason=True))

if reply["stop_reason"] == "toolUse":       # trust the label
    for call in harness.tool_calls(reply):
        messages.append(harness.run_tool(tools, call))
print("the run is over. You hold:", lab.shape(messages))

# a minute later, the user asks again
messages.append(lab.user("Well? What is it?"))
reply = model.complete(harness.SYSTEM, messages, specs)
print(lab.show([reply], stop_reason=True))
assistant -> toolCall c1 read({"path": "const.py"})  [stop_reason=stop]
the run is over. You hold: U A[c1]
assistant -> (nothing)  [stop_reason=error]  [error: 400 invalid_request: messages[1]: toolCall c1 has no toolResult]

Now the lie the other way. The reply is one plain sentence, "FACTOR is 0.", with no call in it anywhere — and the label says toolUse. Your code trusts the label again. What does it do, and what does that cost?

It finds nothing to run, calls the model a second time with a list nobody added anything to, and pays for that request
Two requests, one of which you did not mean to send. Read what the model makes of it.
 It fails looking for the call: the label promises one, so the call must be in there somewhere and the code has to dig it out of the text
That is the label as the authority over the content. tool_calls looks in content and returns a list, which is empty here; nothing digs and nothing raises.
 It calls again, which costs nothing: a round with no tools to run does no work
No work and a full bill. A request is the whole list, and the tool specs when there are any, priced by the token whether or not the round turns out to be useful.

tools to run: [], and then a second request carrying U A — the same conversation with the model's own answer on the end. It cost 77 input tokens across the two calls, and all it bought was the model remarking that its answer had come back with nothing new after it. Content beats label, in both directions: the content is what the model produced, the label is a note about how it stopped producing. From here on your loop never reads it.

Where this comes from: Tau does not trust the provider's word either. Before a reply reaches the loop, its stop reason is recomputed, and a reply holding calls is relabelled toolUse whatever arrived on the wire (src/tau_ai/stream.py:80-85, applied at src/tau_ai/stream.py:208-211).

import harness, lab

def puzzled(request):      # it can only react to what it is sent
    return lab.say("My own answer came back to me with nothing "
                   "new after it. FACTOR is still 0.")

model = lab.ScriptedModel([
    lab.reply(lab.text("FACTOR is 0."),
              stop_reason="toolUse"),       # the opposite lie
    puzzled])

messages = [lab.user("What is FACTOR in const.py?")]
reply = model.complete(harness.SYSTEM, messages)
messages.append(reply)
print(lab.show([reply], stop_reason=True))

if reply["stop_reason"] == "toolUse":       # trust the label
    print("tools to run:", harness.tool_calls(reply))
    reply = model.complete(harness.SYSTEM, messages)
    messages.append(reply)
    print(lab.show([reply], stop_reason=True))

print("model calls:", len(model.calls), "| input tokens paid:",
      model.bill, "| request 2 was", lab.shape(model.calls[-1].messages))
assistant -> "FACTOR is 0."  [stop_reason=toolUse]
tools to run: []
assistant -> "My own answer came back to me with nothing new after it. FACTOR is still 0."  [stop_reason=stop]
model calls: 2 | input tokens paid: 77 | request 2 was U A

Which leaves the question of what the label is for. Here is one that is not a lie at all. [general] A model writes until it is done or until it hits the cap on its output, whichever comes first, and the label is how you find out which happened.

Here the model hit the cap: the reply is the stump "The bug is in", no call, labelled length. The loop in the cell is the one you are about to write, and it continues on calls, not on labels. What does it do?

It ends the run and hands the stump back. There was nothing to run
One model call, and a list holding a prompt and a half-finished sentence.
 It goes round once more. length plainly means "unfinished", so the loop asks for the rest
That is the label as an instruction to the loop again, in its most reasonable costume. The script has a second reply waiting; watch whether the run ever reaches it.
 It raises. A truncated answer is a failure, and handing a caller half a sentence is worse than stopping
That makes the loop the judge of the answer's quality. It cannot be: half a sentence is sometimes all that was needed, and the caller can see the label and decide. An exception here would also throw away the record, which the next question is about.

One call, U A, run over. The last two lines of the cell are the caller doing what the loop refuses to do: reading the label and saying so. That is the division of work — the loop decides whether to go round, the caller decides what to tell the user about how it ended. A screen that prints that stump without a word about length is lying by omission.

import harness, lab

model = lab.ScriptedModel([
    lab.reply(lab.text("The bug is in"),   # cut off mid-sentence
              stop_reason="length"),
    lab.say("This reply is here in case anyone asks again.")])

messages = []
final = harness.run_agent(model, "You are a careful debugger.",
                          messages, [], "Find the bug.")

print("model calls:", len(model.calls), "| you hold:",
      lab.shape(messages))
print(lab.show([final], stop_reason=True))
if final["stop_reason"] == "length":        # the caller's business
    print("(cut short: the model hit its output limit)")
model calls: 1 | you hold: U A
assistant -> "The bug is in"  [stop_reason=length]
(cut short: the model hit its output limit)

What the run leaves behind

The loop hands back one message: the final reply, the one that asked for nothing. The first cell's job, once you had cranked it to the end, made eight. So where are the other seven, and what happens to them on a day when the job does not finish?

In the cell below, const.py is missing from the workspace. On the third turn the read tool raises FileNotFoundError, nothing catches it inside the loop, and run_agent never returns at all.

The tool raises on turn 3. The exception comes out of run_agent and the caller catches it. What does the caller have afterwards?

Six messages, in the list it passed in: everything that happened, up to and including the reply that asked for the file that is not there
The loop appended each one the moment it existed, so a run that dies leaves exactly as much as it got through.
 Nothing usable. run_agent raised instead of returning, so there is no result, and two turns of work are gone
That makes the return value the product of the run. It is a convenience — the last message, which is also the last item of the list. Two model calls have been paid for; losing them would be a choice, not a consequence.
 The prompt and nothing else. The loop keeps the transcript as it goes and hands it over when it returns
That puts the conversation inside the loop. The list the caller passed in is the one the loop appends to, line by line, and it was never a private copy. Lesson 8 asks who should own it; today the answer is simply "not the loop".

Read the second line of the output carefully: final is still the None it was initialised to, because run_agent never returned. And the caller's list holds U A[c1] R(c1) A[c2] R(c2) A[c3], the whole run up to the crash. The product of a run is the list you handed in; the return value is the last line of it, offered for convenience. Notice where it stops — on a call with no result, the exact wreck that lesson 4 has to clear up.

import harness, lab

ws = lab.Workspace({          # const.py is not on this disk
    "main.py": "from util import scale\n\nprint(scale(21))\n",
    "util.py": "from const import FACTOR\n\n"
               "def scale(n):\n    return n * FACTOR\n"})
tools = [harness.make_read_tool(ws)]

model = lab.ScriptedModel([
    lab.reply(lab.call("read", {"path": "main.py"})),
    lab.reply(lab.call("read", {"path": "util.py"})),
    lab.reply(lab.call("read", {"path": "const.py"})),
    lab.say("The bug is in const.py: FACTOR is 0.")])

messages = []
final = None
try:
    final = harness.run_agent(model, "You are a careful debugger.",
                              messages, tools, "Find the bug.")
except FileNotFoundError:
    print("the run died on turn 3, inside the read tool")

print("run_agent returned:", final)
print("the caller holds:  ", lab.shape(messages))
the run died on turn 3, inside the read tool
run_agent returned: None
the caller holds:   U A[c1] R(c1) A[c2] R(c2) A[c3]

Build: turn the crank

Everything is decided. Call the model with the whole list. Append the reply. If it asks for nothing, stop and hand it back. Otherwise run what it asked for, append each result, and go round. Here is the same sentence as a machine, one step at a time, on a one-file job.

The crank: one tool-using run of run_agent run_agent appends the prompt to the caller's list, messages, then repeats: send the whole list to model.complete, append the reply, and look for tool calls in it. If there are some, run each one, append its toolResult, and go again with the longer list. If there are none, return the reply. Here the run is user, assistant with toolCall c1, toolResult c1, assistant: two model calls. messagesthe caller's list 0 userWhat is in config.py? 1 assistantread(path="config.py")c1 2 toolResult · c1DEBUG = True 3 assistantIt sets DEBUG to True. model.complete()ScriptedModel run_tool()tools: read run_agentgo again append-only: it grows downward whole list, every call 1 message3 messages append reply reply tool_calls(reply)? none:return reply some:run each append result none: return reply tool_calls(reply)? whole list,every call 1 message3 messages appendreply appendresult

Figure 3.1 The crank. The job in the figure needs one file, so the handle turns once; the job in your cells needed three. Nothing in the machine knows the difference.

  1. The prompt goes on the record first: messages is the caller's list, and it grows downward from here.
  2. The whole list goes to model.complete() — one message, this first time.
  3. The reply is appended: a sentence and a request for config.py, tagged c1.
  4. tool_calls(reply) finds one, so run_tool() runs it and its result is appended, tied to c1.
  5. The crank: round again with the same two lines. The request now carries three messages, and you pay for all three.
  6. This reply asks for nothing. That is the exit: return it.

The gap is at the bottom of harness.py, inside run_agent. Step 1 is done for you: the prompt is already on the record. About eight lines for the rest.

  • Call the model with system, the whole list and tool_specs(tools), and append the reply. Use the system you were handed, not the SYSTEM constant at the top of the file.
  • If the reply holds no tool calls, the run is over: return that message.
  • Otherwise run every call, in the order the model wrote them, appending each result, and go round again.
  • messages is the caller's list. Append to it. Do not copy it, do not rebind the name, and do not keep a second list of your own.
  • No turn limit. The model decides how long the job is, and a model that never stops is lesson 5's problem.
  • When the tests pass, delete chat. Nothing needs it any more, which is the last thing this lesson has to show you.

Ten hidden tests. They script a model that reads three files, a model that lies about its label in both directions, a model that asks for two files in one reply, and a model that wants twelve turns; they also check what your list looks like after a second run_agent on it, and what each request carried.

  1. Look at your chat. Two of its lines make a model call and append the reply; the same two lines appear again further down, which is the hand-crank. What surrounds a pair of lines you want to run an unknown number of times, and what would make it stop?
  2. while True: around the call and the append. Then calls = tool_calls(reply): if it is empty, return reply and you are out; otherwise for call in calls: append run_tool(tools, call), and let the while carry you back to the top. Everything the next request needs is already in messages, so there is nothing to pass along and nothing to remember between rounds — no counter, no flag, no stop_reason.
  3. Nearly the code, all of it inside the gap: 
    while True:
    reply = model.complete(...)   # system, the list, the specs
    messages.append(reply)
    calls = tool_calls(reply)
    if not calls:
        return reply              # the only way out
    for call in calls:
        messages.append(...)      # one result per call

Eight lines. Every test that mentioned a label passed without your code reading one, and the twelve-turn test passed without a counter. Now delete chat and run Check again: run_agent with an empty tool list makes one model call and returns the answer, which is all chat ever did.

What changed since lesson 02 
@@ -74,2 +74,16 @@
         messages.append(reply)
     return text_of(reply)
+
+
+def run_agent(model, system, messages, tools, prompt):
+    """Put `prompt` to the model and keep going until it stops asking for tools.
+    Returns the final assistant message. `messages` is the caller's list: everything that
+    happens is appended to it, so the caller holds the record even if the run dies."""
+    # 1. the prompt goes on the record first (this step is done for you)
+    messages.append(user_message(prompt))
+    # --- your code ---
+    # 2. loop: call the model with `system`, the whole list and the tool specs; append the reply
+    # 3. if the reply holds no tool calls, the run is over: return the reply
+    # 4. otherwise run each call, in order, append each result, and go round again
+    # 5. when the tests pass, delete chat() above: run_agent with no tools does its job
+    # --- end ---
  1. A job that needs three files takes four model calls, and nobody cranks by hand.
  2. Every request is the whole list as it stood: two messages longer than the one before.
  3. Everything that happened is in the list the caller passed in, and in no other.
  4. A reply that holds a call gets its result and another turn, even when labelled "stop".
  5. After a run, the same list takes another prompt and the strict model accepts it.
  6. A reply with no calls ends the run, even when it is labelled "toolUse" or "length".
  7. One reply asks for two files: both results follow it, in call order, before the next turn.
  8. The loop has no turn count of its own: twelve requests in a row get twelve results.
  9. With no tools and a plain answer, the run is one model call: what chat used to do.
  10. run_agent returns the final assistant message, the one that asked for nothing more.

Four files, one of them a red herring, and a reply that asks for two at once. No tests: read the transcript, and the bill under it. Once the lab has passed, this runs against your code.

import harness, lab

ws = lab.Workspace({
    "main.py": "from util import scale\nfrom fmt import show\n\n"
               "show(scale(21))\n",
    "util.py": "from const import FACTOR\n\n"
               "def scale(n):\n    return n * FACTOR\n",
    "fmt.py": "def show(n):\n    print(f'{n:>6}')\n",
    "const.py": "FACTOR = 0  # should be 2\n"})
tools = [harness.make_read_tool(ws)]

def after_seeing(clue, reply):
    """This reply, but only if some result in the request shows
    `clue`. The model cannot know what it was never sent."""
    def step(request):
        shown = [m["content"] for m in request.messages
                 if m["role"] == "toolResult"]
        if any(clue in content for content in shown):
            return reply
        return lab.say("I cannot go on: nobody showed me "
                       + repr(clue) + ".")
    return step

model = lab.ScriptedModel([
    lab.reply(lab.text("Let me look."),
              lab.call("read", {"path": "main.py"})),
    after_seeing("from fmt import", lab.reply(
        lab.text("Two imports. Both, please."),
        lab.call("read", {"path": "util.py"}),
        lab.call("read", {"path": "fmt.py"}))),
    after_seeing("from const import", lab.reply(
        lab.text("fmt.py is fine. util.py leans on const.py."),
        lab.call("read", {"path": "const.py"}))),
    after_seeing("FACTOR = 0", lab.say(
        "Found it. const.py sets FACTOR = 0, so scale(21) is 0. "
        "The comment beside it says it should be 2."))])

messages = []
final = harness.run_agent(model, "You are a careful debugger.",
                          messages, tools,
                          "show(scale(21)) prints 0. Find the bug.")

print(lab.show(messages, stop_reason=True))
print()
print("files read:", ws.reads)
print("model calls:", len(model.calls), "| input tokens per call:",
      [m["usage"]["input"] for m in messages if "usage" in m],
      "| bill:", model.bill)
user -> "show(scale(21)) prints 0. Find the bug."
assistant -> "Let me look." + toolCall c1 read({"path": "main.py"})  [stop_reason=toolUse]
toolResult c1 -> "from util import scale\nfrom fmt import show\n\nshow(scale(21))\n"
assistant -> "Two imports. Both, please." + toolCall c2 read({"path": "util.py"}) + toolCall c3 read({"path": "fmt.py"})  [stop_reason=toolUse]
toolResult c2 -> "from const import FACTOR\n\ndef scale(n):\n    return n * FACTOR\n"
toolResult c3 -> "def show(n):\n    print(f'{n:>6}')\n"
assistant -> "fmt.py is fine. util.py leans on const.py." + toolCall c4 read({"path": "const.py"})  [stop_reason=toolUse]
toolResult c4 -> "FACTOR = 0  # should be 2\n"
assistant -> "Found it. const.py sets FACTOR = 0, so scale(21) is 0. The comment beside it say... (97 characters)"  [stop_reason=stop]

files read: ['main.py', 'util.py', 'fmt.py', 'const.py']
model calls: 4 | input tokens per call: [87, 168, 308, 388] | bill: 951

Four model calls, four files, and a verdict that quotes the comment in const.py. The input token count climbs 87, 168, 308, 388: each request carries everything before it, so the last round paid for all four file contents again. That is 951 input tokens for a job worth one line of an answer, and lesson 6 is about the day one of those files is a 3,000-line log.

  • Hold a conversation: it knows what was said, and who said it.
  • Run a tool the model asks for and show it the result.
  • Keep going until the model stops asking.

Say it in your own words

People say "the agent worked out that the bug was in const.py". In your own words, a sentence or two: what is the agent in that sentence, and what did it work out?

Two things are in the room: a loop and a model. Which of them chose util.py as the second file to read, and how did the other one find out?

The model chose every file, because it was the only party that could read the previous one. Your loop chose nothing: it asked the content whether anything had been requested, ran what had, wrote the answer down and went round. That is the whole of it, and it is nine lines — which is why the thing you built has a name.

Common answers, and what each one misses

  • "The agent is the model." Then the agent read no files, since the model only ever writes. The reading, the going round and the record are all on your side of the wire.
  • "The agent is the framework doing the orchestration." Open harness.py and look for the framework. There is a while, an if and a for.
  • "It is an agent because it uses tools." Your lesson 2 chat used tools and could not get past the first file. What changed today was that the loop goes round as often as the model asks, and stops when it stops asking.

An agent is a model call in a while loop. The model steers; the loop only turns.

Tau's loop continues on the presence of tool calls in the reply, not on the label the provider sent with it.

calls = tool_calls(reply)
if not calls:
    return reply
for call in calls:
    messages.append(run_tool(tools, call))
            tool_results: list[ToolResultMessage] = []
            calls = list(assistant.tool_calls)
            has_more_tools = bool(calls)
            for call in calls:

Tau is async: read async for as for and await f(x) as f(x) until lesson 15.

The same shapes.

  • The condition is the variable that drives Tau's loop: has_more_tools = True above a while has_more_tools or pending: (src/tau_agent/loop.py:98-100). Your if not calls: return is the same test written as an exit. The pending half and the outer while True: are lesson 9's, for messages that arrive mid-run.
  • Tool calls run one at a time, in the order the model wrote them (src/tau_agent/loop.py:156), and each result is appended as its own message.
  • The caller's list is the product. Tau's test for a plain turn asserts on the list the caller passed in, after the loop has appended to it (tests/test_agent_loop.py:85).

What Tau adds. Its loop is an async generator: instead of returning one message it yields events as it goes — turn_start, message_end, turn_end (src/tau_agent/loop.py:102), which is lesson 7 — and it appends every message to a second list of what this run added, so a consumer can ask what is new (src/tau_agent/loop.py:146-151). It takes a max_turns and leaves a readable error message in the record when it trips (src/tau_agent/loop.py:112-120, lesson 5), ends the run when a reply comes back error or aborted (src/tau_agent/loop.py:148-151, lessons 5 and 15), and carries a cancellation signal into every tool call.

Declared in Tau, read by nothing. An AgentTool has an execution_mode that defaults to "parallel" (src/tau_agent/tools.py:88), and the loop above never looks at it: every call goes through the same sequential for. Yours is sequential too, and says so rather than pretending.

Where the label is fixed, and where it is not. Tau overwrites the provider's finish reason with toolUse whenever the reply holds calls, so the first lie in this lesson cannot reach its loop (src/tau_ai/stream.py:80-85). The second lie can: a reply with no calls whose provider said tool_calls is still labelled toolUse there. It does no harm, because nothing downstream asks.

Where yours is weaker. Every model in this course is a script, and so is Tau's FakeProvider, which records the messages of each call the same way (src/tau_ai/fake.py:31). [general] A real reply arrives a fragment at a time and has to be assembled before any of this can run, nobody promises the arguments are the shape you asked for, and the network is between you and the answer. Those are lessons 4 and 5, and then the capstone.

src/tau_agent/loop.py:153-156 · pinned to commit 9fe6a71 · view on GitHub

One more case

A reply arrives holding the words "All done!" and a read call for const.py. The cell runs your loop on it twice: once with the reply labelled toolUse, once labelled stop. Does the run stop? Pick, and give your reason in one line.

It runs read and goes round again, both times: two model calls each, and const.py in ws.reads
The loop asks one question of a reply, and neither the sentence nor the label is it.
 It stops on the stop run and continues on the other. When the words and the content disagree, the label breaks the tie
That keeps the label as a tie-breaker, which is the same belief as before with a smaller job. There is no tie: the content either holds a call or it does not.
 It stops both times. A model that says it is done is done, and running a tool it has finished with is a waste
That reads the prose as a statement about the run. It is a piece of the reply, like the call beside it, and the only thing that settles whether the run goes on is whether a call is there.

Two lines that differ only in the label you set, and const.py was read both times. The words are words, the call is a request, and the label is a note about how the writing ended. Your loop reads exactly one of the three, which is why the two runs cannot come out differently.

import harness, lab

for label in ["toolUse", "stop"]:
    ws = lab.Workspace({"const.py": "FACTOR = 0  # should be 2\n"})
    model = lab.ScriptedModel([
        lab.reply(lab.text("All done!"),
                  lab.call("read", {"path": "const.py"}),
                  stop_reason=label),
        lab.say("Now I am done: FACTOR is 0.")])
    messages = []
    harness.run_agent(model, "You are a careful debugger.", messages,
                      [harness.make_read_tool(ws)], "Find the bug.")
    print(f"labelled {label!r}:", len(model.calls), "model calls,",
          lab.shape(messages), "| files read:", ws.reads)
labelled 'toolUse': 2 model calls, U A[c1] R(c1) A | files read: ['const.py']
labelled 'stop': 2 model calls, U A[c1] R(c1) A | files read: ['const.py']

Every model you have met on this site is a script you could read. Your ten tests pass against them. In a sentence: what have those green tests not proved?

Two things could be wrong in an agent: your loop, or the model's judgement. Which of them did the scripts hold still?

They proved the loop, and thoroughly: given a reply with calls it runs them in order and goes round; given none it stops; given a lie about the label it does neither differently. They proved nothing whatever about a model, because the model's side was written by us. [general] A real one may ask for a tool that does not exist, hand you arguments of the wrong shape, read the same file six times, or answer without looking at all — and your loop will do exactly what it is told with every one of those. Each is a lesson from here on, and the capstone is where you point this at a model nobody scripted.

Common answers, and what each one misses

  • "That it works against a real provider." No request has left this page. The shape of a request, the streaming, the retries and the wire format are all still ahead of you.
  • "That the answers are correct." The script says const.py is the bug because we typed that sentence. A scripted model is a way of making your code's behaviour repeatable, not a way of judging anybody's reasoning.
  • "That the loop handles failure." It handles none. The one failure you saw today came out as an exception and killed the run, and the record it left ends on an unanswered call.
You hit
a bug three files deep, and a chat that stopped after one round trip with an empty answer.
You built
run_agent(): a while loop that continues on tool calls, appends to the caller's list, and returns the final message. chat is gone; it was this loop with the crank removed.
The principle
An agent is a model call in a while loop. The model steers; the loop only turns.
Your harness now
  • SYSTEM
  • user_message
  • text_of
  • tool_calls
  • tool_specs
  • run_tool
  • make_read_tool
  • run_agent
Your answers
Still open
The model asks for a tool called raed. tools_by_name[call["name"]] raises, the run dies, and the record ends on a call nobody answered. Lesson 4.