Harness from First Principles [02/16]

02 · The model can only talk

Words are not deeds

The model says "I checked config.py: the port is 8080." The file says 9090.

~50 min · 2 labs · builds on 01 The function that forgets

This lesson builds on lesson 01. New here? Start at lesson 01, or carry on: every lab is self-contained.

A colleague trims your lesson 1 chat to save tokens: each request carries only the newest two messages. Turn 1 says "My name is Ada." Turn 4 asks "What is my name?" What comes back?

"Your name is Ada." It was told three turns ago
That has the model carrying turn 1 around with it. It carries nothing. Turn 1 is in your list, and this request left it behind.
 "I don't know your name."
The request held two messages, and neither one names anybody.
 An error: the API notices that the conversation has lost its beginning
To notice, the API would have to remember how the conversation began. It keeps nothing between calls, so a request that starts in the middle is just a short request.

You hold eight messages. The request carried two. The list in your hands is a record; the model knows only what this one request says. Sending less is a real technique, and lesson 12 does it without losing Ada.

import harness, lab

def thrifty_chat(model, messages, text):
    messages.append(harness.user_message(text))
    newest = messages[-2:]             # only the newest two
    reply = model.complete(harness.SYSTEM, newest)
    messages.append(reply)
    return harness.text_of(reply)

model = lab.ScriptedModel([lab.forgetful()])
messages = []
for turn in ["My name is Ada.",
             "I look after the billing server.",
             "It listens on port 9090.",
             "What is my name?"]:
    answer = thrifty_chat(model, messages, turn)

print(answer)
sent = [m["role"] for m in model.calls[-1].messages]
print("you hold", len(messages), "messages;",
      "the last request carried", sent)
I don't know your name.
you hold 8 messages; the last request carried ['assistant', 'user']

It says it checked

Your chat from lesson 1 can hold a conversation. Give it a job: a question about a file.

The file, config.py, sits on a small fake disk, a workspace called ws. The fake has one useful habit: it writes down every path anybody reads, in ws.reads. The model is scripted, so yes, we put the words in its mouth. Commit to a number anyway, before you run anything.

The model is about to answer: "I checked config.py: the port is 8080." When it has said so, how many entries does ws.reads hold?

Run the cell below and watch it happen: the second line of its output is ws.reads, and it is empty. The answer is fluent, specific and wrong. If you said 1, you took the sentence for a report of something done; it is only a sentence. Nothing in chat touches ws, and the model has nothing to touch it with: its side of the call holds the text of your request, and no disk.

Your lesson 1 chat, a disk that keeps a log of its reads, and a model with something confident to say.

import harness, lab

ws = lab.Workspace({"config.py": "PORT = 9090\n"})   # a disk with one file on it
model = lab.ScriptedModel([lab.say("I checked config.py: the port is 8080.")])

messages = []
print(harness.chat(model, messages, "What port does config.py set?"))
print("ws.reads:", ws.reads)                          # every path anyone has read
print("config.py really says:", ws.read_text("config.py").strip())
I checked config.py: the port is 8080.
ws.reads: []
config.py really says: PORT = 9090

So where did 8080 come from? It is a port servers often use, which makes it a likely thing to write, and writing likely things is all a model does. [general] A real model in this spot tends to do one of two things: produce a plausible value like this one, or say that it cannot see your files. Neither one reads the file.

Where this failure comes from: when Tau's read tool cannot show an image to a text-only model, the text it returns says "do not infer or describe them" (src/tau_coding/tools.py:264-266). Nobody writes that sentence until a model has described something it never saw.

From words to deeds

So the model writes, and somebody else has to act. Before you see how anyone else arranged that, arrange it yourself.

You want a file to be read when the model needs one. All the model can do is produce text. Which design do you build? (The code below is design A, tried on four replies. Pick first.)

A. Watch its prose: when it writes "I'll read config.py", read config.py
This bets that the model phrases a wish the same way every time, and never uses those words for anything else. The code is that bet. Run it.
 B. Agree on a fixed shape for a request, one that code can parse without guessing
Prose is for people. A request that a program must act on needs a shape a program can check. The rest of this lesson is that shape.
 C. Give the model access to the files, so that it reads them itself
There is nowhere to plug the files in. The cell after this one shows everything model.complete accepts and prints everything it returns.

Four replies, two hits, and one of the hits is wrong: the last reply asks you a question and gets a file read for its trouble. The two in the middle want the file and never get it. A pattern over prose fails in both directions, and every repair is one more pattern. A program that is about to act needs better than a guess.

import re

def wants_to_read(prose):
    """The file the model says it will read, or None."""
    found = re.search(r"I'll read (\S+\.py)", prose)
    return found.group(1) if found else None

replies = [
    "I'll read config.py and tell you the port.",
    "Let me open config.py first.",
    "I need to look at the contents of config.py.",
    "I could say I'll read config.py, but which server?",
]
for prose in replies:
    print(f"{wants_to_read(prose)!s:10} <- {prose}")
config.py  <- I'll read config.py and tell you the port.
None       <- Let me open config.py first.
None       <- I need to look at the contents of config.py.
config.py  <- I could say I'll read config.py, but which server?

Design C, "give the model access". Here is the whole doorway: what goes in, and what comes out. If you think there is somewhere to plug a disk in, add a fourth argument, say files="config.py", and run it.

import lab

model = lab.ScriptedModel([lab.say("I checked config.py: the port is 8080.")])

# everything you can hand over: these three arguments, and no others
reply = model.complete(system="You are a helpful assistant.",
                       messages=[lab.user("What port?")],
                       tools=[])

# everything you get back
for key, value in reply.items():
    print(f"{key}: {value}")
role: assistant
content: [{'type': 'text', 'text': 'I checked config.py: the port is 8080.'}]
stop_reason: stop
usage: {'input': 24, 'cache_read': 0, 'output': 17}

Three arguments go in, and lesson 1 showed you that they become one text. A dict comes out. No argument takes a file handle, a folder or a function. [general] With a hosted model, the other end is a machine in a data centre that has never heard of your disk. (stop_reason is lesson 3's business. Ignore it for now.)

That leaves B. The model cannot act, and its prose cannot be trusted as a command. What it can do is write a request, in a shape both sides agreed on beforehand, and leave the acting to your program.

Lesson 1 told you that an assistant message's content is a list of blocks, and promised a reason. This is the reason. Beside text blocks, a reply may hold a block like this one:

{"type": "toolCall", "name": "read",
 "arguments": {"path": "config.py"}}

That block is a tool call. The name oversells it, because nothing has been called. It is a request, written down, and it sits in the reply until some program decides to honour it. (One key is missing from the picture. A question further down is about why it has to exist.)

To write such a block, the model has to know that something called read exists and what arguments it takes. That is what the third argument of complete, tools, is for. Here is the read tool you will be given in the lab, complete (re-wrapped to fit this column):

def make_read_tool(ws):
    """The `read` tool, over the workspace `ws`."""
    def execute(arguments):
        return ws.read_text(arguments["path"])

    return {
        "name": "read",
        "description":
            "Read a text file and return its contents.",
        "parameters": {
            "type": "object",
            "properties": {"path": {
                "type": "string",
                "description": "Path of the file to read."}},
            "required": ["path"],
        },
        "execute": execute,
    }

A tool is a dict with four parts: a name, a description, a schema for the arguments (parameters, written in JSON Schema, the usual way to describe the shape of a JSON object), and a function.

Four parts: a name, a description, an argument schema and a function. The lab gives you tool_specs(tools), which decides what goes into a request. The code sends two requests and passes the tools only with the first. What does the model get to see?

All four parts. It has to see the function in order to call it
That has the model running your code. It runs nothing. What it needs is to know that a read exists and what to pass it, not how read works.
 Three parts, not the function. Once told, the API keeps them for the rest of the conversation
That gives the API a memory for tools which it does not have for your name. Request 2 described no tools, so for that call no tools existed. What the model should know goes out again every time, and is paid for every time.
 Three parts, not the function, and only in a request that carries them
A function is not text, and text is all a request can carry. And one request is all the model ever knows.

Three parts went out, as text. execute stayed home: the model knows that a read exists and what to hand it, and what read does it has never seen and never will. Request 2 carried no tools, so for that call there were none. The three parts that travel are called the tool's spec.

import harness, lab

ws = lab.Workspace({"config.py": "PORT = 9090\n"})
read = harness.make_read_tool(ws)
print("your tool has:", ", ".join(sorted(read)))

model = lab.ScriptedModel([lab.say("Ok."), lab.say("Ok.")])
first = [lab.user("What port does the server use?")]
second = [lab.user("And the backup?")]
specs = harness.tool_specs([read])
model.complete(harness.SYSTEM, first, specs)
model.complete(harness.SYSTEM, second)    # no tools this time

for number, request in enumerate(model.calls, start=1):
    names = [spec["name"] for spec in request.tools]
    print(f"request {number} described to the model: {names}")
    for spec in request.tools:
        for key, value in spec.items():
            print(f"    {key}: {value}")
your tool has: description, execute, name, parameters
request 1 described to the model: ['read']
    name: read
    description: Read a text file and return its contents.
    parameters: {'type': 'object', 'properties': {'path': {'type': 'string', 'description': 'Path of the file to read.'}}, 'required': ['path']}
request 2 described to the model: []

The spec rides in every request, so it is on the bill. Guess the input-token count without the tool, with it, and with it again on the next call. Then run. The last line tries to send the whole dict, function and all.

import harness, lab

ws = lab.Workspace({"config.py": "PORT = 9090\n"})
read = harness.make_read_tool(ws)
specs = harness.tool_specs([read])
model = lab.ScriptedModel([lab.say("Hello."), lab.say("Hello."), lab.say("Hello.")])
hi = [lab.user("Hi!")]

for label, tools in [("no tools", []), ("one tool", specs),
                     ("the same tool, next call", specs)]:
    reply = model.complete(harness.SYSTEM, hi, tools)
    print(f"{label:25} input tokens: {reply['usage']['input']}")

print()
print(lab.render_request(harness.SYSTEM, hi, specs))

# and the whole dict, function and all?
refused = model.complete(harness.SYSTEM, hi, [read])
print(refused["error_message"])
no tools                  input tokens: 23
one tool                  input tokens: 78
the same tool, next call  input tokens: 78

system: You are a helpful assistant.
tools:
{"description": "Read a text file and return its contents.", "name": "read", "parameters": {"properties": {"path": {"description": "Path of the file to read.", "type": "string"}}, "required": ["path"], "type": "object"}}
messages:
{"content": "Hi!", "role": "user"}

400 invalid_request: tools[0]: send exactly name, description and parameters, and nothing callable; got ['description', 'execute', 'name', 'parameters']

23 tokens without the tool, 78 with it, and 78 again on the next call. The model forgets its tools exactly as it forgets your name, so you pay for them every time. One small tool more than trebled this request, and a working agent carries several bigger ones: Tau's coding agent ships read, write, edit and bash (src/tau_coding/tools.py:180-185), and the description of its read alone is longer than this whole spec (src/tau_coding/tools.py:362-372).

The middle of the output is the request itself. The tool is one line of text between the standing instructions and the messages, and the model reads it like everything else. The last line is what happens when you try to send the function too: there is no way to write it down. (The lab counts four characters as a token. Real tokenizers differ, and the shape of the bill does not.)

Anatomy of a tool A tool is one dict with four parts: name, description, parameters and execute. tool_specs(tools) sends the first three to the model in every request, where they cost tokens. The execute function never leaves your process. The model can only reply with a toolCall block, a request; run_tool in your harness is the only thing that calls execute. a toolone dict, four parts name"read" description"Read a text file andreturn its contents." parameterspath: string, required the model is told these threethis one stays in your process executedef execute(arguments): model.complete() read(path="config.py")c1 run_tool()your harness tool_specs(tools) in every requestit costs tokens reply: a toolCall a request:nothing ran yet execute(arguments)returns a str tool_specs(tools) in every requestit costs tokens reply: a toolCall a request:nothing ran yet execute(arguments)returns a str

Figure 2.1 One dict, cut in two by the edge of your process. Three parts travel in every request. execute stays home, and only run_tool(), which you are about to write, ever calls it. The small c1 on that toolCall block is the subject of the next question.

A model may ask for two things in one reply. Here it asks for config.py and backup.py. You read both and hold two texts: PORT = 9090 and PORT = 7070. When they go back, how does the model know which file each one came from?

Each call carries a label, and each result has to quote it
Look at the output. You did not invent the labels: the model wrote them.
 By position. The first result answers the first call
Position works until something goes missing. If the first read fails and only one result goes back, position makes it the answer to the first call, and nothing in the message says otherwise.
 From the contents. It will recognise which file is which
Both results look like PORT = .... Asking the model to work out the pairing from the contents is design A again: prose doing a protocol's job.

The missing key is id. The model numbers its own calls, and a result answers one by quoting the number back as tool_call_id. That number is the call id. It is a cloakroom ticket: you do not get your coat back by standing where you stood when you handed it in.

from pprint import pprint
import harness, lab

ws = lab.Workspace({"config.py": "PORT = 9090\n",
                    "backup.py": "PORT = 7070\n"})
specs = harness.tool_specs([harness.make_read_tool(ws)])
model = lab.ScriptedModel([
    lab.reply(lab.call("read", {"path": "config.py"}),
              lab.call("read", {"path": "backup.py"}))])

question = [lab.user("Which ports do the two files use?")]
reply = model.complete(harness.SYSTEM, question, specs)
for block in reply["content"]:
    pprint(block, width=60, sort_dicts=False)
{'type': 'toolCall',
 'id': 'c1',
 'name': 'read',
 'arguments': {'path': 'config.py'}}
{'type': 'toolCall',
 'id': 'c2',
 'name': 'read',
 'arguments': {'path': 'backup.py'}}

A reply can be prose, a call, more prose, another call. Suppose you stored it the easy way: one string text holding all the prose and, beside it, a list calls. What did you lose? And on the day text and calls disagree, which one is right?

Lesson 1: what goes back to the model on the next call? Your two fields, or something else?

You lost the order: which sentence came before which call. "Then the backup, to compare" means something between c1 and c2, and nothing in a heap after them. You also lost the message itself. On the next call the model has to re-read its own reply as it wrote it, and from two fields you cannot rebuild that.

So store what was written: one ordered list of blocks. The text and the list of calls are worked out from it each time somebody asks. Then there is no second copy, and nothing to disagree.

Common answers, and what each one misses

  • "Nothing is lost. I have all the prose and all the calls." It misses the order, and that this message is sent back on every later call. Flattening it is lesson 1's glued string over again.
  • "calls wins, because that is what my code acts on." The model only ever sees the content. What you act on and what the model believes it asked for have to be the same thing.
  • "Keep both, and keep them in sync." That is a promise every future edit has to keep, and lesson 10 edits content. A value worked out on demand cannot go stale.

The same argument as output. One reply, stored as written. Then a "convenient" second copy of its calls, and one later edit.

import harness, lab

reply = lab.reply(lab.text("The live config first."),
                  lab.call("read", {"path": "config.py"}, id="c1"),
                  lab.text("Then the backup, to compare."),
                  lab.call("read", {"path": "backup.py"}, id="c2"))

# one list, in the order it was written
for block in reply["content"]:
    what = block.get("text") or block["name"] + " " + block["id"]
    print(block["type"].ljust(8), what)

reply["calls"] = harness.tool_calls(reply)      # a second copy, "for convenience"
reply["content"].pop()                          # later, something edits the content
in_content = [b["id"] for b in reply["content"] if b["type"] == "toolCall"]
print("content says: ", in_content)
print("the copy says:", [c["id"] for c in reply["calls"]])
print(lab.validate([lab.user("Compare the ports."), reply])[0])
text     The live config first.
toolCall read c1
text     Then the backup, to compare.
toolCall read c2
content says:  ['c1']
the copy says: ['c1', 'c2']
messages[1]: unknown key 'calls' in assistant message

The content says one call. The copy still says two. And the copy cannot travel: a message carries the keys the wire format defines, calls is not one of them, and the strict model refuses the message rather than send it. In the lab, the function that works the calls out is tool_calls(message), and one of its tests edits the content and asks again.

You ran read for call c1 and you hold PORT = 9090. The model sees it only if it goes into the list. The user did not say it. The assistant did not write it. The code builds three placements and sends each to the strict model. Which does it accept?

A. A user message: "Tool output: PORT = 9090"
That is prose again: the model must guess that this sentence answers c1. And lesson 1's forgery is back, because anyone who can type "Tool output: PORT = 1" into the chat has written a tool result.
 B. Inside the assistant's reply, after the call it made
That files the file's contents under things the model wrote, as if the model had done the reading. It read nothing. Roles say who spoke, and the model did not say this.
 C. A message of its own, with a new role, quoting the id
A third speaker needs a third role.

Two refusals with one complaint: c1 has no result. A call is a question left open until a message with role toolResult quotes its id, and the same text anywhere else does not count. That message is a tool result. With it a transcript has three speakers: the user, the assistant, and your program reporting what it did.

Where this refusal comes from: [general] hosted providers refuse a request in which a call has no result. Tau keeps a function whose whole job is to guarantee that "every tool call has exactly one adjacent result" before a provider sees the list (src/tau_agent/tool_history.py:40-47).

import lab

question = lab.user("What port does the server use?")
call = lab.call("read", {"path": "config.py"}, id="c1")
asked = lab.reply(call)
found = "PORT = 9090\n"

as_user = lab.user("Tool output: " + found)
stuffed = lab.reply(call, lab.text(found))
own = {"role": "toolResult", "tool_call_id": "c1",
       "tool_name": "read", "content": found,
       "is_error": False}
placements = {
    "A, a user message": [question, asked, as_user],
    "B, inside the reply": [question, stuffed],
    "C, its own message": [question, asked, own],
}

def answer(request):   # it reports only what it was shown
    shown = request.result_for("c1")["content"].strip()
    return lab.say("config.py says " + shown + ".")

for name, messages in placements.items():
    model = lab.ScriptedModel([answer])
    reply = model.complete("You are helpful.", messages)
    said = reply.get("error_message")
    said = said or reply["content"][0]["text"]
    print(name)
    print("   ", said)
A, a user message
    400 invalid_request: messages[1]: toolCall c1 has no toolResult
B, inside the reply
    400 invalid_request: messages[1]: toolCall c1 has no toolResult
C, its own message
    config.py says PORT = 9090.
One call in, one result out Every toolCall in an assistant message must be answered by exactly one toolResult with the same id, placed right after that assistant message, in call order. The first panel is valid: calls c1 and c2 are each tethered to one result. Three panels are broken: a call with no result; an orphan result whose call is not in the list; and a result that comes late, after a user message, followed by a duplicate for the same call. validone result per call, right after it 0userWhich ports do they use? 1assistantread(path="config.py")c1read(path="backup.py")c2 2toolResult · c1PORT = 9090 3toolResult · c2PORT = 7070 id: the model names each call, c1 and c2tool_call_id: the result names its callthe dotted tether joins each pair message 2 in your code:{"role": "toolResult","tool_call_id": "c1","tool_name": "read","content": "PORT = 9090\n","is_error": False} missinga call that nobody answered 0userWhat is in config.py? 1assistantread(path="config.py")c1 toolResult · c1no result 2userNever mind, do X. orphana result whose call is not in the list 0userWhat is in config.py? 1assistantI will check the file. 2toolResult · c1DEBUG = Trueorphan lateduplicatenot right after its call; answered twice 0userWhat is in config.py? 1assistantread(path="config.py")c1 2userNever mind, do X. 3toolResult · c1DEBUG = Truelate 4toolResult · c1DEBUG = Trueduplicate

Figure 2.2 Two calls, two results, each pair joined by its id. Beside them, message 2 of that list, the result for c1, as the dict your code is about to build.

Two more keys ride along in that dict. tool_name says which tool ran. is_error is False all through this lesson; lesson 4 is about the day it is not.

That is everything the lab needs. A tool is three things advertised to the model and one function that only your program runs. A tool call is a block inside the reply. The answer is a message of its own that points back by id.

Build: give it hands

Your program is the only thing here that can act. That is what the name of your file has meant all along: a harness is the code around the model that holds the list, offers the tools and does the work.

Two small functions, about four lines between them. Each has a gap marked # --- your code ---: scroll down to tool_calls, then on to run_tool.

  • tool_calls(message) returns the toolCall blocks of a reply as a list, in the order the model wrote them. Work it out from content every time you are asked and store it nowhere: the tests edit the content and ask again.
  • run_tool(tools, call) runs the one tool the call names and returns the toolResult message that answers it. Finding the tool by name is done for you. Hand execute the call's arguments dict exactly as the model wrote it.

tool_specs and make_read_tool are given in the same file. Read them first: they are the worked example. This is the happy path only. A misspelt tool name or a missing file will crash, and lesson 4 opens with exactly that. As in lesson 1, Run prints nothing here, because the file only defines functions. Check is what talks back.

  1. text_of, a few lines up, already picks the blocks of one type out of content. What would it look like if it kept the blocks themselves instead of joining their text? And for run_tool, look at Figure 2.2: where does each of the five values in that dict come from?
  2. tool_calls: a list comprehension over message["content"] that keeps the blocks whose "type" is "toolCall". Return a real list, not a generator, and do not store anything on the message. run_tool: the tool's function is tool["execute"]. It takes one argument, the call's "arguments" dict as it is, and returns a string. Then return a dict with five keys. Three of the values come from the call, one from the tool, and one is a constant.
  3. Nearly the code: 
    # tool_calls
return [block for block in message["content"]
        if block["type"] == ...]

# run_tool, after the lookup
content = tool["execute"](...)    # the arguments dict
return {"role": "toolResult",
        "tool_call_id": ...,      # the call's id
        "tool_name": ...,         # the call's name
        "content": content,
        "is_error": False}

The model asked, and this time something happened: ws.reads has an entry. Four lines, and your program has hands.

What changed since lesson 01

The line-by-line diff needs JavaScript. The whole file this exercise starts from is printed at the end of it.

  1. tool_calls returns the calls inside a reply, in the order the model wrote them.
  2. tool_calls is worked out from the content each time: edit the content and it changes.
  3. run_tool runs the tool and returns one toolResult that points back at the call.
  4. The tool named in the call is the one that runs, and it receives the arguments dict.

chat gains a tools parameter, and step 1 is done for you: every request now advertises tool_specs(tools). Your part is about five lines, in the gap at the very bottom of the file.

  • If the reply asks for tools, run each call with run_tool, in the order the model wrote them, once each, and append each result to messages.
  • Then call the model once more, exactly as in step 1, and append that reply too. Its words are what chat returns.
  • If the reply asks for nothing, chat behaves as it did in lesson 1: one model call.

One round trip and no more, cranked by hand. That is deliberate. Lesson 3 is the relief.

The tests describe a conversation in shorthand: U A[c1] R(c1) A is a user message, an assistant message asking for call c1, the result for c1, and an assistant message. Our simplification: the lab's strict model wants results in call order. [general] Providers match them by id and tend to be less fussy about order.

  1. When the tools have run, who has seen their results? What has to happen before chat can return an answer with 9090 in it?
  2. Under an if tool_calls(reply):, loop over tool_calls(reply) and append run_tool(tools, call) to messages for each call. Then make the second model call with the same three arguments as step 1. Assign it to reply again, because the last line returns text_of(reply), and append it. Run each tool once: not every tool only reads.
  3. Nearly the code, all of it inside the gap: 
    if tool_calls(reply):
    for call in tool_calls(reply):
        messages.append(...)      # one result per call
    reply = model.complete(...)   # as in step 1
    messages.append(reply)

Question, call, result, answer: U A[c1] R(c1) A. The port in the answer is the port in the file. You cranked one round trip by hand. Remember how that felt.

What changed since the previous lab 
@@ -61,9 +61,15 @@
 # The code that talks to the model. The model keeps nothing between calls.
 
-def chat(model, messages, text):
-    """Say `text` to the model and return its answer as a string.
+def chat(model, messages, text, tools=()):
+    """Say `text` to the model and return its answer as a string. If the reply asks for
+    tools, run them and ask once more: one round trip, cranked by hand.
     `messages` is the caller's list. It is the only memory this conversation has."""
     messages.append(user_message(text))
-    reply = model.complete(SYSTEM, messages)
+    # 1. every request now also says which tools exist (this step is done for you)
+    reply = model.complete(SYSTEM, messages, tool_specs(tools))
     messages.append(reply)
+    # --- your code ---
+    # 2. if the reply asks for tools: run each call, in order, and append each result
+    # 3. then call the model again, exactly as in step 1, and append that reply too
+    # --- end ---
     return text_of(reply)
  1. The second request holds the question, the reply that asked for the file, then the result.
  2. The answer contains 9090, the port that is really in config.py.
  3. One reply asks for two files: both are read, and the results follow in call order.
  4. Every request tells the model each tool's name, description and parameters, no more.
  5. When the reply asks for no tools, chat makes one model call and returns its text.
  6. The caller's list ends as question, call, result, answer, and a provider would accept it.

Your chat, a model that asks for two files in one reply, and no tests. Once lab 2 has passed, this runs against your code. Read the transcript: each call has its result right behind it, in call order, and the bill is for two requests.

import harness, lab

ws = lab.Workspace({"config.py": "PORT = 9090\n", "backup.py": "PORT = 7070\n"})

# it can only report what the results in this request say
def report(request):
    seen = [m["tool_call_id"] + ": " + m["content"].strip()
            for m in request.messages if m["role"] == "toolResult"]
    return lab.say("Here is what the files say. " + "; ".join(seen) + ".")

model = lab.ScriptedModel([
    lab.reply(lab.text("Let me look at both."),
              lab.call("read", {"path": "config.py"}),
              lab.call("read", {"path": "backup.py"})),
    report])

messages = []
print(harness.chat(model, messages, "Which ports do config.py and backup.py use?",
                   [harness.make_read_tool(ws)]))
print()
print(lab.show(messages))
print()
print("ws.reads:", ws.reads, "| model calls:", len(model.calls),
      "| bill:", model.bill, "tokens")
Here is what the files say. c1: PORT = 9090; c2: PORT = 7070.

user -> "Which ports do config.py and backup.py use?"
assistant -> "Let me look at both." + toolCall c1 read({"path": "config.py"}) + toolCall c2 read({"path": "backup.py"})
toolResult c1 -> "PORT = 9090\n"
toolResult c2 -> "PORT = 7070\n"
assistant -> "Here is what the files say. c1: PORT = 9090; c2: PORT = 7070."

ws.reads: ['config.py', 'backup.py'] | model calls: 2 | bill: 297 tokens
  • Hold a conversation: it knows what was said, and who said it.
  • Run a tool the model asks for and show it the result.

Say it in your own words

People say "the model used a tool". In your own words, a sentence or two: what happened, and who did what?

Follow PORT = 9090 from the disk to the final answer. Whose code touched it at each step?

The model wrote a request. Your harness read it, ran a function, wrote down what came back, and sent the whole list again. The model then read your note and wrote an answer. At no point did it do anything but write.

The trade calls this function calling, or tool use. Both names flatter the model. Nothing was called on its side and no function ever left your process: it wrote a block, and your program decided what to do about it.

Common answers, and what each one misses

  • "The model called my function." It wrote a request for it. run_tool called it, and run_tool is yours.
  • "The API ran the tool for me." Nothing ran until your code ran it. Delete the loop over tool_calls(reply) and the request just sits in the reply, unanswered.
  • "The model read the file." It read a message you wrote that held the file's text, and it paid input tokens for every character.

A tool call is just a specially shaped piece of the reply. I am the one with hands.

When Tau talks to a real provider, three parts of a tool leave the process: name, description and schema. The function is not in the payload.

def tool_specs(tools):
    return [{"name": tool["name"],
             "description": tool["description"],
             "parameters": tool["parameters"]}
            for tool in tools]
def _anthropic_tool(
    tool: AgentTool,
    ...
) -> dict[str, JSONValue]:
    payload: dict[str, JSONValue] = {
        "name": tool.name,
        "description": tool.description,
        "input_schema": dict(tool.input_schema),
    }

The same shapes, with types. Tau is async throughout: where the code below says async def or Awaitable, read an ordinary function until lesson 15.

What Tau adds.

Declared in Tau, read by nothing. prepare_arguments and execution_mode (src/tau_agent/tools.py:87-88) and a result's terminate (src/tau_agent/tools.py:27) are fields today, not behaviour: Tau's agent loop never looks at them. This course does not teach them as if it did.

Where yours is weaker. Our simplification: your messages are plain dicts that only lab.validate checks, where Tau's are validated models. And your run_tool knows only the happy path. The first misspelt tool name will kill it, and so will a call that leaves path out. [general] A schema says what the model should send; whether anything enforces it is the provider's business, and your code still has to survive whatever arrived. That is lesson 4.

src/tau_ai/anthropic.py:687-699 · pinned to commit 9fe6a71 · view on GitHub

One more case

The model needs to know today's date. You can add a today tool, or you can put one line of text, "Today is 2026-09-19.", into the request. Which do you ship? Think about what each costs per call, and give your reason in one line.

A tool. It costs nothing unless the model asks
That treats a tool's description as free. It is text in every request, asked for or not. And when the model does ask, fetching ten characters costs a second model call with the whole list sent again.
 A line of text
Small, needed all the time, and the same for the whole run: that is a line of text.
 Neither. The model knows what day it is
That gives the model a clock. It has the text of your request and what it learned in training, which ended some time ago. Ask it the date with neither a tool nor a line and it will write a likely one.

The tool needs two model calls to fetch one short string, and its description rides in every request whether the date ever comes up or not. The line costs six tokens. A tool earns its place when the answer is large, changes while the run is going, or is rarely needed: a file, not a date. Tau writes the date into its standing instructions as text (src/tau_coding/system_prompt.py:144-146).

import json
import harness, lab

today = {"name": "today",
         "description": "Return today's date.",
         "parameters": {"type": "object", "properties": {},
                        "required": []},
         "execute": lambda arguments: "2026-09-19"}
line = "Today is 2026-09-19.\n"

def tell(request):     # it can only report a date it was shown
    shown = request.result_for("c1")["content"]
    return lab.say("It is " + shown + ".")

ask = lab.reply(lab.call("today", {}))
with_tool = lab.ScriptedModel([ask, tell])
harness.chat(with_tool, [], "What is the date?", [today])

with_line = lab.ScriptedModel([lab.say("It is 2026-09-19.")])
harness.chat(with_line, [], line + "What is the date?")

spec = json.dumps(harness.tool_specs([today])[0])
for name, model, extra in [("a tool", with_tool, spec),
                           ("a line", with_line, line)]:
    print(f"{name}: {len(model.calls)} model call(s),",
          f"{model.bill} input tokens to learn the date,",
          f"{lab.count_tokens(extra)} tokens in every request")
a tool: 2 model call(s), 168 input tokens to learn the date, 31 tokens in every request
a line: 1 model call(s), 32 input tokens to learn the date, 6 tokens in every request
You hit
A confident answer about a file that nobody had read.
You built
tool_calls(), run_tool(), and a chat() that does one tool round trip.
The principle
A tool call is just a specially shaped piece of the reply. I am the one with hands.
Your harness now
  • SYSTEM
  • user_message
  • text_of
  • tool_calls
  • tool_specs
  • run_tool
  • make_read_tool
  • chat
Your answers
Still open
The bug is three files deep, and your chat stops after one round trip. Lesson 3.