16 · Meeting the real world
Your machine
A socket, a disk, a shell and a person at a keyboard. Four edges a browser tab could not give you, and none of them is in harness.py.
Four edges
Everything in the browser was true except its surroundings. The model was a script, the disk was a dict, the shell refused pipes, and the person who says yes was a list of answers. Replacing all four is a couple of hundred lines of ordinary Python, and none of it goes in harness.py.
Three code files come with this lesson, with a README beside them, and they are yours to read rather than to trust. Put them in one directory together with your own two, nowhere near a project you care about. Everything here is free, reference solutions included; nothing in this course is ever held back.
| File | Where it comes from |
|---|---|
harness.py | yours, as you left it at the end of lesson 15 |
adapter.py | yours, as you left it on page 1 |
run_local.py | the wiring, a terminal frontend, and the only event loop in the course |
provider.py | the socket: acomplete over the Messages API, urllib and one thread |
real_tools.py | read, write and bash over pathlib and subprocess |
README.md | the same warnings as this page, beside the code, for the day you come back to it |
lab_wire.py, lab_fixtures.py | optional: they give you --replay, which needs no key and no network |
lab.py, lab_world.py, lab_personas.py, lab_check.py, the test files | optional: they give you the hidden tests, on your machine |
You need Python 3.11 or newer — stop_run uses Task.cancelling(), which arrived in 3.11, and the browser ran 3.12. Nothing to install: the standard library does all of it, urllib included. POSIX only, so macOS, Linux or WSL: the bash tool needs process groups and SIGKILL, Ctrl-C during a run needs add_signal_handler, and Windows has none of the three.
Before the key: the same program, no network
The first thing to run is the one that cannot cost anything.
python run_local.py --replay --project ./somewhere-safe "read the readme"
That is the whole program — your harness, your adapter, the real read tool, your actual disk — driven by page 1's streams instead of by a socket. The swap is one constructor: lab_wire.ReplayModel(fixtures, to_anthropic, parse_sse) where AnthropicProvider(to_anthropic, parse_sse, …) would go, because both were built to take your two functions and hand back an assistant message. If this does not work, nothing further down will, and the problem is in your file rather than in your key.
Then the tests, under CPython, where a traceback is a traceback:
python lab_check.py --harness harness.py --lesson 15 tests_l03.py tests_l09.py …
That command runs every hidden test whose # applies: range includes lesson 15 — over a hundred and fifty of them, from lesson 2 onwards, unchanged, against the file you are about to point at a real model. In a sentence or two: why can they still run, and what have they still not told you?
Those tests never mention a provider. What do they use instead, and what does that make them evidence about?
They run because they never touch a provider: they build a scripted model, a workspace and a shell, and reach your code through one driver. That is what makes them repeatable, and it is exactly what makes them silent about the thing you are about to do. They are evidence that your harness keeps its contract, and no evidence at all about a model's judgement.
You said something close to this in lesson 3, when the loop was nine lines old and its ten tests had just gone green:
(Lesson 3 asked what those green tests had not proved. If you answered it on this browser, your own words would be here.)
Common answers, and what each one misses
- "They run because Python is Python." Half of it. They also run because nothing in
harness.pyimportslab, and nothing in the tests imports a vendor: the seam was there from lesson 1 and it is why today costs nothing. - "They prove the harness works with a real model too." They prove the harness does what it did before. The real model changes what arrives, not what your code does with it — and what arrives is the half of the system those tests cannot see.
- "They have not told me whether the agent is any good." Right, and that is a question with its own method, further down this page.
The socket
Give it a key when the replay works. It is read from the environment and from nowhere else, it is never written to a file, and the tutorial site has no server and has never seen it. Do not put it in a file inside the project directory: the agent can read files in the project.
export ANTHROPIC_API_KEY=…
python run_local.py --project ./somewhere-safe "what does this project do?"
provider.py is the one object in the program that talks over a network, and it is the object page 1's first cell was pretending to be: a constructor, an acomplete, and your two functions passed in. Almost nothing in it knows what this vendor's JSON looks like. What is vendor-specific is a handful of constants and one header.
The interesting part is that urllib has no async form, and a socket read blocks the thread it runs on. So the whole request and the whole read happen on one worker thread, and each event crosses back into the event loop as it is parsed:
with urllib.request.urlopen(request, timeout=self._timeout) as response:
for line in response:
if stop.is_set() or _cancelled(signal):
return
yield line.decode("utf-8", "replace").rstrip("\r\n")
Read that against lesson 15. The signal is looked at between two lines, which is the cooperative half working exactly as designed. A thread already blocked inside read looks at nothing at all, and task.cancel() does not reach into it either; the socket timeout is the only thing that ends that wait. Both facts are in the file's own docstrings, because a comment that says what the code cannot do is worth more than one that repeats what it does.
Which leaves the failures. A real network has a whole vocabulary of them.
Tap every failure that a provider client would be right to retry by itself, without telling anyone.
401: the key is wrong400 invalid_request: the transcript it sent was malformed429: too many requests, for now529or500: the vendor is overloaded or broken- the connection drops before a single word of the reply has been shown
- the connection drops after half the answer is on the screen
- the model asks for a tool that does not exist
Two of those are about the request itself: send it again unchanged and it fails again the same way, so a retry buys a second wait and no new information. The last one is not a transport failure at all: it is lesson 4, and it already has a mechanism — a result marked is_error that the model reads. The interesting one is the connection dropping after words were shown, and the optional question in page 1's comparison is what it costs. Tau retries the statuses that can change their minds (src/tau_ai/anthropic.py:377-380), retries a dropped connection only while nothing has been emitted (src/tau_ai/anthropic.py:345), and waits between attempts in small steps so that a Stop is not swallowed by a backoff (src/tau_ai/retry.py:46-62).
provider.py does none of it. A 429, a 500 or a 529 ends the run with one message you can read, carrying the vendor's own sentence and the request id their support will ask for. That is a gap, it is written down as one, and adding it is a good afternoon's work with a real thing to test against.
A disk, a shell, and someone to ask
real_tools.py has the same three tools you built, with the same names, the same schemas, the same execute(arguments, signal=None) and the same budgets, so harness.py cannot tell which set it is holding. What is new is everything the simulator was too polite to do. Four of the new things are in the first four lines of one call:
process = subprocess.Popen(
["bash", "-c", command], cwd=root, text=True, errors="replace",
stdin=subprocess.DEVNULL, # a command that asks a question gets EOF
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, # one stream, in order
start_new_session=True) # its own process group, so we can kill it all
A command with no standard input dies instead of hanging for ever on a prompt nobody can answer. Errors arrive beside the output that caused them, in order, because the model is reading both. The new session is so that the timeout has something to kill: killing a shell leaves its children running, so the tool signals the whole group, and it does that even when the command has already exited, because what a command starts can outlive it. Tau does the same, in both places (src/tau_coding/tools.py:626-635, src/tau_coding/tools.py:1174-1187).
read and write resolve every path, symlinks included, and refuse anything that lands outside the project directory. That is a fence and not a sandbox, and the file says so: bash walks around it with one cd .., which is exactly why bash is the tool behind lesson 14's gate.
Four stage props held the world still for sixteen lessons. In this order: lab.ScriptedModel, lab.Workspace, lab.Shell, lab.Human. Put the four things that replace them into the same order.
AnthropicProvider:urllib, one worker thread, your two adapter functionsmake_read_toolandmake_write_tooloverpathlib, fenced into--projectmake_bash_tool:subprocess, a real timeout, a process group to killKeyboard:input(), and you
There is a fifth row, and it is the one worth noticing: lab.drive, which ran every exercise in this course, is replaced by two small functions in run_local.py that consume the run and close it however it ends. And there is a row with nothing in it. run_agent, run_tool, Harness, context_for_model, repair_tool_history, build_system_prompt, guard, stop_run — everything between the edges is byte for byte the file you wrote against the fake.
The gate, and the part that is not a joke
Lesson 14's gate is on by default: every bash call asks you first, through confirm_with(Keyboard()). A check that raises has not said yes, so with nothing attached to standard input every gated call is refused rather than run, which is the fail-closed rule doing its job in a place you did not write it for. --yes takes the gate off. Read this before you use it.
The other guard is --max-turns, twenty by default. Every turn re-sends the whole transcript, which is lesson 1's bill with a real invoice attached, and each run prints the input tokens it spent. Watch that number for a session or two before you trust yourself with a long one.
Ctrl-C during a run is lesson 15, wired to a keyboard. It sets the flag, gives the run two wall-clock seconds to notice — stop_run's grace is counted in turns of the event loop, which is microseconds against a socket — and then hands what is left to your own stop_run for the hard half. A second Ctrl-C says out loud that a thread already inside a socket read cannot be interrupted, rather than pretending to do something about it.
Two things to do with it that are not chat
A failure safari. Reproduce three of the failures this course scripted for you, against the real model: a tool that raises, a model that will not stop, a 400 from a transcript with a hole in it. They look exactly the same as they did in the browser. That is not a coincidence and it is not luck; it is what the fakes were built to be true about.
The smallest honest
What this does not do
Small and honest beats big and vague. Every one of these is a real gap rather than an oversight, and each is an afternoon: no retries; no prompt caching, so you pay full price for the whole transcript every turn; no session on disk, because SessionLog wants a Workspace and giving it pathlib instead is about twenty lines; no compaction, which needs the log first; one run at a time; no edit tool, no images, no subagents, no terminal interface. And one more, which is the interesting one, because it is a decision rather than a missing feature.
Your adapter has no wire form for a thinking block: parse_sse cannot read one and to_anthropic refuses to send one back. The model provider.py names thinks by default, and the file records the day it checked that, so such a block arrives on the very first turn. Nothing breaks, because an unknown block takes an empty place in content and later deltas still find their own block by index, which is the one line of your parser keeping the rest of that reply honest. Handling them properly, signatures kept and sent back untouched, is the first thing to add to your adapter.
There is a one-line way out. The API accepts a request that turns thinking off, at the lower effort settings, and this program deliberately does not send it, because the vendor's own documentation warns that with thinking disabled the model occasionally writes a tool call into its visible text instead of emitting a tool-call block. (provider.py records the date that reference was read, which is the only honest way to write down something a vendor can change.) Before you read what that costs, predict it.
The reply comes back whole and well formed. Its content is one text block, and the text reads Let me check the README. and then, on its own line, read(path="README.md"). There is no toolCall block anywhere in it. What does your harness do?
tool_calls() reads content for blocks of a type, and a sentence is not one. Nothing in your program has ever read the model's words.stop_reason of stop. To call it malformed, something would have to read the text and have an opinion about it.One model call, no file read, and a frontend that reports success: the run ended the way every finished run ends. No error, no exception, no damaged transcript — the agent simply said it was reading a file, and never did. A harness whose whole job is tool calls cannot afford a failure that quiet, so the local runner keeps thinking on and controls what it costs with --effort instead. That is the shape of most real engineering decisions here: not the clever option and the stupid one, but a saving with a silent failure attached, and a price with a loud one.
import harness, lab
ws = lab.Workspace({"README.md": "The port is 9090.\n"})
tools = [harness.make_read_tool(ws)]
model = lab.ScriptedModel([
# The reply holds no toolCall block at all. The call is a sentence the model wrote.
lab.say('Let me check the README.\n\nread(path="README.md")'),
])
h = harness.Harness(model, "You are a careful coding agent.", tools)
screen, kinds = harness.FinalTextRenderer(), []
async for event in h.prompt("What port does the server use?"):
kinds.append(event["type"])
screen.render(event)
print("events:", kinds)
print()
print("what the user is shown:")
verdict = screen.finish()
print()
print("the frontend calls the run a success:", verdict)
print("files read: ", ws.reads)
print("model calls: ", len(model.calls))
print("a transcript a provider would accept:", lab.validate(list(h.messages)) == [])
events: ['agent_start', 'turn_start', 'message_end', 'message_end', 'turn_end', 'agent_end'] what the user is shown: Let me check the README. read(path="README.md") the frontend calls the run a success: True files read: [] model calls: 1 a transcript a provider would accept: True
Read Tau
You are now in a position to read somebody else's harness and recognise all of it. Take four files in dependency order first — loop.py, harness.py, tool_history.py, session/memory.py, all of them in tau_agent, which is the brain. Then system_prompt.py, which is one layer out in tau_coding, because what goes into a system prompt is what only the environment knows. The map below tells you which of your own regions you are looking at each time. The imports at the top of Tau's harness are the same rule your six regions follow: the agent layer names the provider protocol and the tools, and knows nothing about a vendor or a terminal (src/tau_agent/harness.py:12-22).
Figure 16.4 The three packages, and where your own file's six regions live in them. One call crosses each boundary, and it is a call you have written.
Then look at how big session.py and tui/app.py are, and be glad the ideas stayed small even where the program did not. The last thing worth knowing before you close this page is that you have already built the hard part of a feature you have not heard of: a execute runs a fresh Harness to the end and returns the child's final text as the call's one result. Twenty-odd lines, no new ideas, and Checkpoint D has you write it.
- You hit
- a socket, a disk, a shell and a person at a keyboard: four edges no browser tab could give you
- You wired
- a provider over
urlliband one worker thread, real tools overpathlibandsubprocess, the gate on by default, and Ctrl-C tostop_run - You changed
- no line of
harness.py, and no line ofadapter.py - The principle
- Tests against a fake prove that the harness keeps its contract. Only an eval says whether the agent is any good.
- Your answers
- Still open
- Sixteen lessons, and nothing has yet asked you to build something you were not walked through. Checkpoint D is eight broken transcripts and one function nobody has shown you.