Harness from First Principles [--/16]

Syllabus

Sixteen lessons in six parts, four checkpoints and two optional side quests. You start with a function from text to text and finish with a coding agent you wrote yourself.

Each row gives the question a lesson answers, what it yields, and the names it adds to your harness.py. From lesson 2 on, every lesson opens by running the code you have just finished against a situation it cannot handle. You can start anywhere, skip a question, or go straight to a lab: every lab starts from a complete reference file, so you never need your own code from an earlier lesson.

Lessons: 16 · about 14 hours. Nothing is locked.

Progress tracking needs JavaScript. Every lesson is readable without it.

  1. The model can only talk

    1. 01 The function that forgetsYou call it twice. What does it remember?yields: a conversation your code owns and resendsharness: chat~35 minnot started
    2. 02 Words are not deedsThe model says it checked the file. How many times was the file read?yields: tools, tool calls, tool resultsharness: tool_calls, run_tool, chat with tools~50 minnot started
    3. 03 Turn the crankThe model asked for a tool. You ran it. Now what?yields: the agent loopharness: run_agent~40 minnot started

  2. Things go wrong

    1. 04 Tell the model what went wrongThe model asks for a tool called raed. Your code crashes. Who needed to know?yields: the tool boundaryharness: run_tool, str_arg~50 minnot started
    2. 05 Knowing when to stopYour loop stops when the model stops asking. What if it never does?yields: every exit leaves a sendable transcriptharness: error_message, max_turns, context_for_model~50 minnot started
    3. 06 The firehoseYour agent reads a 5,000-line log. How many times do you pay for it?yields: an output budget, truncation that says what it cutharness: truncate_head, truncate_tail, a budgeted read and bash~50 minnot started
    4. Checkpoint A Transcript doctorSix runs went wrong, in six different ways. Which idea fixes each one?~60 minnot started
    5. Side quest (optional) S1 One line, exactly onceA one-line fix costs a whole file. What would it take to send only the line?yields: edits that name one place in a fileharness: apply_edits, make_edit_tool~50 minnot started

  3. Watching and steering a run

    1. 07 Show your workYour agent takes forty seconds and shows nothing. Where do you put the print?yields: a loop that reports, and frontends that decide what it looks likeharness: run_agent as a generator, FinalTextRenderer, JsonRenderer~55 minnot started
    2. 08 Who holds the list?Your user double-clicks Send. Who is allowed to write to the transcript?yields: an object that owns the transcriptharness: Harness~45 minnot started
    3. 09 But I had something to sayThe agent is doing the wrong thing and the run is not over. How do you get a word in?yields: two queues the loop pulls fromharness: steer, follow_up, _drain~55 minnot started
    4. 10 The poisoned transcriptThe user walked away mid-tool. Now every prompt is refused. What do you do?yields: repair of the view, on every requestharness: INTERRUPTED, repair_tool_history~50 minnot started
    5. Checkpoint B Transcript doctor IIFive harnesses, five ways of being nearly right. Which idea is missing from each?~50 minnot started

  4. Memory that outlives the process

    1. 11 Pull the plugThe process dies mid-run. What does your agent know when it wakes up?yields: an append-only session log, resume by replay, subscribersharness: SessionLog, persist_to, Harness.subscribe~65 minnot started
    2. Side quest (optional) S2 RegretYou want to go back four messages and try again. What happens to the attempt you are leaving?yields: a session file that is a tree, and one branch of itharness: path, SessionLog.branch_to~50 minnot started
    3. 12 The wallThe prompt is too long and the agent is mid-task. What do you throw away?yields: compaction as a replay ruleharness: summarize, find_cut, compact, maybe_compact~65 minnot started

  5. From a brain to a coding agent

    1. 13 The briefingYour prompt promises a tool this agent does not have. Who wrote that promise?yields: a computed system promptharness: build_system_prompt, discover_context~55 minnot started
    2. 14 The vetoA file your agent reads tells it to delete your build directory. Who says no?yields: a fail-closed gate at the tool boundaryharness: guard, deny_destructive~45 minnot started
    3. Checkpoint C Where does it go?Four runs are broken and twelve features need a home. Where does each one go?~55 minnot started

  6. Meeting the real world

    1. 15 The stop button that does not stopA real build takes ten minutes. The user presses Stop at minute three. Who runs the button's code?yields: cooperative cancellation, with a backstopharness: CancelToken, Harness.cancel, stop_run~70 minnot started
    2. 16 Capstone: out of the browserSixteen lessons of harness, and it has never met a model. What has to change?yields: adapter.py, and a harness nobody had to editharness: to_anthropic, parse_sse~60 minnot started
    3. 16 Your machineThe fake kept its side of the bargain. What happens when nothing is fake?yields: a running local agent, and an honest list of what it cannot doharness: run_local.py, provider.py, real_tools.py~60 minnot started
    4. Checkpoint D Transcript doctor IIIEight runs from sixteen lessons, shuffled. Then one function nobody has shown you.~60 minnot started

Reading the map

The word at the right of each row is your state for that page. It is worked out from two things only: the questions you have locked an answer into, and the labs whose hidden tests your code has passed.

not started
Nothing on the page has been answered or passed.
in progress 3/9
You have done three of the page's nine items: its questions plus its labs.
built
Every lab on the page passes its hidden tests. The row also says how many questions are still open.
built with help
Everything is done, and for at least one lab you opened the reference solution before your own code passed. There is no penalty. The label is there so you know which labs to rebuild, and the next checkpoint is the place to do it.
mastered
Every question answered, and every lab passed without opening the solution first. Comparing with the reference after your tests pass does not count as help.

Progress records that you committed to an answer, never whether it was right. A wrong prediction is the method working: you are asked before you are told, and the reveal is a program you run. There are no timers, streaks or points.

All of it is stored in this browser and nowhere else. The links at the foot of every page export it to a file, import it in another browser, or reset it.

Six parts

One file, harness.py, grows through the whole course. This is what you own at the end of each part.

A. The model can only talk lessons 1 to 3
An agent: a model call in a while loop, over a transcript that you hold.
B. Things go wrong lessons 4 to 6, checkpoint A
A loop that survives bad tool calls, a model that never stops, provider failures and enormous tool output.
C. Watching and steering a run lessons 7 to 10, checkpoint B
Events for any frontend, a harness object that owns the transcript, user input in the middle of a run, and a transcript that one interrupted tool call cannot brick.
D. Memory that outlives the process lessons 11 and 12
An append-only session log, resume by replay, and compaction as a rule for reading the log, not a deletion.
E. From a brain to a coding agent lessons 13 and 14, checkpoint C
A system prompt that is computed from what is really enabled, skills loaded only when needed, and a gate around your tools that fails closed.
F. Meeting the real world lessons 15 and 16, checkpoint D
Async and a Stop button that stops, an adapter for a real provider's streaming API, real tools, and a real model on your own machine.

Three invariants

Three sentences carry the course. Each is introduced once, by a failure you cause, and after that it is only recalled by name.

  • I1 The transcript is the only memory, and all of it is re-read and re-paid on every call. planted in lesson 1 · named in lesson 6 · used in 12 and 13
  • I2 One call in, exactly one result out, right after it. half-built in lesson 2 · named in lesson 4 · probed in most lessons after
  • I3 The record is not the view: keep everything that happened; compute what you send. planted in lesson 5 · named in lesson 10 · used in 11 and 12

Every other term the course names is in the glossary, with the lesson that earns it.

Time and prerequisites

A lesson takes 35 to 70 minutes, and the sixteen together about 13 to 15 hours. Those figures are estimates from how the lessons are built, not yet measurements of real learners. The capstone, lesson 16, is two sessions in the browser followed by work on your own machine. Every lesson marks a good place to stop between its questions and its lab, and a two-lab lesson marks another between the labs.

You need Python functions, dicts, lists, classes and exceptions. You do not need generators or asyncio: lessons 7 and 15 each open with a three-cell warm-up that you can skip. You need no API key and no install. The Python runs in your browser, against a scripted model, until the last page of lesson 16, where you take your files home and may point them at a real one.

Checkpoints and side quests

A checkpoint follows lessons 6, 10, 14 and 16. It is not a lesson and it introduces nothing new. It mixes broken transcripts, event logs and session logs from the lessons before it, asks you to say what went wrong and which idea fixes it, and then has you rebuild one central piece in an empty editor, against tests you have already met. Checkpoint D is the final review: it samples the whole course and adds one small build that uses your harness in a way no lesson asked for.

The two side quests are optional, and nothing later depends on them. S1, after lesson 6, builds an edit tool that changes one line of a large file exactly once. S2, after lesson 11, turns the session log into a tree, so you can go back to an earlier message and try another approach without losing the first.