checkpoint A

Transcript doctor

Nothing new today. Six runs are broken, and nothing on the page says which lesson each one comes from.

~60 min · 2 labs · builds on 06 The firehose

What this page is

This is a checkpoint: a page with nothing new on it. Every idea it uses comes from lessons 1 to 6, the six problems below are shuffled, and nothing tells you which lesson each one belongs to. That is the whole design. Recognising an idea while it is being explained to you is easy; recognising it in a run that is already broken, on a Thursday afternoon, is the part that is worth practising.

Three parts. Six runs that are broken in six different ways, and for each one: what broke, and which idea fixes it. Then two labs. In the first you write the fake model and the test, which happens exactly once in this course. In the second you write the loop of lessons 3 to 5 back from an empty gap, against the tests you have already passed.

A word about the map, since this is the first time it matters. A row on the syllabus page counts two things: the questions you have committed an answer to, and the labs whose tests have passed. Finish them all and the row ends on one of two words — mastered, or built with help if you opened a solution on the way. That second mark stays where it is; nothing on this page rubs it out, because it is a record of what happened. What this page offers instead is the same work with the scaffolding gone. Nothing here is ever locked, nothing is timed and nothing is scored; the states exist so that you know which parts you have actually got, and the only person that information is for is you.

Six runs, and what went wrong in each

Each one is a loop somebody wrote, a transcript somebody saved, or a report from somebody using an agent. Nothing in any of them is exotic: what differs from the code you have is one line, one function, or the order of four messages. Commit an answer first, then run it and read what the difference cost. Two of the six arrive the way this sort of problem usually arrives — as a sentence from somebody who cannot see your code at all.

One. A teammate's loop, otherwise yours. When the model asks for a tool the loop does not have, there is nothing to run, so the loop writes a line to the log and moves on. Today the model asks for raed.

The model mistypes read as raed. Their loop logs a warning and skips the call. What happens to the run?

It goes round again carrying a call nobody answered, and the request is refused. The model is never told about the typo

One letter, and the run is over. Worse, so is the session: the record holds that call for ever.

The model notices that its call got no answer and asks again, spelling it correctly

That treats a record with a hole in it as an untidy detail the model can work around. It never gets the chance: the shape of the list is checked before the model is given anything to read.

The run finishes normally. An unknown tool is a mistake in your own tool list, and the warning in the log is how a developer hears about it

The warning goes somewhere only you can read it, and you are not the one who can fix this. The model wrote the name; the model is the reader who could spell it right the second time.

400 invalid_request: messages[1]: toolCall c1 has no toolResult, and every later prompt on that list gets the same answer. The message that would have told the model about the typo is the message the loop decided not to write. Yours writes it — Tool raed not found — and the model reads it and retries: three model calls, an answer, a record you could send again. A skipped call is not a call that never happened; it is a call with no result — lesson 4's rule, broken this time by the loop rather than by a tool.

Where this comes from: Tau answers a call to a tool it does not have with an ordinary result carrying Tool <name> not found, flagged as an error (src/tau_agent/loop.py:308-311). Dropping the call is not one of the options there either.

import harness, lab


def their_loop(model, system, messages, tools, prompt, *, max_turns=None):
    """Yours, with one line added: a call to a tool this loop does not know is
    written to the log and skipped."""
    known = {tool["name"] for tool in tools}
    turn = 0
    messages.append(harness.user_message(prompt))
    while True:
        turn += 1
        if max_turns is not None and turn > max_turns:
            reply = harness.error_message(f"Agent stopped after max_turns={max_turns}")
        else:
            reply = model.complete(system, harness.context_for_model(messages),
                                   harness.tool_specs(tools))
        messages.append(reply)
        if reply["stop_reason"] == "error":
            return reply
        calls = harness.tool_calls(reply)
        if not calls:
            return reply
        for call in calls:
            if call["name"] not in known:
                print("   log: WARNING unknown tool", repr(call["name"]))
                continue                        # the added line
            messages.append(harness.run_tool(tools, call))


def retry(request):
    """The model reads the newest result and tries again, if it is told anything."""
    result = request.last_result
    if result and result["is_error"] and "raed" in result["content"]:
        return lab.reply(lab.call("read", {"path": "config.py"}))
    return lab.say("I cannot continue.")


def script():
    return [lab.reply(lab.call("raed", {"path": "config.py"})), retry,
            lab.say("The port is 9090.")]


SYSTEM, PROMPT = "You are a careful debugger.", "What port does config.py use?"


def run(loop, name):
    ws = lab.Workspace({"config.py": "PORT = 9090\n"})
    model, messages = lab.ScriptedModel(script()), []
    loop(model, SYSTEM, messages, [harness.make_read_tool(ws)], PROMPT)
    print(f"{name}: {len(model.calls)} model call(s), record {lab.shape(messages)!r}")
    print("   ends on:", lab.show(messages[-1:]))
    return messages


their = run(their_loop, "theirs")
print()
run(harness.run_agent, "yours ")
print()
their.append(lab.user("Well? What port?"))
print("tomorrow, on their record:",
      lab.show([lab.ScriptedModel(script()).complete(SYSTEM, their, [])]))

   log: WARNING unknown tool 'raed'
theirs: 2 model call(s), record 'U A[c1] A(error)'
   ends on: assistant -> (nothing)  [error: 400 invalid_request: messages[1]: toolCall c1 has no toolResult]

yours : 3 model call(s), record 'U A[c1] R(c1) A[c2] R(c2) A'
   ends on: assistant -> "The port is 9090."

tomorrow, on their record: assistant -> (nothing)  [error: 400 invalid_request: messages[1]: toolCall c1 has no toolResult]

Two. A message from somebody using an agent built from these lessons. There is no code attached and nothing in the logs.

"I asked it whether we define def target anywhere in big.log. It said no. It is in there, on line 2,400. Nothing crashed, nothing was flagged. It just answered wrong, and it sounded certain."

Their read tool is yours: the same budget of whole lines, 2,000 of them or 50 KB, whichever runs out first, and the same description telling the model that offset reads on from anywhere. The only thing missing from it is the last line of the result.

Nothing raised, no result was flagged as an error, and the model answered exactly what it believed. Which mechanism is missing?

None is missing. The model should have noticed that a 5,000-line log does not end at a suspiciously round place, and asked for the rest

It received a string that ends line 01785: INFO request ok. Nothing in that string says the file goes on. The cut was visible on your side of the tool and invisible on the other.

The truncation notice. The result stopped at the budget and said nothing about it, so the part it was shown looked like the whole file

One line of text on the end of the result, and the run comes out differently. Watch what the model does with it.

The budget is too small. Raise it until real files fit and the answer comes out right

Right for this file. The next one is 200 KB of minified JavaScript with no line breaks at all, and every byte you let through is re-sent and re-paid on every turn that follows. The budget is not the bug; the silence about it is.

Their tool showed 1,785 lines of 5,000 and the model answered honestly about the text it had been given. Yours cuts in exactly the same place and ends the result with [Showing lines 1-1785 of 5000. Use offset=1786 to continue.], after which the model asked for the next page itself and found line 2,400 on the second read. Nobody wrote a loop that pages through a file. That behaviour is what a tool gets for free when it tells the truth about what it did — lesson 6, in one line of text — and a confident wrong answer is what it gets when it does not.

Where this comes from: when Tau's read stops at the budget, the result ends in a bracketed notice carrying the file's total line count and the exact offset to carry on from (src/tau_coding/tools.py:336-346).

import harness, lab


def make_quiet_read_tool(ws):
    """Their read tool: yours exactly - same budget, same offset argument, same
    description - with the last line of the result taken off."""
    tool = dict(harness.make_read_tool(ws))
    honest = tool["execute"]

    def execute(arguments):
        result = honest(arguments)
        lines = result.splitlines(keepends=True)
        return "".join(lines[:-1]) if lines and lines[-1].startswith("[") else result

    tool["execute"] = execute
    return tool


ws = lab.Workspace({"big.log": lab.big_log()})      # 5,000 lines, `def target` on line 2,400

for name, tool in (("theirs", make_quiet_read_tool(ws)), ("yours ", harness.make_read_tool(ws))):
    model, messages = lab.ScriptedModel([lab.pager("def target")]), []
    harness.run_agent(model, harness.SYSTEM, messages, [tool],
                      "Is there a `def target` in big.log?")
    for message in messages:
        if message["role"] == "toolResult":
            print(f"   {name} read ->", message["content"].splitlines()[-1])
    print(f"{name}: {len(model.calls)} model call(s),",
          f"{len([m for m in messages if m.get('is_error')])} error(s), it answered:",
          harness.text_of(messages[-1]))
    print()

print("the file, as the workspace holds it:", len(ws.read_text("big.log").splitlines()), "lines")
print("what their one result showed:      ",
      len(harness.truncate_head(ws.read_text("big.log"))[0].splitlines()), "lines")

   theirs read -> line 01785: INFO request ok
theirs: 2 model call(s), 0 error(s), it answered: `def target` does not exist in big.log.

   yours  read -> [Showing lines 1-1785 of 5000. Use offset=1786 to continue.]
   yours  read -> [Showing lines 1786-3571 of 5000. Use offset=3572 to continue.]
yours : 3 model call(s), 0 error(s), it answered: `def target` is on line 2400 of big.log.

the file, as the workspace holds it: 5000 lines
what their one result showed:       1785 lines

Three. A chat, not an agent: no tools, two turns. Their version saves tokens by sending only what is new, since the earlier messages already went out on the last call. The full list is still there in memory, untouched, for whenever it is needed.

Turn one, the user types My name is Ada. Turn two, they ask What is my name?. The model is lab.forgetful(), whose first printed rule is that it knows only what is in this request. What comes back, and what did the saving buy?

"Your name is Ada." The name was sent on the first call, to the same model, on the same account

That has the account keeping your conversation for you. Nothing on the other side joins two requests together: each one arrives alone and is answered from what is inside it.

"Your name is Ada," and cheaply: the provider has the first message cached, so not sending it again is exactly the saving they were after

[general] Providers that cache do it on a prefix of the request you send, to discount text you send again. A discount on what is in the request is not a stand-in for what you left out of it.

"I don't know your name." — and the saving is real: it is 29 tokens

Both requests were one message long. The list in their process is perfectly correct and nobody reads it.

Their two requests were U and U; yours were U and U A U, and the difference between 52 input tokens and 81 is the entire memory of the conversation. Sending the whole list on every call is not a thing your code does until you get round to optimising it. It is the only reason the second answer can be right. Lesson 1, with a price tag on it.

Where this comes from: Tau's loop builds the list it sends out of the session's messages, again, on every model call (src/tau_agent/loop.py:130). There is no shorter list kept anywhere for it to send instead.

import harness, lab


def their_chat(model, system, messages, text):
    """Their saving: the model has already been sent the earlier messages, so this
    request carries only what is new since the last call."""
    new_from = len(messages)
    messages.append(harness.user_message(text))
    reply = model.complete(system, messages[new_from:], [])       # the changed line
    messages.append(reply)
    return reply


def your_chat(model, system, messages, text):
    """Yours: run_agent with an empty tool list, which is all `chat` ever was."""
    return harness.run_agent(model, system, messages, [], text)


for name, chat in (("theirs", their_chat), ("yours ", your_chat)):
    model, messages = lab.ScriptedModel([lab.forgetful()]), []
    chat(model, harness.SYSTEM, messages, "My name is Ada.")
    chat(model, harness.SYSTEM, messages, "What is my name?")
    for number, request in enumerate(model.calls, start=1):
        print(f"   {name} request {number}: {lab.shape(request.messages)!r}")
    print(f"{name}: it answered {harness.text_of(messages[-1])!r},",
          f"for {model.bill} input tokens")
    print()

   theirs request 1: 'U'
   theirs request 2: 'U'
theirs: it answered "I don't know your name.", for 52 input tokens

   yours  request 1: 'U'
   yours  request 2: 'U A U'
yours : it answered 'Your name is Ada.', for 81 input tokens

Four. A transcript of one turn, four messages, jumbled by a tool that sorted them on the way to disk. One reply, two calls, two results. Put them back into the only order a provider will accept.

Put these in the order that goes out in the next request.

user — "Read a.py and b.py."
assistant — a reply holding toolCall c1 read(a.py) and toolCall c2 read(b.py)
toolResult c1 — "A = 1"
toolResult c2 — "B = 2"

Both results go after the reply that asked for them, and in the order that reply asked. The second half is the one people drop: put c2 first and the list is refused, although every result is present and every id is correct. The refusal names the first thing it trips over — toolCall c1 has no toolResult — and the checker in the cell says the rest: toolResult c2 must directly follow its assistant message, in call order. The ids are not a way out of keeping the order; they are what tells you which result is which once the order is kept. Lesson 2 paired a call with a result by hand; lesson 4 made the pairing a rule that holds on every path.

The cell below sends all three arrangements to a strict model.

Where this comes from: [general] hosted providers refuse a list whose tool results are not beside the calls they answer. The rule is strict enough that Tau keeps a function whose whole job is to guarantee "every tool call has exactly one adjacent result" before a provider is given the list (src/tau_agent/tool_history.py:40-47). Moving a result back beside its call is one of the four things it does.

The same four messages, arranged three ways: what the provider answers each time, and every complaint lab.validate can make about the list.

import lab

c1 = lab.call("read", {"path": "a.py"}, id="c1")
c2 = lab.call("read", {"path": "b.py"}, id="c2")
prompt = lab.user("Read a.py and b.py.")
asked = lab.reply(c1, c2)
got_a = lab.tool_result(c1, "A = 1\n")
got_b = lab.tool_result(c2, "B = 2\n")

arrangements = {
    "results first, in the order they arrived": [prompt, got_a, got_b, asked],
    "after the call, b.py first":               [prompt, asked, got_b, got_a],
    "after the call, in call order":            [prompt, asked, got_a, got_b],
}

for name, messages in arrangements.items():
    model = lab.ScriptedModel([lab.say("a.py sets A = 1 and b.py sets B = 2.")])
    reply = model.complete("You are a careful debugger.", messages, [])
    print(f"{name}\n   {lab.shape(messages)}")
    print("   the provider:", lab.show([reply]))
    for complaint in lab.validate(messages):
        print("   ", complaint)
    print()

results first, in the order they arrived
   U R(c1) R(c2) A[c1,c2]
   the provider: assistant -> (nothing)  [error: 400 invalid_request: messages[1]: toolResult c1 has no toolCall before it]
    messages[1]: toolResult c1 has no toolCall before it
    messages[2]: toolResult c2 has no toolCall before it
    messages[3]: toolCall c1 has no toolResult

after the call, b.py first
   U A[c1,c2] R(c2) R(c1)
   the provider: assistant -> (nothing)  [error: 400 invalid_request: messages[1]: toolCall c1 has no toolResult]
    messages[1]: toolCall c1 has no toolResult
    messages[2]: toolResult c2 must directly follow its assistant message, in call order
    messages[3]: toolResult c1 must directly follow its assistant message, in call order

after the call, in call order
   U A[c1,c2] R(c1) R(c2)
   the provider: assistant -> "a.py sets A = 1 and b.py sets B = 2."

Five. A second message, from somebody else, about a different agent.

"Our provider had an outage on Tuesday afternoon. One prompt of mine failed while it was down, which is fair enough. Since then every prompt in that conversation fails, and not with the provider's error — with a 400. A brand-new conversation works perfectly."

Their loop is yours, one function short.

The provider has been back for two days. Every prompt in that one session is refused; a new session is fine. What is happening?

The provider remembers the request that failed and refuses the conversation it belongs to until the session is started again

Nothing on the other side knows these requests belong together. Each one arrives on its own and is judged on what is inside it, which is why a new conversation works: it is not new, it is just shorter.

The failed reply is on their record with nothing in it, and it is being sent. Nothing stands between the record and the request, so every request built from that record carries it

[general] Hosted providers do not reliably accept an assistant turn with nothing in it. The outage lasted a minute; the damage is permanent, because it was written down.

The record is corrupt. Delete the failed message from it and the session works again

It does work, and it throws away the only trace that anything went wrong. The failure happened; a record that leaves it out is a record that lies about a minute of the afternoon. There is something else you are allowed to edit.

400 invalid_request: messages[1]: assistant message has empty content, three prompts running, and it would not have stopped. The missing function is context_for_model: the record keeps the failed reply, because it is what happened, and the request is computed from the record with that one message left out. Yours sends U U and gets an answer on the first try, with nothing deleted and nothing repaired: lesson 5's split between the record and the request, doing its job two days later.

Where this comes from: Tau computes the request from the record the same way, once per model call, in a function whose docstring is the other half of this run — failures are kept for diagnostics, and an empty failed turn "is not model context and must not poison the next request" (src/tau_agent/loop.py:185-201).

import harness, lab


def their_loop(model, system, messages, tools, prompt, *, max_turns=None):
    """Yours, with one line changed: the request is the record itself."""
    turn = 0
    messages.append(harness.user_message(prompt))
    while True:
        turn += 1
        if max_turns is not None and turn > max_turns:
            reply = harness.error_message(f"Agent stopped after max_turns={max_turns}")
        else:
            reply = model.complete(system, messages, harness.tool_specs(tools))  # the change
        messages.append(reply)
        if reply["stop_reason"] == "error":
            return reply
        calls = harness.tool_calls(reply)
        if not calls:
            return reply
        for call in calls:
            messages.append(harness.run_tool(tools, call))


script = [lab.fail("503 overloaded"), lab.say("The port is 9090."),
          lab.say("Still 9090."), lab.say("Yes, still 9090.")]

# Tuesday, 16:40: the provider fell over on one prompt.
model, messages = lab.ScriptedModel(list(script)), []
their_loop(model, harness.SYSTEM, messages, [], "Find the port.")
print("after the outage, their record:", lab.shape(messages))

# Thursday: three more prompts on the same session.
for prompt in ("try again", "are you there?", "hello?"):
    their_loop(model, harness.SYSTEM, messages, [], prompt)
    print(f"   {prompt!r} ->", lab.show(messages[-1:]))

print()
print("the record, sent as it stands:", lab.shape(messages[:2]))
print("the same record, through context_for_model:",
      lab.shape(harness.context_for_model(messages[:2])))
model, messages = lab.ScriptedModel(list(script)), []
harness.run_agent(model, harness.SYSTEM, messages, [], "Find the port.")
harness.run_agent(model, harness.SYSTEM, messages, [], "try again")
print("yours, the same two prompts:", lab.shape(messages), "->",
      harness.text_of(messages[-1]))

after the outage, their record: U A(error)
   'try again' -> assistant -> (nothing)  [error: 400 invalid_request: messages[1]: assistant message has empty content]
   'are you there?' -> assistant -> (nothing)  [error: 400 invalid_request: messages[1]: assistant message has empty content]
   'hello?' -> assistant -> (nothing)  [error: 400 invalid_request: messages[1]: assistant message has empty content]

the record, sent as it stands: U A(error)
the same record, through context_for_model: U
yours, the same two prompts: U A(error) U A -> The port is 9090.

Six. A loop that begins by making its own list — messages = list(messages) — so that a run in progress cannot disturb the caller's list. Every other line is yours. It answers the first question correctly, and then the user asks a second one.

Run one takes two model calls and answers correctly. The caller then hands the same list to a second run with a second question. How many messages does that run's first request carry?

One: the new prompt, on its own. The caller's list is empty and always will be, so every run starts from nothing, reads the same file again and pays for it again. The loop was guarding the caller's list against a writer that does not exist, and what it was guarding them from was the conversation. Lesson 3 in one line: the caller's list is the product of a run, and the return value is a convenience. Run the cell and look at the second line of each block.

Where this comes from: Tau's loop appends to the list it was handed, and the test for an ordinary turn asserts on that same list afterwards (tests/test_agent_loop.py:85). A loop that copied the list would fail that line.

Their loop and yours, the same two prompts: what the caller holds afterwards, what the second run's first request carried, which files were read, and what the two runs cost between them.

import harness, lab


def their_loop(model, system, messages, tools, prompt, *, max_turns=None):
    """Yours, with one line added at the top: the loop works on a list of its own."""
    messages = list(messages)                   # the added line
    turn = 0
    messages.append(harness.user_message(prompt))
    while True:
        turn += 1
        if max_turns is not None and turn > max_turns:
            reply = harness.error_message(f"Agent stopped after max_turns={max_turns}")
        else:
            reply = model.complete(system, harness.context_for_model(messages),
                                   harness.tool_specs(tools))
        messages.append(reply)
        if reply["stop_reason"] == "error":
            return reply
        calls = harness.tool_calls(reply)
        if not calls:
            return reply
        for call in calls:
            messages.append(harness.run_tool(tools, call))


def answer(request):
    """Answers from what this request shows it, and reads config.py when it must."""
    shown = "\n".join(m["content"] for m in request.messages if m["role"] == "toolResult")
    wanted = "host" if "host" in request.user_text.lower() else "port"
    line = next((x for x in shown.splitlines() if x.lower().startswith(wanted)), None)
    return lab.say(f"The {wanted} is {line.split(' = ')[1]}.") if line else \
        lab.reply(lab.call("read", {"path": "config.py"}))


for name, loop in (("theirs", their_loop), ("yours ", harness.run_agent)):
    ws = lab.Workspace({"config.py": "PORT = 9090\nHOST = 0.0.0.0\n"})
    model, messages = lab.ScriptedModel([lab.forever(answer)]), []
    loop(model, harness.SYSTEM, messages, [harness.make_read_tool(ws)], "What port?")
    print(f"{name}: after the first run the caller holds {len(messages)} message(s):",
          repr(lab.shape(messages)))
    loop(model, harness.SYSTEM, messages, [harness.make_read_tool(ws)], "And the host?")
    print(f"   the second run's first request: {lab.shape(model.calls[2].messages)!r}")
    print(f"   {len(model.calls)} model call(s) in all, {ws.reads} read,",
          f"{model.bill} input tokens")

theirs: after the first run the caller holds 0 message(s): ''
   the second run's first request: 'U'
   4 model call(s) in all, ['config.py', 'config.py'] read, 653 input tokens
yours : after the first run the caller holds 4 message(s): 'U A[c1] R(c1) A'
   the second run's first request: 'U A[c1] R(c1) A U'
   3 model call(s) in all, ['config.py'] read, 552 input tokens

What three of them had in common

Three of the six ended with a list a provider would not read: the skipped call, the jumbled transcript and the outage. Two different rules were broken between them. Name both, and say which one your harness keeps by never being able to break it, and which one it keeps by computing something else.

One of the two is about what must sit next to what. The other is about two lists that people keep mistaking for one.

The first rule is one call in, exactly one result out, right after it, in the order the reply asked. The skipped call broke it by writing no result at all; the jumbled transcript broke it by moving the results around. Your loop has no branch that can drop a call: the only thing it ever does with one is hand it to run_tool, which returns exactly one result whatever happened — unknown tool, raised tool, bad arguments — and the loop appends them in the order it ran them.

The second rule is that what happened and what you send are not the same list. The outage left a perfectly honest record, and nothing in that harness computed a request from it. Your context_for_model builds the view fresh before every model call and leaves the empty failed reply out of it; the record keeps it, because it is true.

Common answers, and what each one misses

"Never let a run crash." Two of those three never crashed. Nothing raised, nothing was logged, and the session was dead all the same — which is why "it did not throw" has never been evidence of anything on this site.
"Check the transcript before every request and fix what is broken." Worth doing, and it is a different job from either rule: a check can tell you a result is missing, and only the loop was ever in a position to write it. What to do about damage you did not cause is a lesson of its own, later in the course.
"Keep the record clean: drop whatever the provider will not take." That is the one edit you may not make. The record is the only account of what happened, and it is what tomorrow's resume reads. Edit the request instead: it is computed, it is cheap, and it is thrown away the moment it has been sent.

Figure A.1 The four shapes, from lesson 4. Only the first is a list a provider will read. The skipped call left the second: a call nobody answered. The jumbled transcript lost nothing — every message is there and every id is right — and was refused twice over anyway: once for a result standing in front of the call it answers, once for a result that is not right after its call.

Build one: the script on which two loops part company

Every model you have met on this site was a script somebody wrote. This is the one place in the course where that somebody is you. Beside the editor sits loops.py, read-only, holding two loops that differ by one line. One of them is wrong. On almost every script they behave identically, and your job is to find the almost.

reference is the loop you wrote in lesson 5. buggy is the same file with one line changed: it goes round again while the reply is labelled "toolUse", rather than while the reply holds calls. Both are in loops.py, with the handful of helpers they need copied in so that the file runs on its own.

Two functions in your editor, ten lines or so between them.

script() returns the model's side of the run: a list of steps for lab.ScriptedModel, one per model call, in order. The builders are lab.reply, lab.say, lab.text, lab.call and lab.fail, and lab.reply(..., stop_reason="stop") is how a reply is given a label its content does not support.
check(model, messages, final) holds one assert about what the run left behind. messages is the record the loop was given and appended to, model.calls is every request that went out, and final is whatever the loop returned. lab.shape, lab.show and lab.diff are there to make the failure say something.

The hidden tests build the model and run both loops themselves, on the same tools and the same prompt: one read tool that always returns PORT = 9090, the prompt "Find the port.", and a cut-off at twelve model calls for a script that never lets a loop stop. Your check is handed the run and nothing else, so it cannot ask which loop it is looking at.

Four tests. Your script is one the reference loop can finish; the reference loop passes your check; the buggy loop fails it; and when it fails it says what the loop did and what you expected instead. That last one is the one a test can barely police — all it can insist on is that your assert carries a message at all — and it is the one that matters longest. It is the sentence somebody reads in six months, at the moment they least want to go and read the test.

Both loops append the reply, and then they ask different questions about it. Which reply gives those two questions different answers — and can you write that reply down with lab.reply and lab.call? Then work out what each loop does for the rest of the run, and which of the three things your check is handed would show the difference.
One reply does it: a reply that holds a read call and is labelled "stop". The reference loop asks the content, finds a call, runs it, appends the result and goes round; the buggy loop asks the label, does not see toolUse, and returns on the spot with the call unanswered. So your script needs a second step as well — the answer the reference loop will ask for on its second call — or it runs out of script and the test about finishing fails. Then assert something true of the run the reference made and false of the run the buggy one made; the whole record in one string, from lab.shape(messages), is the shortest thing to compare.

In outline.

script():
    two steps:
      1. a reply holding lab.call("read", {"path": "config.py"}),
         given stop_reason="stop" so the label lies
      2. lab.say(...): what the model answers once it has
         been shown the file

check(model, messages, final):
    expected = the record a loop that reads the content leaves:
               the prompt, the reply, the result, the answer
    assert lab.shape(messages) == expected, a message that says
        what the model asked for, how many model calls were made
        (len(model.calls)), what came back (final), what the run
        left (lab.shape(messages)), what you expected, and why a
        call is a call whatever the label says; end it with
        lab.show(messages)

You now own a test: two functions that state what a loop must do, and that would notice the day it stopped doing it. Look at what made it work: your script contains a reply whose label and content disagree. A script of honest replies — and that is nearly every script anyone writes — runs through both loops identically and proves nothing. That is what a regression test is — the one input on which correct and nearly-correct part company — and it is why the suite in the next lab has a test about labels in it at all.

script() is a non-empty list of steps, and lesson 5's loop runs it to the end without the lab having to cut the run off.
The loop that reads the reply's content is handed to your check, and your check says nothing is wrong.
The loop that believes the reply's label is handed to your check, and your check fails it.
When your check fails, it says what the loop did and what it should have done, not just that something is false.

"""Checkpoint A: write the test that catches the bug. `loops.py`, beside this editor, holds two loops. `buggy` decides whether there is more to do by reading the reply's label, `stop_reason`; `reference` reads the reply's content. Write the script a model would have to follow for that difference to show, and one assert about what the run left behind. Your script and your check are then run twice: against `reference`, where the check must pass, and against `buggy`, where it must fail. This is the one place in the course where you write the fake. """ import lab def script(): """The model's side of the run: one step per model call, for lab.ScriptedModel.""" # --- your code --- # One step per model call, in order. The builders are lab.reply, lab.say, lab.text, # lab.call and lab.fail; lab.reply(..., stop_reason=...) is how a reply is given a label # that its content does not support. return [] # --- end --- def check(model, messages, final): """One assert about what the run left behind: true of a loop that reads the reply's content, false of a loop that believes its label.""" # --- your code --- # `messages` is the record the loop was given and appended to, `model.calls` is every # request that went out, `final` is what the loop returned. lab.shape, lab.show and # lab.diff are there to make the failure message say what went wrong. # --- end ---Running code needs JavaScript. Above is the file this exercise starts from (my_test.py). You can run its tests on your own machine: python lab/lab_check.py --harness my_test.py --lesson 6 tests.py

Your script and your check against both loops, with the message your assert prints when it fails. No tests: read your own failure message and ask whether it would have been enough. Once the lab has passed, this runs your file.

import harness, lab, loops

CONFIG = "PORT = 9090\n"
read = {"name": "read", "description": "Read a text file and return its contents.",
        "parameters": {"type": "object", "required": ["path"],
                       "properties": {"path": {"type": "string"}}},
        "execute": lambda arguments: CONFIG}

for name in ("reference", "buggy"):
    loop = getattr(loops, name)
    steps = await lab.drive(harness.script())
    model, messages = lab.ScriptedModel(list(steps), max_calls=12), []
    final = await lab.drive(loop(model, "You are a careful debugger.", messages,
                                 [read], "Find the port."))
    print(f"{name}: {len(model.calls)} model call(s), record {lab.shape(messages)!r}")
    try:
        await lab.drive(harness.check(model, messages, final))
        print("   your check passed it.")
    except AssertionError as exc:
        print("   your check failed it:", exc)
    print()

reference: 2 model call(s), record 'U A[c1] R(c1) A'
   your check passed it.

buggy: 1 model call(s), record 'U A[c1]'
   your check failed it: The model asked to read config.py in a reply labelled "stop", and would have answered on the next call. The loop made 1 model call(s), came back with 'assistant -> toolCall c1 read({"path": "config.py"})' and left 'U A[c1]'; expected 'U A[c1] R(c1) A'. A call is a call whatever the label says: the tool runs and its result goes back.
user -> "Find the port."
assistant -> toolCall c1 read({"path": "config.py"})

Build two: the loop, from an empty gap

The same harness.py you left at the end of lesson 6, with three names cut out of it. No spec paragraph, no step comments, no example test: the gaps hold the names and their argument lists and one sentence saying that nothing here is new. That is the difference between reading code you understand and being able to produce it, and it is the only way to find out which of the two you have.

Gone from the file: error_message from region 1, run_agent and context_for_model from region 3. Everything else is where you left it — the tools, the budgets, the truncation, run_tool. Thirty-two lines in the reference, docstrings included: error_message is four, context_for_model six, run_agent twenty-two.

Twenty-three hidden tests. Twenty-two of them are copied out of lessons 3, 4 and 5 word for word — ten, two and ten — so that what you rebuild is measured by the ruler that measured what you built: the job that needs three files each named inside the last, the request that is the record so far, the reply whose label lies in both directions, two calls in one reply, the twelve-turn job, the typo the model recovers from, three calls of which the second fails, max_turns against a model that will not stop, the 503 that is written down instead of raised, and the view that drops an empty failed reply and keeps a half-finished one. The twenty-third is this checkpoint's own: while the gaps are empty every one of the other twenty-two can only report a missing name, so one test says which names are missing and what each of them is for.

Nothing on this page has told you how to write them, and that is deliberate. If you get stuck, lessons 3 to 5 are still there and reading them costs you nothing except the answer to the question this lab is asking.

Three questions, one for each name. In run_agent: what ends a run? Write down every way one can end before you write a line — there are three, and two of them arrived in lesson 5. In context_for_model: of everything on the record, which single kind of message may a provider not be shown, and why is it on the record at all? In error_message: the words belong in a field of their own rather than in content — which field, and what does stop_reason say?
Append the prompt to the caller's list, then loop. At the top of each turn count the turn; if the count has passed max_turns, the reply is an error_message saying so and no model call is made and nothing is paid for. Otherwise call the model with the system prompt you were handed, the view of the record and the tool specs. Append the reply whatever it is. If it came back failed, return it. If it holds no calls, return it. Otherwise run every call in the order the model wrote them, appending each result, and go round. context_for_model returns a new list and leaves the record alone; the only thing it drops is an assistant message that failed and has nothing in it, because a failed reply that got some text out is something the model did say.

In outline.

error_message(text):
    an assistant message, content [], stop_reason "error",
    and the words under the key "error_message"

run_agent(model, system, messages, tools, prompt, *, max_turns=None):
    turn = 0
    append user_message(prompt) to the caller's list
    forever:
        turn += 1
        if there is a limit and turn is past it:
            reply = error_message(f"Agent stopped after max_turns={max_turns}")
        else:
            reply = model.complete(system, context_for_model(messages),
                                   tool_specs(tools))
        append reply
        if reply's stop_reason is "error":   return reply
        calls = tool_calls(reply)
        if not calls:                        return reply
        for each call: append run_tool(tools, call)

context_for_model(messages):
    a new list: every message except an assistant one that has
    empty content and stop_reason "error"

Twenty-three tests written for three different lessons, passed by thirty-two lines you produced from an empty gap. Nothing in them was new, which is exactly what makes it worth the half hour: the loop is now something you can write, not something you can follow. If any of lessons 3, 4 or 5 is marked built with help on the map, the mark stays — it records what happened that afternoon — and it is now out of date.

The three names of the rebuild are in the file. Until they are, every test below can only say so.
A job that needs three files takes four model calls, and nobody cranks by hand.
Every request is the whole list as it stood: two messages longer than the one before.
Everything that happened is in the list the caller passed in, and in no other.
A reply that holds a call gets its result and another turn, even when labelled "stop".
After a run, the same list takes another prompt and the strict model accepts it.
A reply with no calls ends the run, even when it is labelled "toolUse" or "length".
One reply asks for two files: both results follow it, in call order, before the next turn.
The loop has no turn count of its own: twelve requests in a row get twelve results.
With no tools and a plain answer, the run is one model call: what chat used to do.
run_agent returns the final assistant message, the one that asked for nothing more.
The model mistypes a tool name, reads the error, retries with the right name: three model calls.
Three calls in one reply, the second fails: three results, in call order, only the second an error.
error_message(text) is an assistant message with no content, stop_reason "error", and the text in error_message.
max_turns=1 against a stuck model: one model call, its tool call answered, then an in-band stop.
max_turns=3 allows exactly three model calls, in every run, however long the list already is.
max_turns=None means no limit: twelve requests in a row are all served, and a limit of 20 changes nothing.
The view is worked out again for every model call: the model sees the result of the tool it just asked for.
The provider answers 503: the failed reply goes on the record and the run ends without raising.
After a 503 the user says "try again": the failed message stays on the record and is not sent.
A reply that failed after producing text is sent back to the model like any other message.
context_for_model drops assistant messages that failed and are empty, keeps the rest, and never edits the record.
A run stopped by max_turns leaves a list the next prompt is accepted on, and the model recovers.

# == 1 messages == # A conversation is a list of messages. Each message is a dict tagged with the role that spoke. SYSTEM = "You are a helpful assistant." # standing instructions, sent with every call def user_message(text): """A message from the user. Its content is a plain string.""" return {"role": "user", "content": text} def text_of(message): """The words of an assistant message. Its content is a list of blocks, not a string (lesson 2 explains why); the words are in the blocks of type "text".""" return "".join(block["text"] for block in message["content"] if block["type"] == "text") def tool_calls(message): """The tool calls an assistant message asks for, in the order it asked. Worked out from `content` on every call and stored nowhere else.""" return [block for block in message["content"] if block["type"] == "toolCall"] # --- your code --- # error_message(text) # --- end --- # == 2 tools == # A tool is a dict. "name", "description" and "parameters" are advertised to the model. # "execute" is a function, execute(arguments) -> str, that only this program ever runs. def str_arg(arguments, name): """The string argument `name`, checked. The schema asked the model for a string and nobody made it send one, so every tool reads its arguments through here. Raises ValueError("<name> must be a string") when it is missing or not a string.""" value = arguments.get(name) if not isinstance(value, str): raise ValueError(f"{name} must be a string") return value def int_arg(arguments, name, default): """The optional whole-number argument `name`, checked; `default` if the model left it out. Raises ValueError("<name> must be an integer") for anything else.""" value = arguments.get(name) if value is None: return default if not isinstance(value, int): raise ValueError(f"{name} must be an integer") return value def tool_specs(tools): """What the model is told about the tools: everything except the function.""" return [{"name": tool["name"], "description": tool["description"], "parameters": tool["parameters"]} for tool in tools] def run_tool(tools, call): """Run one tool call and return the toolResult message that answers it. One call in, exactly one result out, on every path. Tools just raise; this is the one place that turns a failure into a result the model can read. is_error means the tool could not do its job. It never raises Exception.""" tools_by_name = {tool["name"]: tool for tool in tools} tool = tools_by_name.get(call["name"]) if tool is None: content, is_error = f"Tool {call['name']} not found", True else: try: content, is_error = tool["execute"](call["arguments"]), False except Exception as exc: content, is_error = str(exc), True return {"role": "toolResult", "tool_call_id": call["id"], "tool_name": call["name"], "content": content, "is_error": is_error} MAX_LINES = 2000 # The budget of one tool result. Whatever a tool returns is re-read, and MAX_BYTES = 50_000 # paid for, on every later turn. Bytes, not characters: "é" is two. def truncate_head(text, max_lines=MAX_LINES, max_bytes=MAX_BYTES): """The start of `text` that fits the budget, for files: (kept, lines_shown). Whole lines only, newlines included, sized in UTF-8 bytes. If even the first line is too big, the answer is ("", 0): half a line is worse than none.""" kept, size = [], 0 for line in text.splitlines(keepends=True): size += len(line.encode()) if len(kept) == max_lines or size > max_bytes: break kept.append(line) return "".join(kept), len(kept) def truncate_tail(text, max_lines=MAX_LINES, max_bytes=MAX_BYTES): """The end of `text` that fits the budget, for command output, where the verdict comes last: (kept, lines_shown). Whole lines only, newlines included, sized in UTF-8 bytes.""" kept, size = [], 0 for line in reversed(text.splitlines(keepends=True)): size += len(line.encode()) if len(kept) == max_lines or size > max_bytes: break kept.append(line) return "".join(reversed(kept)), len(kept) def make_read_tool(ws, max_lines=MAX_LINES, max_bytes=MAX_BYTES): """The `read` tool, over the workspace `ws`. One call returns at most the budget, and says so in a last line that tells the model how to ask for the rest.""" def execute(arguments): path = str_arg(arguments, "path") offset, limit = int_arg(arguments, "offset", 1), int_arg(arguments, "limit", None) if offset < 1 or (limit is not None and limit < 1): raise ValueError("offset and limit must be 1 or more") if not ws.exists(path): raise FileNotFoundError( f"File not found: {path}. Files here: {', '.join(ws.listdir('.'))}") lines = ws.read_text(path).splitlines(keepends=True) wanted = lines[offset - 1:] if limit is None else lines[offset - 1:offset - 1 + limit] kept, shown = truncate_head("".join(wanted), max_lines, max_bytes) if offset > max(len(lines), 1): raise ValueError(f"offset {offset} is past the end of {path}: {len(lines)} lines") if wanted and shown == 0: skip = f" Use offset={offset + 1} to skip it." if offset < len(lines) else "" return (f"[Line {offset} of {path} is {len(wanted[0].encode())} bytes, over the " f"{max_bytes}-byte budget of one read, so none of it is shown.{skip}]") last = offset + shown - 1 if last < len(lines): kept += (f"[Showing lines {offset}-{last} of {len(lines)}. " f"Use offset={last + 1} to continue.]") return kept return { "name": "read", "description": "Read a text file and return its contents. One call returns at most " f"{max_lines} lines or {max_bytes} bytes; use offset to read on.", "parameters": { "type": "object", "properties": {"path": {"type": "string", "description": "Path of the file to read."}, "offset": {"type": "integer", "description": "First line, from 1."}, "limit": {"type": "integer", "description": "How many lines."}}, "required": ["path"], }, "execute": execute, } def make_write_tool(ws): """The `write` tool, over the workspace `ws`: create a file, or replace all of it.""" def execute(arguments): path, content = str_arg(arguments, "path"), str_arg(arguments, "content") ws.write_text(path, content) return f"Successfully wrote to {path}." return { "name": "write", "description": "Write a text file. Creates the file, or replaces its whole contents.", "parameters": { "type": "object", "properties": {"path": {"type": "string", "description": "Path of the file to write."}, "content": {"type": "string", "description": "The new contents."}}, "required": ["path", "content"], }, "execute": execute, } def make_bash_tool(shell, max_lines=MAX_LINES, max_bytes=MAX_BYTES): """The `bash` tool: run one command line in `shell` and return what it printed. A command that fails has still been run: its exit code is news for the model, not a tool error, so it comes back as an ordinary result that ends with a status line.""" def execute(arguments): output, code = shell.run(str_arg(arguments, "command")) total = len(output.splitlines()) output, shown = truncate_tail(output, max_lines, max_bytes) if shown < total: output += f"\n[Showing the last {shown} of {total} lines. Narrow the command for more.]" if code != 0: output += f"\nCommand exited with code {code}" return output return { "name": "bash", "description": "Run a shell command and return its output (stdout and stderr together). " f"Only the last {max_lines} lines or {max_bytes} bytes come back.", "parameters": { "type": "object", "properties": {"command": {"type": "string", "description": "The command line."}}, "required": ["command"], }, "execute": execute, } # == 3 loop == # The code that talks to the model. The model keeps nothing between calls. # --- your code --- # From nothing, against the tests of lessons 3, 4 and 5: # run_agent(model, system, messages, tools, prompt, *, max_turns=None) # context_for_model(messages) # Nothing here is new. Those tests are the ones you have already passed. # --- end ---Running code needs JavaScript. Above is the file this exercise starts from (harness.py). You can run its tests on your own machine: python lab/lab_check.py --harness harness.py --lesson 6 tests.py

Four runs, no tests. A mistyped tool name, and then a second prompt on the record it left. Five thousand lines read through a budget. A model that will not stop, cut off at three turns. A provider that falls over, and the prompt after it. Then the same question of all four records: could you send this list again? Once the lab has passed, this runs your code.

import harness, lab

ws = lab.Workspace({"config.py": "PORT = 9090\n", "big.log": lab.big_log()})
tools = [harness.make_read_tool(ws)]


def run(label, model, messages, prompt, **options):
    harness.run_agent(model, harness.SYSTEM, messages, tools, prompt, **options)
    print(f"{label:10}: {len(model.calls)} model call(s), {lab.shape(messages)}")
    print("            ends on:", lab.show(messages[-1:]))


def retry(request):
    result = request.last_result
    if result and result["is_error"] and "raed" in result["content"]:
        return lab.reply(lab.call("read", {"path": "config.py"}))
    return lab.say("I cannot continue.")


# 1. A mistyped tool name, and then a second prompt on the record it left.
typo = []
model = lab.ScriptedModel([lab.reply(lab.call("raed", {"path": "config.py"})), retry,
                           lab.say("The port is 9090."), lab.say("You are welcome.")])
run("a typo", model, typo, "What port does config.py use?")
run("and again", model, typo, "Thanks.")

# 2. Five thousand lines, and a model that follows the notices.
firehose = []
run("a firehose", lab.ScriptedModel([lab.pager("def target")]), firehose,
    "Is there a `def target` in big.log?")

# 3. A model that will not stop, cut off at three turns.
runaway = []
run("a runaway", lab.ScriptedModel([lab.stuck(lab.call("read", {"path": "missing.py"}))]),
    runaway, "Find the port.", max_turns=3)

# 4. A provider that falls over, and the prompt after it.
outage = []
model = lab.ScriptedModel([lab.fail("503 overloaded"), lab.say("The port is 9090.")])
run("an outage", model, outage, "Find the port.")
run("try again", model, outage, "try again")

print()
for label, messages in (("a typo", typo), ("a firehose", firehose),
                        ("a runaway", runaway), ("an outage", outage)):
    print(f"{label:10}: record valid: {lab.validate(messages, record=True) == []}"
          f" | what the next request would carry: "
          f"{lab.shape(harness.context_for_model(messages))}")

a typo    : 3 model call(s), U A[c1] R(c1) A[c2] R(c2) A
            ends on: assistant -> "The port is 9090."
and again : 4 model call(s), U A[c1] R(c1) A[c2] R(c2) A U A
            ends on: assistant -> "You are welcome."
a firehose: 3 model call(s), U A[c1] R(c1) A[c2] R(c2) A
            ends on: assistant -> "`def target` is on line 2400 of big.log."
a runaway : 3 model call(s), U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3) A(error)
            ends on: assistant -> (nothing)  [error: Agent stopped after max_turns=3]
an outage : 1 model call(s), U A(error)
            ends on: assistant -> (nothing)  [error: 503 overloaded]
try again : 2 model call(s), U A(error) U A
            ends on: assistant -> "The port is 9090."

a typo    : record valid: True | what the next request would carry: U A[c1] R(c1) A[c2] R(c2) A U A
a firehose: record valid: True | what the next request would carry: U A[c1] R(c1) A[c2] R(c2) A
a runaway : record valid: True | what the next request would carry: U A[c1] R(c1) A[c2] R(c2) A[c3] R(c3)
an outage : record valid: True | what the next request would carry: U U A

Four runs, four different ways for an afternoon to go wrong, and four records that each still yield a request a provider will read. The last four lines are the ones worth looking at twice: the run that was cut off at three turns ends, as a request, on a tool result, and the run that met the outage goes out as U U A — the failed reply is on the record and not in the request. Nothing in your loop is watching for these cases. They fall out of where the checks sit and of what the view leaves out.

Say it in your own words

The first lab asked for a script on which a correct loop and a nearly-correct one part company. In a sentence or two: what makes a script like that, and why would "a model that reads a file and answers the question" not have done?

Run your script through the reference loop and through the buggy one in your head. On how many of the two do they do the same thing? Now do the same with an ordinary three-file job.

A test separates two implementations only on an input where they behave differently; on every other input it is a test of nothing, however many asserts it has. The two loops here ask different questions of one reply, so the script has to contain a reply where those questions give different answers: a call in the content under a label that says otherwise, or a label saying toolUse over content that holds no call at all. An ordinary job never produces either, which is why a whole afternoon of ordinary jobs can leave a bug exactly where it was.

Common answers, and what each one misses

"A script with more steps in it, so more of the loop is exercised." Length is not discrimination. Twelve honest turns run identically through both loops, cost more to read and prove nothing; one dishonest reply settles it.
"A script that makes the buggy loop crash." It does not crash. It returns, politely, with a record one message short, which is the normal way for a loop to be wrong. That is why your check looks at what the run left behind rather than at whether anything was raised.
"Any script with a tool call in it — that is the part that differs." Both loops run tool calls, and on a reply labelled toolUse they run them the same way. What differs is what each one asks about the reply, so the script has to make those two questions disagree.

Before you go on

Six diagnoses and two labs, and not one new idea: everything on this page was already in your hands when you arrived. That is what a checkpoint is for. If any of the six took longer than you liked, its reveal names the lesson it came from, and that lesson has not gone anywhere.

Two more things sit beside this page. One is optional and hangs off lesson 6: a side quest about the edit tool, where a one-line fix to a 3,000-line file costs 3,000 lines of output. Nothing later needs it. The other is your progress export, at the foot of every page: it is a plain JSON file, it lives only in this browser, and it is the only copy.

You diagnosed

a skipped call, a silent cut, a request with the conversation left out of it, a jumbled transcript, an outage that poisoned a session for days after the provider came back, and a loop that kept the conversation to itself

You wrote

a fake model of your own: a script on which a correct loop and a nearly-correct one part company, and one assert about what the run left behind

You rebuilt

error_message, context_for_model and run_agent, from an empty gap, against the tests of lessons 3, 4 and 5

Your harness now

exactly what it was at the end of lesson 6; a checkpoint adds nothing to it.

SYSTEM
user_message
text_of
tool_calls
error_message
str_arg
int_arg
tool_specs
run_tool
truncate_head
truncate_tail
make_read_tool
make_write_tool
make_bash_tool
run_agent
context_for_model

Your answers

Still open

Every run on this page printed nothing at all until it was over. Several took three model calls and two tool results before a word came back, and from the outside a job that is working looks exactly like a job that has hung. Lesson 7.

Previous 06 The firehose Next 07 Show your work Your agent takes forty seconds and shows nothing. Where do you put the print?