Skip to content

Engineering & Code

LangGraph Checkpoints Part 3: Time Travel and Recovery

9 min read AI · LLM · LangChain

Part 1 explained what a checkpoint holds. Part 2 put those checkpoints in a file and read the rows.

This part spends them. A saved history is worth having because you can go back into it. Four moves matter:

  • Re-run a step you already ran.
  • Change the past and take a different branch.
  • Answer a paused question differently.
  • Pick up after a node fails, without redoing the work that succeeded.

Versions are the same throughout the series: langgraph 1.2.11, langgraph-checkpoint 4.2.0, and langchain 1.3.16. Most examples here use plain graphs and no model, so the traces are short and every run repeats exactly.

Replay

Start with a two-node graph. Each node prints when it runs, which is the only way to be sure what re-executed:

def pick_device(state: State):
    print("  [node] pick_device runs")
    return {"device": "dfw-core-01"}


def write_plan(state: State):
    print("  [node] write_plan runs")
    return {"plan": f"reboot {state['device']} in the approved window"}

Run it once, then find the checkpoint from just before the second node:

history = list(graph.get_state_history(config))
before_plan = next(s for s in history if s.next == ("write_plan",))

Now invoke with that checkpoint’s config and None as the input:

graph.invoke(None, before_plan.config)
=== first run ===
  [node] pick_device runs
  [node] write_plan runs
result: {'device': 'dfw-core-01', 'plan': 'reboot dfw-core-01 in the approved window'}

=== replay from that checkpoint ===
  [node] write_plan runs
result: {'device': 'dfw-core-01', 'plan': 'reboot dfw-core-01 in the approved window'}

pick_device did not print the second time. Its result was already saved, so LangGraph loaded it and started at write_plan.

Two properties of replay matter in practice. Nodes after the checkpoint really do re-execute, including their model calls and their API requests. Interrupts re-trigger as well. Replay is a re-run of the tail, not a recording of it.

The input is None rather than a state dict. If you pass a state dict instead, the graph adds a fresh input checkpoint rather than a continuation of the saved one.

Fork

update_state writes a new checkpoint rather than editing the one you point at. Change the device and resume:

fork_config = graph.update_state(before_plan.config, {"device": "iad-edge-02"})
graph.invoke(None, fork_config)
=== fork: change the device, then resume ===
  [node] write_plan runs
result: {'device': 'iad-edge-02', 'plan': 'reboot iad-edge-02 in the approved window'}

The original result still exists. Part 1 showed the history as a straight line, and after a replay and a fork it is a tree:

  id=1f1a248e-3d0e-6e1a-8003-d5d26cebebcc
     parent=1f1a248e-3d0d-6e16-8002-eb6877bdd2a4
     step=3 source=loop plan='reboot iad-edge-02 in the approved window'
  id=1f1a248e-3d0d-6e16-8002-eb6877bdd2a4
     parent=1f1a248e-3d09-6c26-8001-10af3b8e509b
     step=2 source=update plan=''
  id=1f1a248e-3d0d-602e-8003-b40efcb5b013
     parent=1f1a248e-3d0c-6476-8002-40e5f2aee446
     step=3 source=loop plan='reboot dfw-core-01 in the approved window'
  id=1f1a248e-3d0c-6476-8002-40e5f2aee446
     parent=1f1a248e-3d09-6c26-8001-10af3b8e509b
     step=2 source=fork plan=''
  id=1f1a248e-3d0a-6874-8002-dca1b7494e07
     parent=1f1a248e-3d09-6c26-8001-10af3b8e509b
     step=2 source=loop plan='reboot dfw-core-01 in the approved window'
  id=1f1a248e-3d09-6c26-8001-10af3b8e509b
     parent=1f1a248e-3d08-6b1e-8000-20c61cf374ff
     step=1 source=loop plan=''

Read the parent column. Three checkpoints share the parent ending in 509b, which is the one before write_plan. That single checkpoint now has three children: the original run, the replay, and the fork.

Note the source values. The original branch says loop, update_state produced update, and the replay produced fork. The prose documentation names only input, loop, and update. The type definition in langgraph.checkpoint.base lists four, and fork means “created as a copy of another checkpoint”. Trust the type.

step numbers repeat across branches, so use checkpoint_id to identify a checkpoint and metadata["step"] only within one branch.

Reducers Apply to Your Edits

LangGraph treats update_state as a node’s output, and that has one consequence worth knowing in advance. Channels with reducers accumulate:

class State(TypedDict):
    device: str
    steps: Annotated[list[str], add]
after the run:    {'device': 'dfw-core-01', 'steps': ['picked', 'planned']}
after the update: {'device': 'iad-edge-02', 'steps': ['picked', 'planned', 'edited']}

device has no reducer, so the update replaced it. steps has add, so the update appended to it. If you meant to replace the list, a reducer stands in your way, and you need a reducer that supports replacement.

as_node Chooses What Runs Next

update_state accepts as_node, which tells the graph which node the update came from. That decides where execution resumes. Take a three-node line. Update at the checkpoint before step_b. Vary only as_node:

  as_node=None       next=('step_b',)    trail=['a', 'edit', 'b', 'c']
  as_node='step_a'   next=('step_b',)    trail=['a', 'edit', 'b', 'c']
  as_node='step_b'   next=('step_c',)    trail=['a', 'edit', 'c']

The default matches step_a, because that is the node that last wrote at this point. If you claim step_b instead, the graph skips step_b entirely, and the trail never gets its b.

Use this to substitute for a node. Supply the output a node would have produced, name that node, and the graph carries on as though it ran.

Interrupts

An interrupt call pauses the graph and saves everything. The graph requires a checkpointer, because the pause is itself a checkpoint:

def ask_approver(state: State):
    answer = interrupt("Approve the reboot? Reply approve or deny.")
    return {"trail": [f"approver:{answer}"]}

The first run stops instead of finishing:

=== first run pauses ===
  next=('ask_approver',)
  interrupt: Approve the reboot? Reply approve or deny.

next names the node that is waiting. Resume by invoking with a Command:

graph.invoke(Command(resume="approve"), config)
=== resume with approve ===
  [node] ask_approver resumed with 'approve'
  [node] apply_change runs
  trail: ['approver:approve', 'applied']

Now combine this with a fork. Replay from before the interrupt, and the graph pauses again with the same question:

=== fork from before the interrupt and answer deny ===
  replay paused again: Approve the reboot? Reply approve or deny.
  [node] ask_approver resumed with 'deny'
  trail: ['approver:deny', 'applied']

That is a human-in-the-loop workflow you can rewind. An approver answered, someone questioned the outcome, and you re-ran the same decision point with a different answer. Both trails are still in the history.

Recovering From a Failed Node

This is the feature that earns a checkpointer in production, and it is the least visible one.

Put two nodes in the same super-step. slow_lookup succeeds. flaky_api fails on its first call and succeeds on its second. A third node joins them afterwards:

builder.add_edge(START, "slow_lookup")
builder.add_edge(START, "flaky_api")
builder.add_edge("slow_lookup", "summarize")
builder.add_edge("flaky_api", "summarize")

The first attempt raises. Look at the state it left behind:

=== first attempt with durability=sync, flaky_api fails ===
  [node] flaky_api runs (call 1)
  [node] slow_lookup runs (call 1)
  raised: the change management API timed out
  state values: {'facts': ['site=TX-ALPHA-3']}
  next: ('flaky_api',)
  task slow_lookup: error=None result={'facts': ['site=TX-ALPHA-3']}
  task flaky_api: error=RuntimeError('the change management API timed out') result=None

Three things are already true after the failure. The saved state holds slow_lookup’s result. The next tuple names only flaky_api, so the successful node is not queued again. The task list records which task failed and what the other one returned.

Resume with None and only the failed work re-runs:

=== resume ===
  [node] flaky_api runs (call 2)
  [node] summarize runs
  result: ['window=Tuesday 02:00', 'site=TX-ALPHA-3', 'summary of 2 facts']

slow_lookup ran 1 time(s)
flaky_api ran 2 time(s)

slow_lookup ran once across both attempts. This is the writes table from Part 2 doing its job. Each node’s output lands there the moment the node finishes, so a sibling’s failure cannot erase it.

That result held on two runs each under all three durability modes.

One caveat is worth your attention, because I hit it before I made the example deterministic. In an earlier version both nodes finished at almost the same moment, and slow_lookup ran twice on the resume. The failure tore down the executor before the successful node’s write was persisted, and the console showed RuntimeError: cannot schedule new futures after shutdown.

Recovery protects work that finished and got written. A node that completes in the same instant as the failure may not clear that bar.

The obvious next question is whether durability="sync" closes that gap. It does not. Here is the same test with the delay removed, so both nodes finish together, run three times per mode:

### durability=sync ###
slow_lookup ran 2 time(s) flaky_api ran 2 time(s)
slow_lookup ran 2 time(s) flaky_api ran 2 time(s)
slow_lookup ran 2 time(s) flaky_api ran 2 time(s)
### durability=async ###
slow_lookup ran 2 time(s) flaky_api ran 2 time(s)
slow_lookup ran 2 time(s) flaky_api ran 2 time(s)
slow_lookup ran 2 time(s) flaky_api ran 2 time(s)
### durability=exit ###
slow_lookup ran 1 time(s) flaky_api ran 2 time(s)
slow_lookup ran 1 time(s) flaky_api ran 2 time(s)
slow_lookup ran 1 time(s) flaky_api ran 2 time(s)

sync lost the write every time, and so did async. Only exit kept it, which reverses the ordering you would expect from the durability names.

I can offer a likely reason rather than a proven one. Under sync and async the write goes to an executor that the raised exception shuts down. Under exit the persist happens when the run exits. An error is one of the ways a run exits, so the write stays on the main path. I did not instrument the runtime to confirm that, so treat it as a reading of the symptom.

The practical rule survives either way. Part 2 recommended sync when you cannot afford to lose a step, and that still holds for a crash between super-steps. It does not protect a sibling that finishes at the same instant as a failure inside one super-step.

Checkpointer Against Store

LangGraph has a second persistence system, and mixing them up costs a day. A checkpointer saves graph state for one thread. A store saves application data across threads.

Compile with both and the difference shows in one run:

graph = builder.compile(checkpointer=InMemorySaver(), store=InMemoryStore())
thread=t1 checkpoints_on_this_thread=3
  devices seen across all threads: ['dfw-core-01']
thread=t2 checkpoints_on_this_thread=3
  devices seen across all threads: ['dfw-core-01', 'iad-edge-02']

Each thread kept its own three checkpoints and saw nothing of the other. The store accumulated across both.

CheckpointerStore
PersistsGraph state snapshotsKey-value data you define
ScopeOne threadAcross threads
Memory typeShort-term, thread-scopedLong-term, cross-thread
Use forConversation continuity, human-in-the-loop, time travel, fault toleranceUser preferences, facts, shared knowledge
Reached byA thread_id in the configstore.put and store.search in a node

The rule is short. If it belongs to this conversation, the checkpointer already has it. If it should outlive the conversation, write it to the store.

The Checklist

Three posts, and it comes down to a handful of decisions.

  • Pass a checkpointer and a thread_id on every call, or nothing here works.
  • Use InMemorySaver for tests. Use SqliteSaver for local work and a database-backed saver in production.
  • Read metadata["source"] and next when you inspect history.
  • Identify a checkpoint by its checkpoint_id rather than by its step number.
  • Set durability="sync" when losing a step is expensive, and exit for a batch job you would restart anyway.
  • Watch storage. Total storage across a thread’s checkpoints grows with the square of the turn count on an accumulating channel. Use DeltaChannel or trim old threads.
  • Turn on EncryptedSerializer for any thread that carries customer data.
  • Remember that a fork does not delete anything. The old branch is still there, which is the point.

Everything above came from scripts on one laptop against langgraph 1.2.11. The counts and the storage figures are deterministic and repeat exactly. Re-check the timings and the Postgres behavior on your own backend.