Skip to content
ClickHouse Loom
Esc
↑↓navigate↵open⌘Jpreview
On this page

Rolling upgrades: builds N-1 and N side by side

Every Loom process is a full replica behind one URL, and an upgrade replaces them one at a time. For the length of a rollout, and again during a rollback, the previous build (N-1) and the new one (N) serve the same clients against the same ClickHouse, on N’s schema. These rules keep that safe. The operator procedure is in upgrades.

Retirement of an N-1 participant also requires its authority-store work to settle. Under Postgres, a client without participant session tags is recertified only after all generic loom authority backends are absent, including idle pools. Stop and drain those legacy clients before the final retirement census; tagged participants can remain live. ClickHouse-authority recovery uses the verified deployment’s ClickHouse checks. Neither path treats an unknown authority as empty. See the participant lifecycle baseline.

The rules

  1. Migrations are additive and run at boot under the migration lock. N-1 keeps reading and writing every table it knew (tests/unit/test_migrations_additive.py). N-1 booting on N’s schema logs one WARNING naming the migrations it does not carry (server/boot.py::warn_newer_schema) and serves.

  2. A version row keeps columns its build does not know. loom.memory, loom.shared, loom.memory_embeddings, loom.session_summaries, loom.entity_summaries and loom.documents are ReplacingMergeTree(version): the newest row replaces the whole previous one. Before each insert into them, ch.insert fills every stored column outside its build’s row builder from the newest stored version of the row’s key (server/core/rmt_carry.py), so N-1’s touch, revision or tombstone keeps the values N wrote to N’s new columns. This costs a cached lookup while every build agrees with the schema, and one extra keyed read per insert while a newer column exists. A moved document’s row is a new key (its new namespace). A move N-1 makes after N’s migration added a loom.documents column fills that column from the new key’s newest row (a document deleted there, or the column’s default), not from the source’s: the copy lacks that value for good, and a restore of the source, whose deleted row keeps it, brings it back there. A move N makes builds the row with every column; its chunks and table rows are copied with SELECT * and keep every column.

    • Each process checks loom._migrations and the carried tables’ metadata time every 2 s in the background and then re-reads the column lists.
    • A write that took the lists before the ALTER would still drop the column. So each process acknowledges the lists it carries by in loom.carry_acks (a fingerprint per carried table), once no write that took older lists is in flight, and a booting process waits, after its migrations and before it serves, until every other live process acknowledges the schema it now has (await_acks, 120 s, or LOOM_CH_TIMEOUT_WRITE_S + 60 s when that is longer, then the boot fails naming them). A process killed without withdrawing can leave a carried INSERT running after its acknowledgement expires, so each carried query id names its process, and the boot also waits while system.processes on any replica shows a carried insert of a process that no longer acknowledges (when it cannot read them, while such a process acknowledged within the longest an insert can run). A process withdraws at shutdown (acknowledges no table) only once no carried write of it is in flight or pinned, and the boot also waits for a withdrawn process whose carried insert still runs. N therefore writes a new column only once no writer that would drop it is left. The acknowledgement gates the writes: a carried write runs only while this process’s last published acknowledgement is younger than 15 s (half the 30 s a boot counts it for). Otherwise (never enrolled, a failed publish, a pause past expiry) the process publishes, re-reads the lists and publishes again before writing, and refuses the write (503, retryable) when any step fails, as it does when it cannot read the lists. The watcher re-publishes every 7.5 s, so a write rarely renews it, and concurrent writes that find it stale wait for one renewal. ClickHouse fences each carried batch on arrival: it is sent as INSERT … SELECT … FROM input(…) whose WHERE throwIf(…) refuses it when, as the query starts, the table has a stored column outside the lists the rows were built against: a request held in transit (a proxy, a retry) past an ALTER lands no row, and the write answers 503 and is retried with fresh lists. A query that started earlier is bounded by its max_execution_time (60 s when the write class has no limit); the guard reads the receiving replica’s metadata. Each batch also carries a start deadline on the server’s clock (send time + 15 s, the clock measured from this process): one that starts later is refused, so a request still in transit can start only until then. A write that ends in an error or a cancellation after a send stays in flight, no newer lists acknowledged, until the server’s clock passes its deadline plus its max_execution_time (when no query of it can run or start); once the deadline has passed, KILL QUERY … SYNC and a count of 0 in system.processes on every replica release it earlier. A held write whose queries cannot be counted shows on /v1/health as carry_write_pinned. The acknowledgements keep old writers from meeting that refusal during a rollout; the guard makes a late one harmless. It costs a carried insert about 5 ms at p50. This fails closed: while loom.carry_acks cannot be written, every carried write (memory.set among them) answers 503.
    • A new key’s row leaves the column out, so ClickHouse computes its DEFAULT.
    • Only state columns may be added to these tables. A column derived from the value goes stale when N-1 revises the value, so one that must exist records what it was computed from (for example the content_sha256 it describes), and N can tell it is stale.
    • A hand-written INSERT … SELECT with a column list is not carried. Only operator commands (loom reembed) write those; run them when one build serves.
  3. New enum or status values: N ships readers that accept them, and only N+1 writes them. N-1’s Pydantic Literal would answer 500 on a value it does not know. A projection failure’s name is a plain string that N-1 already tolerates: its repair worker leaves a name it does not know active as unsupported_projection. So N records a triples repair that failed after its insert as entity_triples_repair, not entity_triples, which N-1 would replay by inserting the triples again. A replay of an entity_triples failure that inserts them and then fails its repair keeps the failure: one mutation records that they landed and moves its entries and results to keys N-1 does not read, so N-1 refuses it (no entries) and leaves it active. N writes a failure’s abandoned status because N-1 reads that status as a plain string and lists only active ones; the status="abandoned" list filter works only on N replicas. The conflict resolver’s owed follow is triples_follow_owed, not triples_follow_rows, for the same reason: N-1’s follow does not check that the loser is still retired. While N-1 leads, its repair re-reads every failure whose name it does not know on every pass (oldest first, each answered unsupported_projection and left active), so they take places in its batch (LOOM_PROJECTION_REPAIR_MAX_FAILURES_PER_PASS); past that many, N-1 replays nothing else until an N replica leads.

  4. Wire formats (cursors, session and run ids, worker resume cursors, JSON stored in rows): N reads the old and the new format, and only N+1 writes the new one. Exception: a cursor for a request N-1 answers for namespace alone (a knowledge.list naming namespaces) is new in N, and each build refuses the one the other wrote (rule 12).

  5. Signing keys: rotate with --activate-in of at least twice the slowest keyring refresh interval (the command refuses less), so no process signs with a kid an N-1 replica cannot verify yet.

  6. Workers: one leader runs them, elected by a ClickHouse lease. Workers stay idempotent under one overlapping in-flight statement, since there are no fencing tokens. An older build leaves the lease to a newer one only when their schema versions differ (each build’s last migration number): a code-only release never hands over, and the old leader keeps the workers until it stops. Either way the order replicas are replaced in does not matter.

  7. Settings: a removed setting warns at boot rather than refusing to start. A new setting’s default accepts what N-1 is configured with.

  8. Retiring a column or table: stop using it in N, drop it in N+1 or later.

  9. Replica-wide state lives in the deployment’s stores, never in a process. Anything a replica must agree on (leases, runs, the conflict resolver’s queue, the similarity calibration, the frontend bundle of every live build) is in a ClickHouse table, so a request can land on any replica of either build. Under the Postgres memory authority (storage authority) Postgres holds the memory heads, the revision journal, the deliveries, the scope generations and the requests’ Idempotency-Key records, and N and later read and write them there (rule 33); the ClickHouse rule holds for everything else. The one exception is the bytes of document originals with LOOM_DOCUMENT_OBJECT_STORE=s3: they live in an S3-compatible bucket that every replica reads the same way, and the catalog row naming each object is in ClickHouse. A build that does not read that row’s location answers such a download 410 and cannot erase such an object, so the store is switched on after the rollout, and no rollback goes past this build while rows name objects.

  10. A task.create key’s exclusion holds only among N’s replicas. task.create with an idempotency key holds a lease on the poster and its key (_one_create_per_key) and inserts under the key’s dedup token (insert_created_task); the coverage watch’s ready-event delivery holds one on the event’s task id (insert_ready_event). One whose lease ran out before its insert can still race the next, so both insert under one dedup token (named by the same poster and key, or the event’s task id) and storage drops the later row. An erased row’s token can outlive it and drop a new row too: with nothing stored, the row goes in again under a fresh token. The build before takes the delivery’s lease and writes its token too, so deliveries exclude across builds. Creates do not: the build before names the create’s lease and token without the poster, so neither build’s create of a key excludes the other’s, and racing they can store two tasks. A build that looks a key up without the poster answers another poster’s task to a create with the same key until it stops serving. Each task is whole (its event and its lease row) and claimable: a create of N cut off after its task row is finished by its retry, which under the key’s lease writes the created event the first one left out, and by the lease sweep, which writes its lease row after its grace. The created event takes a fresh seq, so cursor readers deliver it, unless the task has events since: then it goes just before the first of them and is history only. A first create that was only stalled past its lease, or cancelled while its insert ran on, can still land its own event. Both inserts carry one token, the task’s (insert_event), so storage keeps whichever lands first. Past the dedup window both can be stored: the backfill is marked backfilled in its patch, and N’s readers drop it when the task has an earlier created event, so history shows one creation and a cursor delivers at most one. A reader that passed the first one’s seq before it landed and reads the backfill after misses the creation, as it misses any event landing behind it. A retry that reads an N-1 create’s task between its row and its event logs such a backfill too. N-1’s readers do not drop it; that ends once N-1 stops serving.

  11. A link N adds is not yet complete. N links each value to the triples and edges it stated (entity_triple_rows, entity_edge_rows); N-1 links only keys, just after it writes the row. So a value with no row link still holds by its key link when that link is no older than the value and no row of the key links the fact, and a close N runs keeps what an N-1 row states. A key that rows of both builds link to one fact reads as N’s: there an N-1 row can lose that fact until it is stated again.

  12. A request field N adds is ignored by N-1 until N-1 stops serving: it answers the call as if the field were not sent, naming it in ignored_args. So memory.restore’s preconditions if_superseded_by and only_if_superseded (the dashboard’s Undo sends them) do not hold on an N-1 replica, which restores a deleted row or one retired behind another memory, and the relations of memory.set, memory.update and memory.update_by_id are dropped there, so that write graphs none of the facts it names. memory.get_by_id’s omit_high_value does not hold there either: N-1 answers a high row with its value and hash, and audits that read as a sensitive one. N’s conflict resolver does not read the memory.unsupersede record of an Undo made on an N-1 replica, or before the upgrade: it carries neither the restored hash nor the restore version. The restored row then reads as a statement made at the Undo, newer than the memory it was retired behind, so a confident update verdict can retire that memory although the person kept both. Pinning it prevents that. N-1’s knowledge.list rows carry no namespace, on any list; one naming namespaces lists namespace alone there, and each build refuses the cursor the other wrote for such a list, so a client paging it starts again. memory.set’s if_match_sha256='absent' uses an existing, nonempty precondition field. A build without this meaning refuses it as a hash mismatch, including on a missing key, instead of storing an unconditional write. First creation may therefore answer 409 while such a build serves; clients must not retry without the condition. Ordinary content hashes still work across builds, and rollback needs no data change.

  13. A name N spells differently is a different node. N keeps combining marks and NFKC-normalises an entity’s name (दिल्ली, İstanbul as istanbul); N-1 split them into fragments (द ल ल, i stanbul). ASCII names spell the same in both, and so do Latin ones but for compatibility characters, which N folds and N-1 keeps (the ligatures fi and ij, full-width letters, superscripts, fractions, ª, º, ſ). For the other scripts the two builds write separate nodes until N-1 stops serving, and a node N-1 wrote keeps its fragment name until a write states that entity again.

  14. An erasure N-1 runs misses the tables N adds. N-1’s saga and its census do not know entity_triple_rows and entity_edge_rows, so an erasure it runs during the rollout leaves their rows in its scope and still records the saga completed. Once N-1 stops serving, re-verify the erasures completed since the rollout began: loom erasure-re-verify --completed-since <rollout start> --resume deletes, in each one’s scope, the rows of those two tables whose memory has no live row and no row written since the saga’s census (orphan row links). It runs no saga step again and touches nothing else: data written in the scope after the erasure finished, its links included, stays, tombstoned by memory.delete or not.

  15. A marker N writes in a column N-1 reads is not one N-1 reads as its own. The conflict resolver’s queue lives on memory_embeddings.conflict_checked_at, where an N-1 with the per-write judge on reads the epoch itself as a judgement it owes and re-runs it. N writes 00:00:01 (owed) and 00:00:02 (judged), which N-1’s statements read as settled (they match the epoch exactly); N treats N-1’s epoch marks as owed once N-1’s own settle had a minute. N-1’s Python is wider: its is_mark takes any time before 2000 as its pending mark, so where N-1 runs its per-write judge (LOOM_MEMORY_CONFLICT_JUDGE, off by default) its settle of a row it wrote can stamp its settle time over N’s mark on that row; N then reads an older build’s verdict, owed again only while the row is conflicted, and that one row may go unjudged. While N-1 holds the worker lease (a code-only release keeps it until it stops) N’s marks wait, and the first N leader judges them. A pin is the other way round: N’s memory.pin holds the key’s write lock, N-1’s does not, so during the rollout an N-1 pin racing an N retirement can leave the pinned row’s vector superseded until the row’s next write (or loom reconcile-status).

  16. A triple row N-1 writes arrives at its whole second. N-1 does not set entity_triples.seen_at (migration 067), so its rows take the column’s default, their last_seen second (.000000). Two values of a single-valued relation starting the same day and arriving in the same second, one from each build, then order the N-1 row first, whichever arrived later; the next write of either fact orders them again.

  17. N-1 does not know N’s retirement intents. A user erasure N-1 runs deletes no projection failure (N-1 erases them by namespace only), so a point-write retirement intent naming that user in retired[] stays until N erases the user again, or the intent is resolved and its TTL ends it. A projection repair on N-1 leaves every projection kind N adds (rule 3), triples_follow_owed, entity_edges_drop_undeclared, entity_edges_restore and sensitivity_facts_owed among them, active as unsupported_projection; N’s repair replays them once N-1 stops serving.

  18. A new revocation key (such as SCIM’s SCIM_PREFIX): N-1 does not read it, so a token it revokes still works on an N-1 replica. The feature that writes it is turned on only once no N-1 replica serves: SCIM provisioning (LOOM_SCIM_TOKEN) after the whole rollout.

  19. A new access rule (such as collections, sharing): N-1 does not know that a col- namespace opens only through grants, so an org admin’s wildcard token reads one on an N-1 replica, and a personal collection is one too. An end-user token is refused there (N-1 has no key for its kid), but on N its first @me write creates a personal collection. N-1 also rewrites a document (a metadata edit, a delete) without the provenance columns, so a user erasure would miss that version; it registers a col- name that N refuses (the boot warning lists the first 20); and it resumes a user purge without purging (it reads no purge column). Collections are created (collection.create, which N-1 does not serve) and end-user issuers registered (LOOM_JWT_END_USER_ISSUERS_JSON) only once no N-1 replica serves.

  20. Saved conversation messages: older builds can compact concurrently saved turns and rebuild embeddings without carrying their saved state. Clients use conversation.save_message only once every serving replica runs the build that preserves saved messages. A rollback pauses saved-message writes and compaction until every replica supports that state again.

  21. An extraction policy (extraction_policy.set): N-1 does not read loom.extraction_policies and extracts without it. Set a policy only once no N-1 replica serves. A rollback leaves the rows in place but no replica applies them until the build that reads them is back.

  22. A new side effect of a write (such as memory webhook events): N-1 does not perform it. A memory write served by an N-1 replica leaves no loom.memory_webhook_outbox row, so it sends no event, and an erasure saga run by N-1 has no step for the outbox and its census does not count it. Webhooks are registered (webhook.register) only once no N-1 replica serves, so no outbox row exists while an N-1 replica can complete an erasure. A rollback to N-1 after webhooks exist leaves the outbox rows written before it: N-1 never reads or erases them, and they leave the disk within about 4 days (the outbox is read for 3 days and drops a day’s part once its last row is 3 days old). Rolling forward again, N’s worker sends none of an erased user’s events, because it skips an event whose key has no history left in loom.memory_versions, which every build’s erasure deletes.

  23. Assistant-turn learnings (memory_extract_agent_learnings): N-1 does not erase loom.learnings_supporters, where a user’s quotes behind a bot’s rule live, and on learnings.review it promotes an assistant_turn candidate as the approver’s own memory. Turn the setting on, and review assistant_turn candidates, only once no N-1 replica serves. Before a rollback, turn it off and delete the rows of loom.learnings_supporters and the assistant_turn rows of loom.learnings_queue (extraction). N-1 has no per-agent filter on knowledge reads, so after a rollback it serves a promoted bot rule to every agent in the namespace. To keep a rule to its bot, delete the documents tagged source-kind:assistant_turn with knowledge.delete before the rollback; otherwise accept the wider serving.

  24. A namespace guide (namespace.guide_set): N-1 does not serve the guide ops, shows no guide in namespace.browse, namespace.get or the session-start pack, and a namespace erasure it runs has no step for loom.namespace_guides. So namespace.guide_set answers 503 OVERLOADED while any live replica’s schema version (loom.instances) is below 72, or no heartbeat has been read: no guide is written while an N-1 replica could erase its namespace. A rollback to N-1 after guides exist leaves them in place, and a namespace erasure N-1 runs then leaves them too; run that erasure again once N serves. tests/unit/test_namespace_guide.py checks the refusal; the mixed-version gate does not. The gate counts the deployment’s processes (loom.instances of its kind): an older process outside it on the same ClickHouse, such as a branch server, is not counted.

  25. Platform-admin bots (loom.machine_identities.platform_admin, migration 076_machine_platform_admin.sql): the column’s UInt8 DEFAULT 0 reads 0 for every bot made before it. N-1 refuses an admin bot’s exchange but accepts a flat admin JWT issued by N, and its SSO account routes lack the human-only check. Both OAuth admin exchanges and loom mint-token --platform-admin therefore refuse until a fresh, nonempty fleet view shows every replica at migration 76 or later, including processes outside the deployment on the same ClickHouse. Neither path uses the kind-filtered replica cache. OAuth answers 503 temporarily_unavailable; the CLI refreshes the fleet before minting and prints no token when it is unreadable or contains an older replica. Register and use such bots only after the rollout. Before starting any N-1 replica for a rollback, disable admin bots and revoke every standalone CLI-minted admin subject or token. N-1 writes the column back to 0 when it disables or replaces a bot.

  26. A bot’s longest token (loom.machine_identities.max_access_token_s, migration 077_machine_access_token_ttl.sql): the column’s UInt32 DEFAULT 3600 reads an hour for every bot made before it. N-1 ignores a request’s expires_in and issues an hour, its /v1/admin/bots.create refuses the field (422), and when it saves a bot (a disable, a replace, a namespace release that drops one of its roles) it writes 3600 back. This can lengthen a token requested or capped below an hour, or shorten one allowed longer. So create_bot (the admin route and loom bot create) refuses a maximum other than 3600 until a fresh fleet view shows every replica, in every deployment, at migration 77 or later; an unread or empty view (no server registered yet) refuses too. A request’s expires_in needs no gate: N-1 answers it with an hour. After a rollback, reapply each maximum once N serves again (loom bot create --replace with the bot’s key or trust and roles as registered, which also ends its tokens). The mixed-version gate does not check this rule.

  27. An erasure saga’s audit actor (loom.erasure_requests.actor, migration 081_erasure_request_actor.sql): N-1 inserts ledger rows without the column, so a saga it checkpoints reads '' afterwards, and a later resume by N names the resuming process (erasure-recovery-worker or the CLI command) on its failed and completed rows instead of the principal that asked. N-1 also still writes the body’s requester as the actor of the erasure.* rows of a request or export it serves, or of a saga it resumes. No gate holds the rollout: the cost is the actor named on those audit rows, nothing is lost or refused, and it ends once no N-1 replica serves. The mixed-version gate does not check this rule.

  28. A new document-source kind (recording or notion in loom.knowledge_sources.kind): N-1 refuses to add one (BAD_KIND) and lists and inspects one like any source. Its check and rotate of one fail with a server error, since it has no probe for the kind. An N-1 sync worker that takes such a source’s turn reads it as a shared folder: the credential is not a service account key, so the turn ends credential_invalid before any request and retires nothing, and the next turn by N writes the real status. An N-1 mining step keys a document by its revision alone, while N keys a notion document by its revision and text hash, so a hand-off between the builds mines a notion document again. No gate holds the rollout: the cost is that status, shown until N’s turn, and those extra mining calls, and it ends once no N-1 replica serves.

  29. Typed datasets (loom.datasets, loom.dataset_rows, migration 083_knowledge_datasets.sql): N-1 ingests a file without its datasets and serves no dataset op. A dataset is served only while the source’s live document row has the version it was published with, so a write or a delete N-1 makes of the source stops serving N’s datasets as N’s own would; N-1’s delete or ingest of the source leaves their rows, which the source’s next ingest or delete by N removes (a delete of a source already deleted included). A metadata edit N-1 makes of an ingested source carries nothing, unlike N’s: its datasets stop being served, and the next ingest infers their types again, an earlier approval dropped; ingest the file again after the rollout to serve them, and approve again. An erasure N-1 runs does not know the two tables, so the rows N wrote in its scope stay and the saga still records completed: once N-1 stops serving, request the erasures completed since the rollout began again, which N’s saga runs through every table. No gate holds the rollout.

  30. Extracted decisions in the model’s words (loom decisions extract, the document-source sync’s mining step): N records a decision’s reasons in the model’s words and retires a row only when its support check finds it gone twice; N-1 retires the extractor’s row whose reasoning is not a span of the document. So an N-1 run on a document N mined (an operator’s N-1 command, an N-1 replica’s sync turn on a changed document, every changed document after a rollback) deletes the extractor’s unpinned rows N wrote and tags its pinned, attested or acted-on ones source-changed, and writes its own. The rows are the extractor’s own and the next run by N writes them again from the document; a source-changed tag stays until a run by N emits the key and finds it stated. No gate holds the rollout: the cost is that churn while N-1 serves, and after a rollback until the document is mined by N.

  31. A new shared-folder file kind (docx in a gdrive source’s include): N accepts the kind and writes it only when a caller names it; a config without include, and the dashboard form’s default, take the four kinds N-1 knows. N-1 has no docx in its file-kind table, so it refuses to add a config that names it (BAD_CONFIG), its check and rotate of such a source answer a server error, and its sync worker fails such a source’s turn before any request: the source’s state shows status failed, error KeyError, nothing is retired, and the next turn by N syncs it. No gate holds the rollout: the cost falls only on sources whose config names docx, and it ends once no N-1 replica serves. After a rollback it stays for those sources, every file kind of them unsynced, until each is removed; a second source that names only docx beside one that names the other kinds (the way document sources add a kind) confines that to the .docx files. N+1 may take docx by default. This departs from rule 3 the way rule 28 does for a new source kind: holding add back a release would hold back every folder of .docx files, while the cost falls on the sources a maintainer opts in, ends with the rollout, and loses nothing. The mixed-version gate does not check this rule.

  32. A native document’s pictures out of its export (the gdrive sync’s docs conversion): N drops each pasted picture’s base64 reference definition from a native document’s Markdown export; N-1 stores the export as it comes. A document N-1 writes is found by N on its next full listing, whose stored-revision read checks each stored text for such a definition, and converted again at the same revision. N writes every native document with a text_sha256 in its metadata, so the mining step of either build, which keys a document carrying one by its revision and that hash, mines the converted text once; a document N-1 writes carries none and is keyed by its revision alone, which is new whenever N-1 writes; N-1 rewrites the document only when its file changes. No gate holds the rollout: the cost is the base64 a document N-1 writes holds until N’s next full listing, and the check, which reads each document’s stored text on every full listing, stays while a build that stores raw exports may serve. The mixed-version gate does not check this rule.

  33. The memory authority (loom.storage_authority, migration 085_storage_authority.sql; schemas/pg_migrations/): N records the deployment’s authority at the first boot that runs its migrations (LOOM_BOOT_MIGRATE) and refuses to serve against another. A build before N does not read the record and writes loom.memory directly, so once a Postgres deployment is recorded every replica must run N or later, and it never rolls back past N. Postgres migrations follow rule 1: additive, applied at boot under an advisory lock, and a build at or after N serves on a later build’s Postgres schema with a warning naming the files it does not carry (Postgres migrations). The mixed-version gate does not check this rule.

  34. A user purge deletes the memory rows last (memory.retire): N-1 has no memory.retire, and its loom erasure-re-verify --resume runs a user saga as a tombstone, so a user saga N started and N-1 resumes can keep the answers that quoted a summary, and a purge can end with the user’s rows or the summaries built from them only tombstoned; resume user sagas N started from N. A saga’s mutation tag names its step on N, so a step resumed across builds submits its mutation again; the statements are idempotent. No gate holds the rollout, and the mixed-version gate does not check this rule.

  35. One token mint (service_accounts.create, caller_home.minted_home, token_saves_to, me_service_accounts.dev_caller, admin_routes.admin_token_mint).

    • Where a token saves. N sends a delegated token’s unaddressed calls to the namespace its claim names before * when it may write there; N-1 sends them to the owner’s saved pick, as for a bare * token. A token of that shape (one from service_accounts.create with a namespace before *, or an admin/token.mint token whose namespaces start with one) writes to either namespace while both builds serve, depending on the replica; both are namespaces it may write, and the split ends with the rollout. After a rollback such a token follows the pick for the rest of its life.
    • Where a list token saves (saves_to claim and loom.service_accounts.saves_to, migration 093_service_account_saves_to.sql). N records the list token’s target in the token and its row, and sends its unaddressed calls there, then down its own list, never to the owner’s saved pick. N-1 ignores the claim: on an N-1 replica the token follows the saved pick, as every list token did before, until the rollout ends. A list mint N-1 serves drops saves_to (both mint routes ignore an unknown field) and mints a token without the claim, which follows the saved pick for its life; service_accounts.create answers where it saves under that rule, and admin/token.mint answers no saves_to. A * token’s home stays the name before *, and one that may no longer write there still falls back through the saved pick. On N a delegated token, with the claim or without, never saves to a collection unasked (one it may only read can still be its default for reads, caller_home._pick_default): a list token whose only writable entry is a collection saves there by default on N-1 and nowhere by default on N, so its calls name where they save. N-1 revoking a row writes saves_to empty: a revoked token saves nowhere.
    • Mints refused by the other build, before anything is saved. N-1 refuses * in a list, namespaces: "*" or none at all (create_token(), loom token create without --namespace) and a mint without a name (the dashboard’s form with its Name left empty, loom token create without --name). N has no /v1/me/setup/mcp_config, which N-1’s dashboard calls. N refuses saves_to beside * and a collection before *, which N-1 mints (dropping saves_to). A retry once the rollout ends works.
    • A mint N-1 accepts. A named list mint is saved by N-1, which answers without saves_to, mcp_url, reads_everything, roles and clients; N’s dashboard fills those from the page (its own /mcp, the claim) and shows the token. A list token N-1 mints gets no bound tracker record. N-1’s dashboard posting a list mint to N sends no saves_to, so N records the first writable memory entry, kept for the token’s life; N’s answer names it, and the created dialog shows it.
    • The list. N-1 matches a token’s maker by the exact subject, so a person signed in under another spelling of theirs neither sees nor revokes there a token made under the first, and N-1’s dashboard lists a * token N minted by the names in its claim (it reads every namespace all the same).
    • Who owns an auth-off token. With auth off and no bearer, N saves a token with the single user you as its owner and N-1 with dev. Each build’s Tokens page shows the tokens of its own owner (and, as for any owner, those on a namespace it administers); every token stays valid. On N, a token N-1 saved is listed and revoked by naming its owner: subject=dev and the token’s org (default for one made on the Tokens page) on GET /v1/me/service_accounts and POST /v1/me/service_accounts.revoke.
  36. The memory a cached code.ask answer quoted (loom.code_answers.memory_namespace and memory_keys, migration 086_code_answer_memory_scope.sql): N-1 writes an answer without them, and a tombstone N-1 writes of an entry N stamped (a stale entry, a negative code.outcome) drops them, so its text stays where no erasure finds it. N-1’s saga, census and memory.redact do not know the table, so an erasure or redact N-1 runs leaves the answers N stamped in its scope and still records completed. A saga N-1 left in_progress runs code_answers.delete when N resumes it, since the step is not in its steps_completed; one N-1 completed is not run again, and loom erasure-re-verify does not reach this table. So an erasure N runs before the cache is cleared can leave text in the cache. N also changes the answer policy in the cache key, and each build reads an entry of another prompt version as dead: it serves it to nobody and tombstones it on sight. While both serve, a question asked of each in turn is synthesized again at every switch, and N-1’s tombstones of N’s entries, which carry no stamp, are the main way the stamp is lost. Where a memory namespace is bound, clear the cache once N serves alone (upgrade procedure, step 9); after a rollback, clear it once N-1 serves alone and again once N serves again. No gate holds the rollout.

  37. What a summary of the user’s memories fed (its history, chunks, entity vectors, graph rows and shared copies; erasure saga): N-1’s user saga and census reach only the user’s own, so a user erasure N-1 runs leaves the summary’s rows and records completed. Outside a purge, a saga N-1 left in progress that N resumes fails N’s census when N-1 ran those rows’ steps, and the recovery worker’s next resume runs them again; the shared copies, which the user-scope census does not count, stay when N-1 passed memory.retire. In a purge, N’s memory.retire runs them again, except for the search rows of shared copies of the user’s own memories when N-1 ran its shared.tombstone, which deleted the rows they are found by. After a purge that passed memory.retire, including one N-1 completed, the user’s keys are gone and nothing finds the summaries’ rows. Every build before N leaves them, and the ledger does not name the build that ran a saga: once N serves alone, request again each user erasure without purge that was requested before then (the user_id rows of loom.erasure_requests), and after a rollback, those requested since the rollback once N serves again (upgrade procedure, step 10, which names what the new saga also erases); no gate holds the rollout.

  38. Knowledge document heads (loom.knowledge_heads, Postgres migration 002_knowledge_documents.sql): under the Postgres authority N commits each document write as its source’s head; N-1 writes loom.documents alone. N reads both and serves the newer version, and a commit over a head older than the row its writer read replaces it, so N serves N-1’s writes and a rollback loses none. Two writes of one source are refused one against the other only when both run N. N-1’s erasure has no pg_knowledge.delete step and leaves the scope’s heads. Its tombstone stamps the rows at their version, and at a tie N serves ClickHouse’s copy, so the document reads as deleted; after a purge the first N read finds no ClickHouse row at the head’s version and nothing owed, and deletes the head instead of serving it. A delivery still owed or parked when N-1 erases its scope stays, and N’s delivery worker publishes it again; an erased document’s text stays in its head. Once N-1 stops serving, run again with N every erasure that completed while an N-1 replica served since N first served, a rollback’s included, before an operator retries a parked delivery (upgrade procedure, step 11). N refuses to boot with LOOM_RETENTION_DOCUMENTS_DAYS set under this authority (unset it first, step 2) and puts no namespace retention TTL on loom.documents or loom.chunks; while an N-1 replica leads the retention_reconcile pass it applies a namespace’s retention_days to both again and N’s next pass removes it, so a document whose rows age out in that window is gone from ClickHouse, and the first N read of it drops its head as a purged one. N-1’s originals sweep and user erasure read loom.documents alone, so while N-1 runs them they may delete an original only an unpublished N commit names (any owed or parked one for a user erasure; for the sweep, one whose original is older than LOOM_DOCUMENT_BLOB_MIN_AGE_SECONDS): that document’s download then answers 410 once its row lands, until the file is uploaded again. N-1’s restore does not wait for a delivery still owed, or for one the worker settled whose replaced request may still be sending: one that restores a source whose N tombstone landed in ClickHouse before its delivery finished (an insert that timed out and landed, a failed completion) has its chunks retired again when N’s worker publishes that delivery or the replaced request’s retirement lands, and the document serves no search hits until it is written again. No gate holds the rollout, and the mixed-version gate does not check this rule.

  39. The chunks a knowledge head serves (chunks_revision, chunks_version on loom.knowledge_heads, Postgres migration 004_knowledge_chunk_heads.sql): under the Postgres authority N’s search, outline, synthesis and memory.stack knowledge reads serve only a head’s chunks (a source with no head serves what ClickHouse holds) (knowledge search); N-1 serves every chunk ClickHouse reads as live, as before, so while it serves, a search it answers can show an older version still being retired or a write not committed. N-1’s commits leave both columns as they were, so after one that numbers a new revision chunks_revision is no longer the head’s and N judges that source’s chunks by revision alone, which also serves a refused writer’s chunks of that revision until the source’s next N write that inserts chunks (a delete, a metadata edit or a restore keeps the columns as they are). A head N-1 created holds none. N-1’s knowledge.add and bulk_add again keep the earlier version’s chunks, which N does not serve and retires at the source’s next N write that inserts chunks. No gate holds the rollout, and the mixed-version gate does not check this rule.

  40. Knowledge ingestion jobs (loom.knowledge_jobs, Postgres migration 005_knowledge_jobs.sql): under the Postgres authority N keeps each job in Postgres and projects each committed version into loom.knowledge_ingestions, best effort, so N-1 may read a job a version behind until N’s next save, or until N’s worker next takes it, and may run again a job N ended whose last projection did not land (a published one N-1 finds published); N-1’s originals sweep and user erasure read open jobs from ClickHouse alone, so while N-1 runs them they may delete the original of an N job whose projection has not landed and whose upload has expired, and that job then fails original_gone; N-1 keeps its jobs in ClickHouse alone. N’s status and cancel read the newer version of the two stores, and its worker’s queue interleaves its own due jobs with ClickHouse’s, the two leading in turn from one pass to the next. A job N-1 queued runs in ClickHouse alone, as N-1 runs it; a run of one Postgres holds first takes the version N-1 saved when it is the newer, and waits while an earlier open job of its source is held in Postgres or only in ClickHouse, so a source’s jobs still run one at a time, oldest first. N thus runs, reports and cancels the jobs N-1 queued or ran. A cancel N accepts is also written to loom.knowledge_ingest_cancels, which N-1 reads, and N reads the cancels N-1 writes there. The deployment-wide job lease alone keeps a run of each build from overlapping the other’s: N-1 does not raise a job’s lease_token, so the token does not fence an N run whose lease lapsed against an N-1 taker. N-1’s erasure has no step for loom.knowledge_jobs and leaves the scope’s jobs there, and N’s worker projects an open one back into loom.knowledge_ingestions and runs it; once N-1 stops serving, run such an erasure again with N, as for rule 38 (upgrade procedure, step 12). N breaks a tie between two jobs of a source created in one millisecond by the id’s text, N-1 by its UUID, so the two builds may each run one of such a pair at once. No gate holds the rollout, and the mixed-version gate does not check this rule.

  41. The erasure receipt (loom.erasure_requests.census, migration 091_erasure_census.sql): N writes every count of a complete census with the verify columns of a saga’s terminal and failed rows, and the recovery worker’s abandonment checkpoint carries it forward; a resume writes in-progress rows without it until its own census lands. N-1 neither writes nor reads it, so a row N-1 checkpointed reads '', which N’s erasure.status answers as no receipt (census: null). N also writes the purge a saga applies to the purge column (false for a single namespace), which N-1 reads as before, since N-1 never purged a single namespace either. No gate holds the rollout, and the mixed-version gate does not check this rule.

  42. The knowledge dataset catalog (loom.knowledge_datasets, Postgres migration 007_knowledge_datasets.sql): under the Postgres authority N commits each catalog row there and projects it into loom.datasets, best effort, so N-1 may read a catalog row a version behind until N’s next write of it; N-1 writes loom.datasets alone. Every read of the catalog by N (datasets_repo.at_version: knowledge.dataset_get, knowledge.query, an approval’s reads, a publish’s and a metadata edit’s carry, and a publish’s check that no approval landed) takes Postgres’s rows and those only ClickHouse holds, the newer version of a dataset winning, so N serves and queries what N-1 published or approved. N’s approval commits in the transaction that holds the source’s head, over the version of the catalog row it read: Postgres’s, or a newer one N-1 wrote to ClickHouse, which the approval then commits to Postgres. N-1’s approval writes ClickHouse alone, and N reads it as the newer version once the replica it reads has caught up: until then N serves Postgres’s older row, whose replaced snapshot N-1 may already have deleted, so the dataset reads empty, and an approval by N in that window outranks N-1’s. Both builds take the source’s claim for an approval and a publish, so an approval and a publish of one source never overlap, whichever builds run them. N-1’s deletes (knowledge.delete, a later publish’s retirement) and its erasure remove loom.datasets rows alone and leave N’s catalog rows in Postgres until that source is next published or deleted by N: ask such a delete again with N, which removes them; once N-1 stops serving, run such an erasure again with N (upgrade procedure, step 12). No gate holds the rollout, and the mixed-version gate does not check this rule.

  43. A revision’s delivery time (delivered_at on loom.memory_revisions, Postgres migration 010_revision_delivered_at.sql): under the Postgres authority N stamps a revision it delivers more than 15 minutes after its commit, and the payload clearing pass takes an unstamped one as delivered within 15 minutes of committed_at (payload clearing). N-1 completes a late delivery without the stamp, so a revision it delivered late counts as delivered earlier than it was, by up to that delivery’s lateness, and stays so. Turn LOOM_PG_PAYLOAD_CLEAR_ENABLED on no sooner than one clearing horizon after N serves alone: every revision N-1 delivered is then past the horizon from its real delivery. A rollback to N-1 starts that wait again: turn the switch off before it, and on again no sooner than one clearing horizon after N serves alone after the roll forward. The switch is off by default, and N-1 runs no clearing pass. N-1’s delivery cannot send a revision whose payload was cleared. Once a clearing pass has run, loom reconcile runs only with a build that carries 010, and that build is the one the deployment starts on after it; a rollback waits until N owes no claimable delivery for a cleared revision, and while N-1 serves a retry of parked deliveries leaves the cleared revisions out (rollback). No gate holds the rollout, and the mixed-version gate does not check this rule.

  44. Document sources on any namespace (knowledge.source.* without the trunk check): N-1 still refuses add, check, rotate and sync_now on a namespace that is not a repository’s trunk (NOT_A_CODE_NAMESPACE), so while it serves, the dashboard’s Add (the list the Library’s Sources button opens offers it on every shared namespace), Check, Sync now and Rotate of such a source fail on the replicas it answers from, and its sync worker skips such a source until N leads the workers. N-1’s namespace.delete, collection.delete and erasure do not retire a namespace’s sources. A source whose namespace N-1 deleted stays active with its credential, and the sync skips it (its namespace is inactive); one whose namespace N-1 erased keeps syncing, and its next full listing writes the namespace’s documents back. An erasure N-1 started and N finished runs knowledge_sources.retire late, and its final census reopens the steps for what the source wrote back meanwhile; loom erasure-re-verify (rule 14) runs no step. N-1’s add does not read the ledger, so a source it adds during an erasure N runs, after N’s knowledge_sources.retire, stays active and syncs after the erasure completes. Once N serves alone, the upgrade procedure, step 13, lists what N-1 left, with its queries and cleanup. No gate holds the rollout: the cost is those refusals and that cleanup, and it ends once no N-1 replica serves. The mixed-version gate does not check this rule.

  45. A namespace’s promoted rows (shared.tombstone in the namespace saga): N-1’s namespace erasure has no step for loom.shared and its census does not count the table, so a namespace or namespace_list erasure N-1 runs completes with the namespace’s promoted rows live, and so did every one an older build completed. Their search rows are gone (memory_embeddings.tombstone), but memory.lineage and the shared reads still return them. Once N serves alone, the upgrade procedure, step 14, lists each namespace an erasure in the ledger (kept 90 days) reached and runs over it what N’s step runs: rows promoted before the erasure are tombstoned, or deleted after a purge. An older erasure’s rows cannot be found and stay. No gate holds the rollout, and the mixed-version gate does not check this rule.

  46. An /assistant history entry (the dashboard’s expanded assistant window has no address): an N-1 tab pushed /assistant entries, and after the ReloadBar’s reload on N, Back or Forward onto one reaches N in the same document. N answers it: the address becomes the landing page’s (landingTarget for the tab’s namespace; the view the tab shows when that is Setup), the page arrives there, and the assistant opens over it (expanded from 1024 px; nothing opens on Setup, on a namespace the tab cannot read, or at phone widths). The server keeps serving the assistant page segment for bookmarks, which N answers the same way at boot (except that on a phone the sheet opens over the landing page), so this answer stays as long as the segment is served. The mixed-version gate does not check this rule; frontend/e2e/assistant-launch.spec.ts (“an entry an earlier build pushed opens the assistant on Back after a reload”) does.

  47. A cell’s org triggers and loom.tenancy guard keep their names and shape (cells). The session marker loom.tenancy_checked, the functions loom.tenancy_refuse_unheld() and loom.tenancy_refuse_unmarked(), the trigger names tenancy_insert, tenancy_update and tenancy_record, and their shape (AFTER, row level, on their events) do not change, since builds N-1 and N create and read them on one cell. Each function’s body changes only through a numbered migration (CREATE OR REPLACE FUNCTION), and missing accepts the body and settings any shipped file writes. A changed body ships in two releases: release K adds the new body to missing as an accepted value beside the shipped files’, and the migration that writes it ships no earlier than K+1, so an N-1 loom doctor or attach on a migrated cell names no defect. The WHEN text lives in the code that creates the triggers (server/core/cell_triggers.py), so a changed WHEN ships in three releases, in the body’s order: release K adds the new WHEN to missing (which loom doctor and attach read) as an accepted value beside the one it creates; K+1 creates the new WHEN and keeps the old one accepted, so a K pod that creates a trigger on a skipped table, and a K missing reading one K+1 created, both name no defect; and the migration that drops the triggers, which the same boot then creates with the new WHEN, ships no earlier than K+2, so the N-1 that could create a skipped table’s trigger already carries the new text. The gate on processes with LOOM_CELL unset, the marker and the triggers ship in one release: a release that carried the triggers without the marker would refuse Loom’s own writes on a cell, and one with the gate alone would leave its unchecked writers unguarded. It breaks side-by-side serving in one place: on a later cell already prepared, the release’s first migrating boot creates the triggers, and an older pod still serving there has its inserts of a tenant org’s rows refused, so such a cell is rolled out in one step (upgrades, step 2). The mixed-version gate does not check this rule.

  48. Knowledge gaps (loom.search_misses and loom.knowledge_gaps, migration 95): N-1’s erasure saga has no step for either table. So while any live replica’s schema version is below 95, or no heartbeat has been read, no miss is recorded, the knowledge_gaps worker writes nothing and knowledge.gaps.list answers 503, as namespace.guide_set does under rule 24. The operator turns LOOM_KNOWLEDGE_GAPS_ENABLED on only once every replica runs N. Before a rollback, turn it off and delete both tables’ rows, then delete again after the first N-1 replica serves (knowledge gaps). tests/unit/test_knowledge_gaps.py checks the refusals; the mixed-version gate does not.

  49. A document source’s items (loom.knowledge_sources.source_kind, items_json, secret_enc, migration 096_knowledge_source_items.sql): N writes a source whose items the id-list config can hold (only non-recursive folders, at most 20 of anything) as N-1 writes it, kind, config_json and the credential in credential_enc, with items_json beside them, so N-1 lists, checks, rotates and syncs it as before. Any other source is stored with kind empty, its kind in source_kind, its credential in secret_enc (purpose knowledge_sources.secret), credential_enc empty and config_json holding no item. N-1 lists such a source with kind ''; its check answers credential_invalid (its credential_enc is empty) and asks for a rotate, and its rotate fails with a server error before writing, since it has no probe for kind ''; and its sync turn reads it as a shared folder with no credential, so the turn ends credential_invalid before any request and retires nothing. N-1’s remove of it, and its retirement when its namespace goes, write an inactive row whose new columns read empty. So knowledge.source.add and knowledge.source.update answer 503 OVERLOADED (retry_after_s 30) for a source they would store with kind empty while any live process of any deployment on the ClickHouse service (loom.instances, read_live(all_deployments=True)) reports a schema version below 96, or none has been read. N reads source_kind first and kind when it is empty, and items_json first and config_json when it holds no item, so a row N-1 rewrote reads as it wrote it. N-1’s sync_now without full writes full: false over an update’s waiting full request, so while it serves, an added item’s files older than the stamp wait for the periodic full listing. N-1’s loom secrets reseal does not reseal secret_enc, and its --check reports clean, so run the reseal from build N before dropping LOOM_JWT_SECRET_PREVIOUS: a secret_enc left under the old key reads as no credential and its source’s turns end credential_invalid until rotated. A later release, once no build below 96 runs, writes kind for every source and keeps one credential column. tests/unit/test_knowledge_sources.py checks the refusal, and tests/unit/test_knowledge_source_sync.py that a folder turn over such a row ends before any request; N-1’s check and rotate failing is read from its code, and the mixed-version gate does not check this rule.

  50. A source’s document bound (LOOM_KNOWLEDGE_SOURCE_MAX_DOCUMENTS): N reads a source’s stored copies in pages and holds back new files past the setting, for every kind. N-1 reads at most 50,001 of them: its gdrive turn holds back new files past 50,000 and ends failed with too_many_documents over it, retiring nothing; its linear, notion and recording turns check no bound, so past the cut a stored copy reads as new and is fetched again each turn, and the cut read retires less. Both rewrite and retire under the same source lease. While N-1 serves, a setting at or below 50,000 is safe; raise it only once every replica runs N. The bound tests in tests/unit/test_knowledge_source_sync.py, test_knowledge_source_sync_linear.py, test_notion_source_sync.py and test_knowledge_source_recordings.py check N’s side; N-1’s is read from its code, and the mixed-version gate does not check this rule.

  51. Mirror names are reserved (migration 097_mirror_names_reserved.sql): N’s gate refuses every request that names a namespace of the form acl- and 24 hex digits (only a platform admin’s erasure.request may name one), with auth on or off. namespace.register and namespace.restore refuse the form, and N’s org-wide scopes, the sql_v1_ views and the analytics export leave it out. N-1 admits the form as any other name. The migration refuses to run while any namespace of the form is registered or holds live rows in ClickHouse (the registry, memory, documents, conversations, threads, sources and code repositories), naming up to ten as org/namespace with their tables and counting the rest. Since N refuses every request naming such a namespace, the operator clears it with N-1: an erasure of what it holds, code.repo.remove for a repository and namespace.delete. The build that writes mirrors comes after this one, so the build beside it already refuses the form; the one gap is a name an N-1 process registers or writes after the migration ran. The mirror writer draws a new name again while the one it drew is in use (server/knowledge/app/mirrors.py, rule 54), so no such namespace is ever taken for a mirror. tests/unit/test_mirror_names_refused.py checks N’s side, and the mixed-version gate does not check this rule.

  52. The mirror registry (migration 098_knowledge_mirrors.sql): N adds loom.knowledge_mirrors and the access column of loom.knowledge_sources, whose default (namespace) is what every row N-1 writes means. N stores a source whose access is mirrored with kind empty, so a build without items finds no probe or credential for it. N serves none either: its sync pass skips one (rule 54’s build syncs a mirrored gdrive one into the mirror), its retire pass leaves its delete request alone, and check, rotate, update, sync_now and an add of a removed one’s name refuse with SOURCE_NOT_SERVED. So beside the build that writes mirrors, N never syncs a mirrored source into the named namespace or orphans its documents.

    On N, each namespace erasure step reads the registry, which keeps every mirror the named namespaces have had, so every run of the step, a resume’s or a replay’s included, covers them: a scope that holds a mirror is erased as a purge, in one pass over the named namespaces and their mirrors, and the census counts it the same way. The last Loom step, knowledge_mirrors.forget, writes the mirrors deleted, and a live one keeps the name in use. N-1 runs no mirror pass and counts no registry row; N writes no mirror, so N-1 meets an empty registry, and the build that writes mirrors (rule 54) runs beside N, which reads them. tests/unit/test_knowledge_mirrors.py and the mirrored cases in tests/unit/test_knowledge_sources.py and tests/unit/test_knowledge_source_sync.py check N’s side.

  53. A document’s permission list (migration 099_document_access_lists.sql): N adds loom.documents.access_json, the permission list of a mirrored document (document sources); under the Postgres authority it rides in the head’s row, with no Postgres migration. No writer in N passes a list, and the row builder carries none forward: every row N builds, a tombstone included, has the column empty, so it reads unread and admits nobody; an erasure purges a mirror’s rows outright (rule 52). N-1 never meets a list: N writes none, and the first build that writes one runs beside N, never beside N-1, which matters because N-1 would carry a list through its rewrites (rule 2). No op’s input or output has the field. tests/unit/test_access_list.py and tests/unit/test_document_row_builder.py check N’s side.

  54. Mirrored folder sources sync (migration 100_knowledge_source_listings.sql): N’s sync worker runs a mirrored gdrive source, while its ops still refuse one (SOURCE_NOT_SERVED, rule 52). Each turn lists every file with its permissions, reading permissions.list for a file the listing leaves them out of, and writes into the namespace’s mirror (made under the namespace’s sources lease when it has none) at <address>#<source id>, tagged source-id:<id>. Once the turn starts, it brings every stored copy’s list to the store’s (only narrowing a copy behind its file’s revision), or to unread for a copy a whole listing does not have at its address (bounded as rule 55 says); then it fetches and writes each new or changed file with its list, and at the full interval retires a copy whose file is gone, with no audit row, since the audit log outlives an erasure. The listing’s start goes to loom.knowledge_source_listings once every stored copy’s list is written. No request can add a mirrored source yet, so N meets none it did not write, and N runs every mirrored gdrive source as one shared service account: the build that adds sources of another credential mode stores them in a form mirror_synced rejects while N may run. By rule 52, N-1 serves no mirrored source, and it reads no listing row; an N-1 namespace erasure has no listings step, so it leaves the mirror’s listing rows (source names and times, no memory content), which the next erasure of the namespace on N removes, since its scope holds every mirror the namespace has had. N counts a document as searchable when its chunks are those of its version or of its revision, which a row-only rewrite (a new list, a metadata edit) keeps, so such a rewrite stays until the file changes; N-1 checks the version alone, so after a metadata edit it fetches and writes the file again, as it does today. tests/unit/test_knowledge_source_sync_mirror.py and tests/unit/test_knowledge_source_revisions_sql.py check N’s side.

  55. The mirrored sources’ share of a pass: N runs the mirrored sources first, until a third of the time the pass starts work in has passed (a quarter of knowledge_source_sync_max_pass_seconds) and under their own document bound (LOOM_KNOWLEDGE_SOURCE_MIRROR_DOCUMENTS_PER_PASS), the one synced longest ago first, a sync request included; the namespace sources then have the rest of the pass under their own bound. A mirrored source’s turn keeps its own state under its mirror and source id, leaving only its request, status and time in the namespace row. Its model calls are attributed to the mirror. It reads the lists on from a cursor in that state, in at most half of what is left of the mirrors’ share when its turn reaches them, so its new and changed files still get time, and a turn whose list reads stop there is still complete, so it retires gone files and keeps the full interval; a cycle’s listing row carries the start of the turn that began it. N-1 runs a mirrored source under the whole pass’s bounds and keeps its whole state in the namespace row; N reads neither N-1’s fields there nor N-1 the mirror row, so a mixed fleet reads some lists again and retries some waits, and the next N turn drops N-1’s fields from the namespace row. No request can add a mirrored source yet. tests/unit/test_knowledge_source_mirror_pass.py checks N’s side.

  56. Mirror erasure receipts (migrations 101_erasure_mirror_mark.sql and Postgres 013_erasure_mirror_mark.sql): N adds a Boolean mirror_marked, false for ordinary receipts, to both erasure ledgers. Before a namespace step or its failure census reaches mirrors, N persists true. Request replies and namespace audit rows then carry codes only; status and list expose counts only to a freshly authorized flat platform admin. Recovery, reverify resume and restore replay preserve the mark; cross-store copies promote it without replacing terminal state. A replay promotion changes no receipt or pending fence. Marked versions reserve bit 63 of the UInt64 version, so even a later unmarked checkpoint cannot replace one; timestamps order marked writes. N-1 ignores the column and cannot advance a marked saga. Mirrored sources remain refused until every process implements this protection and the later read and session protections. Before N-1 may serve after enablement, the enablement and rollback procedure must settle outstanding marked sagas, remove mirrored content and scrub marked receipts and audit counts while preserving their mark and version priority; a schema downgrade alone provides no privacy protection.

  57. Stopped mirrored sources retain their retirement requests: N writes a knowledge_source_retire:<source id> key under the mirror in code_index_state before inactivating the registration. Its always-on knowledge_source_retire worker deletes that source id’s owned copies under its own cursor, deadline and document bound, even after the name is reused. N-1 ignores those queue keys and cannot serve the mirrored registration; its ordinary cleanup skips it. Queue rows remain until N finishes, and namespace erasure removes them with the mirror. Mirrored registration remains unavailable. tests/unit/test_knowledge_source_retirements.py checks name replacement, bounded progress and cursor persistence.

  58. Mirrored work keeps diagnostics inside its namespace: N suppresses shared logs, spans, metrics and work figures while syncing or retiring mirrored content. Its worker summaries carry fixed codes and no elapsed time or document figures; billing rows stay in the mirror namespace. N-1 has no equivalent boundary, so mirrored source creation remains refused until every process implements the read, session and enablement protections. Before N-1 can serve after enablement, stop writers, settle outstanding inserts and sagas, scrub marked receipts and audit figures, and erase every mirror store, including agent_events, agent_events_private and agent_cost_rollup_daily, before clearing the enablement latch. Stopping a client after its timeout does not prove a sent insert has completed.

  59. Personal-agent reader claims and verified directory evidence (migration 103_sso_domain_evidence.sql): N adds empty-default hosted_domain and issuer String columns to sso_links. Direct verified sign-in writes them only for a fresh active passwordless account under its ownership lock, after the SCIM check. N-1 ignores them and its ordinary link writes clear them to defaults; missing evidence denies domain access. No backfill infers a domain from an email. N accepts the optional Boolean reads_as_creator claim, absent meaning false. Only the primary first-party sign-in route POST /v1/me/personal_tokens.create and the assistant’s one-answer bridge set it; ordinary delegation omits it. Saved personal tokens use the existing String kind column with value personal_agent, which N-1 lists and preserves during revocation. N-1 ignores the claim and has no mirrored document leg, so it grants no mirrored access. Mirrored sources remain unavailable until every process implements the document and session protections. tests/unit/test_mirror_reader_identity.py checks the claim and delegation boundaries.

  60. Checked mirror reads use source-scoped group evidence (migration 104_knowledge_group_members.sql): N reads each membership snapshot by org, mirror, source id and group, and admits only a nondeleted snapshot within LOOM_KNOWLEDGE_SOURCE_ACL_STALE_SECONDS. An empty table admits no group. N-1 ignores the table; namespace erasure removes its rows with the mirror. Mirrored source creation remains refused until every process implements the read, marked request, session and enablement protections.

  61. Coarse permission keys and private lexical statistics (migration 105_mirror_access_candidates.sql): documents.access_keys is materialized from the existing permission JSON; N-1 writers need no new field. N still checks the strict current list, and Postgres heads do not depend on published keys. Each chunk-frequency view changes its query in place, retaining the input, target and output columns. Both views exclude mirror namespaces for either build. Mirror creation remains refused until the read, request, session and deployment protections are complete.

  62. User cost caps and private budget warnings (migration 107_user_cost_budgets.sql): activate the global and namespace user caps and alert inboxes only after every serving build supports the migration; keep them disabled and do not rely on user caps while any build is older or unknown. N-1 cannot enforce user caps, filters no private warning events, and has no user-erasure step for those events. Before the first N-1 process starts, the operator stops all warning producers for maintenance, settles scheduled and uncertain writes and their writer leases, deletes only budget.warning task events, and verifies completed deletion and zero warning rows on every storage replica. There is no interval in which N-1 serves with private warnings stored. Save policy before rollback: N-1 namespace writes omit the user-cap and alert-inbox columns, so their defaults replace the saved values. Restore and verify the global default and namespace policies only after every serving build supports the migration again. The activation and rollback procedure defines the maintenance boundary; the warning fleet check alone does not make a downgrade safe.

What checks them

scripts/gates/mixed_version.py (make test-mixed-version, the mixed-version CI job, once with a Keeper and once without) boots the base build on an empty ClickHouse, then this build, which applies its migrations, then the base again on the newer schema, and a third process of this build with a changed dashboard bundle and one no-op migration more (N+1). It checks:

  • each build answers ready after the cross-boot;
  • memories, a run begun on one build and completed on the other, and a conflict verdict through the other build’s version write read back whole across builds;
  • a column added as N+1’s migration would survives a version write by each build, made as soon as every process has acknowledged the column (rule 2);
  • a build on a ledger newer than itself logs the WARNING;
  • every asset each dashboard page names loads from a replica of the other build (rule 9);
  • with every process on LOOM_BACKGROUND_WORKERS=auto, exactly one process runs the workers, and it is N+1: N, which took the lease first, hands it over to the higher schema version (rule 6).

It does not check new enum values (rule 3), wire formats (rule 4), signing keys other than default (rule 5), new revocation keys (rule 18), an /assistant history entry (rule 46), or a process killed mid-migration. It skips with exit 0 where docker is unreachable; CI passes --require, which fails there instead.

Analyst event storage cutover

The CHECK proof requires ClickHouse with system.constraints, as supplied by the pinned deployment release. Upgrade the database before provisioning the event catalog; a server without that metadata refuses the cutover.

agent_events retains ordinary event rows, its columns and the analyst’s signed, namespace-scoped grant. Both builds can insert and query ordinary events while the entity catalog advances to version 5. The ordinary_events_only CHECK rejects marked rows and reserved mirror namespaces. Ordinary namespaces matching the reserved mirror spelling are refused too; they cannot be used as an alternate private sink.

Marked requests and mirror workers write agent_events_private, which has no analyst grant. The typed, signed analysis_agent_events view unions both stores and masks private fields before predicates run. Trusted cost, budget and history-policy reads also cover both stores. A row policy on a shared physical table cannot replace this split: table metadata could disclose private rows even when a SELECT returns none.

Before granting access, provisioning verifies the exact CHECK and both table UUIDs on every configured writer, freezes the identities of all running INSERTs, waits for that finite set to finish, and counts forbidden ordinary rows. Internal async flushes participate even after their submitting client disconnects. Queued requests that have not prepared a pipeline encounter the installed CHECK when they execute. An ALTER alone does not settle old pipelines. Missing writers, changed topology, unknown metadata, a deadline or any surviving forbidden row refuses provisioning; existing rows remain untouched. Writers continue accepting valid ordinary inserts.

Negative observations use a checked administrative HTTP session on each writer. The configured service origin is included; LOOM_CH_ANALYSIS_OBSERVER_URLS adds explicit trusted HTTP origins using the same existing service credentials. Every writer UUID needs exactly one route. Missing routes, aliases to the same backend, redirects, backend changes and incomplete replies refuse. Routes must preserve non-replaying POST semantics, including any intermediary. HTTPS cannot downgrade. The proof never discovers another credential or derives HTTP ports from native cluster addresses.

Each writer supplies its complete native SHOW CREATE ROW POLICIES catalog. A SELECT from system.row_policies cannot establish visibility because policies can filter that catalog too. Policies on process, topology or CHECK metadata sources, database-wide policies and unknown event policies refuse. Generated analyst event policies require their matching unrestricted policy for other users: an analyst-only policy can implicitly hide every row from another user. The observer cannot be the analyst user or hold its enabled role. Catalog, backend, principal and table identities remain fixed through the drain and census. The complete proof has a ten-second deadline; exact owned sessions close concurrently with one bounded cleanup interval.

Run loom ch-setup --apply-analyst-sql with explicitly configured existing access-management credentials to apply only the analyst boundary. It uses the same live proof and lease without rebuilding the reader user. --print-analyst-sql refuses with a nonzero result and prints no grants or secrets: an unchecked SQL file cannot enforce the native policy-catalog observation. General --print-sql still prints database and reader preparation with a password placeholder.

Mirrored creation remains disabled until the fleet, read and session protections and enablement procedure are complete. Before an older build serves after enablement, the rollback procedure must settle writes and erase every mirror store, including agent_events_private; restoring a grant or downgrading the schema is insufficient.

Analyst probes across catalog versions

The analyst probe checks its signed key canary before reading the stored entity version. When a newer build owns the entities, the older probe leaves that build’s catalog and grant checks to it. It does not query retired tables or overwrite the newer entities. A failed key canary still refuses analysis, regardless of the version stamp.

Deploy this probe ordering across the fleet before a later catalog change revokes a raw relation. Builds with the earlier probe order can refuse insights.query after that cutover because they probe the retired relation before checking the entity version. Statements naming a removed relation still fail its grant check; trusting the newer catalog does not restore a retired grant.

Connected-account registrations

The account registry is additive and is read only by builds that understand connected sources. Such a source stores an empty legacy kind and credential, with its provider kind in source_kind; an older worker cannot interpret it as a shared credential. Its owner_subject and connection_id bind every credential checkout and publication to the canonical person’s current account generation. The source id also distinguishes its mirror copies from another registration using the same item address.

Keep connected and mirrored creation disabled until the fleet’s mirror read and session protections are complete. A rollback or restore must settle source writers and perform the mirror cleanup described in document sources before an older build serves. Account rows restored without their matching Keeper fence refuse credential checkout. That refusal does not prove physical erasure: restored provider identities, sealed credentials and source history still need canonical-owner cleanup and its census. Namespace or end-user erasure cannot substitute for that exact owner key. loom reconcile --apply ends connection authority before publication for either restored store. On a ClickHouse-only deployment, stop all serving processes, restore the configured stores, then run python dev/restore_connected_accounts.py first without flags and then with --apply. It refuses recent worker activity and never remaps a scratch restore target into the deployment client. Both paths fence all historical connection keys and verify physical owner-history deletion. Complete the separate mirror-content cleanup before serving; this account procedure does not replace it.

Was this page helpful?