deploy-drill-live-account

id: deploy-drill-live-account
kind: platform-limit
measured_on: 2026-08-27
stale_when: >
  a `[[workflows]]` entry becomes valid without a `name`, or wrangler gains any way to interpolate the
  Worker's name into a config value — either would make the Workflow derive like every other resource and
  retire the second edit below; or
  wrangler changes whether `versions upload` shifts traffic, whether it can create a Worker that does not
  exist, or whether auto-provisioned D1/R2 bindings are created before a deploy; Cloudflare makes Workflow
  bindings scriptable rather than account-level, or makes a second script claiming one an error; or the
  default for `preview_urls` changes again; or Cloudflare stops excluding Workers with Durable Objects from
  preview URL generation, which would reopen the alias path this drill found closed
values:
  deploy.versions_upload_shifts_traffic: 0
  deploy.versions_upload_creates_worker: 0
  deploy.migrations_before_first_deploy: 0
  deploy.workflow_name_is_account_level: 1
  deploy.second_node_reassigns_workflow: 1
  deploy.second_node_per_account_supported: 1
  deploy.workflow_name_derives_from_worker: 0
  deploy.workflow_collision_refused_by_plan: 1
  deploy.second_node_first_install_seconds: 108
  deploy.second_node_canary_deploy_seconds: 76
  deploy.canary_override_attempts_named_node: 1

Correction, 10 September 2026: the canary gate was refusing on a propagation race

mailda deploy publishes the canary at 0% and immediately asks it for a report, addressed with Cloudflare-Workers-Version-Overrides. For three consecutive deploys against the live Node it refused:

the override did not reach the canary: 68aac295-… answered.

The refusal was correct. When an override cannot be applied the request is routed by traffic percentage instead, so the incumbent answers and nothing says so, and promoting on that would move every request onto a version nothing examined. But the cause was not what the refusal’s fix suggested.

Cloudflare’s version-overrides page states it:

A version override will only be applied if the specified version is in the current deployment. It can take up to a couple of seconds to be available globally after a recent change.

So the deploy was racing its own publish. Waited out instead, six attempts over roughly fifteen seconds, and the first live run took two retries before the override landed, after which the gate compared 33 findings and promoted.

A gate that has to be overridden by hand every time is not a gate, which is the lesson promotionVerdict was already written from: it replaced a check that refused a canary whose only finding the incumbent already had. This is the same failure arriving through timing rather than through logic. The bound stays because a version that never becomes overridable is a real condition (a Node with no version_metadata binding cannot report its version at all), and the refusal is the honest answer to that.

Measured: a real first install and two deploys into a live Cloudflare account (Mystraits.ai@gmail.com, dc8d1b7d…), 27 August 2026, wrangler 4.118.0. A Mailda Node was deployed, all 39 migrations applied, the delivery-events consumer attached, doctor run against it, a second Node deployed alongside it, and everything the second Node created removed afterwards. The first Node was left running at https://mailda.mystraits-ai.workers.dev.

The drill was run because #92, #98 and #99 all rested on assumptions nobody had tested. It found five facts, three of which were bugs in code written the day before, which is the argument for running it rather than reasoning about it.

The assumption #98’s whole design rests on: confirmed

deploy.versions_upload_shifts_traffic: 0. Across three wrangler versions upload calls, the version serving 100% of traffic never changed. It stayed pinned at the previously deployed version until an explicit wrangler deploy or versions deploy. Checked with wrangler deployments status before and after each upload.

This is what makes “a failed check needs no rollback” true rather than hopeful: the canary is published, checked, and promoted only on passing, and the previous version is still serving the whole time. There is no undo step that has to run correctly during an incident.

`mailda deploy`’s ordering broke the first install

Two separate refusals, both on a fresh account, both fatal to the sequence as it was written:

$ wrangler d1 migrations apply CATALOG --remote
✘ Couldn't find an auto-provisioned D1 DB named 'mailda-catalog' for binding 'CATALOG'.
  Run 'wrangler deploy' to provision it, or add 'database_name' / 'database_id' to your config.

$ wrangler versions upload
✘ You cannot upload a new version of a Worker that does not yet exist. Please run the `deploy` command first.

deploy.migrations_before_first_deploy: 0 and deploy.versions_upload_creates_worker: 0.

The cause is a decision this project made on purpose: wrangler.jsonc declares its D1 and R2 bindings with no ids and no names, because ADR 24 requires the repository byte-identical across installs. So the resources are provisioned by the deploy, and neither of the first two steps of expand-canary-check-shift can come first on a Node that does not exist yet.

mailda deploy now branches. A first install deploys directly, which is safe there for exactly the reason the canary exists: no previous version to protect and no user to serve a broken one to. Every later deploy takes the canary path.

The Workflow collision is real, and it is a silent theft rather than a refusal

deploy.workflow_name_is_account_level: 1, deploy.second_node_reassigns_workflow: 1.

Every other resource derives from the Worker’s name. A second Node called mailda2 got mailda2-catalog, mailda2-evidence and mailda2-sending-events, colliding with nothing. The Workflow does not: its name is written in wrangler.jsonc as mailda-butler-runs, and wrangler workflows list shows a Workflow is owned by exactly one script.

Deploying the second Node succeeded, exit 0, with no warning, and the ownership moved:

	before	after
`mailda-butler-runs` → `Script name`	`mailda`	`mailda2`

So the first Node kept a BUTLER_RUNS binding pointing at a Workflow whose class is now served by the second Node’s code, against the second Node’s bindings. That is a cross-Node execution path into another organization’s D1, the same shape as the queue collision that was already found and fixed, in the one resource whose name is not derived. #99 suspected this; it is now measured, and the failure mode is the worse of the two possibilities: it does not refuse, it reassigns.

The canary has no reachable URL, so #98’s gate cannot run here

Not given a values key, because it is a negative result whose cause is not established and a number would imply otherwise.

wrangler versions upload --preview-alias canary printed a version id and no preview URL. Neither the documented alias form nor the versioned form answered:

canary-mailda.mystraits-ai.workers.dev      404
d711b326-mailda.mystraits-ai.workers.dev    404
mailda.mystraits-ai.workers.dev             200   (the live version, for contrast)

The 404 body is Cloudflare’s generic “Page not found” page, so the hostname is not routed at all rather than reaching a Worker that refused. Declaring "preview_urls": true in wrangler.jsonc and redeploying (the documented fix, and worth doing regardless because Cloudflare’s default for that setting changed three times between September 2025 and October 2025) did not change the result.

What this means operationally: mailda deploy reaches its own could not find the canary's preview URL refusal, which fails safe. It does not promote, it says the canary is uploaded and serving no traffic, and it prints the exact wrangler versions deploy command. So the sequence degrades to “upload, then promote by hand after checking yourself”, which is weaker than designed and not dangerous.

Observed a second and third time on 27 August 2026, deploying migrations 0040 and 0041 with the two search layers to the same Node. Same result, same refusal, and the refusal did what it was written to do: it printed wrangler versions deploy <id>@100, the previous version kept serving 100% of traffic throughout, and the schema change ahead of it was additive so nothing was serving against a schema it did not understand. The sequence degrading to “upload, check by hand, promote” is now a measured property of this account rather than a prediction: weaker than designed, and not dangerous.

Established on 31 August 2026, and it was none of the three things suspected. The account’s own API had the answer the whole time:

GET /accounts/{account}/workers/scripts/mailda/subdomain
  → {"enabled": true, "previews_enabled": true}

Preview URLs were already enabled, the alias was recorded on every version, and the version API carries no preview-URL field. The cause is a documented platform limitation (Cloudflare does not generate preview URLs for Workers that implement a Durable Object), and this Worker declares KEY_VAULT and OUTBOX_SWEEPER, because ADR 28 put both root keys in a Durable Object. No setting on this account could ever have produced the hostname, and the dashboard visit this section twice asked for would have shown the toggle already on.

Two drills recorded “cause unestablished, needs the dashboard” when one read of the API and one line of the Limitations documentation settled it. The lesson worth keeping is not about preview URLs: the suspected cause was checkable without the dashboard, and neither drill checked it.

The gate no longer uses a preview URL. mailda deploy places the canary in the deployment at 0% and reaches it through Cloudflare-Workers-Version-Overrides on the production hostname, requiring the report to name the uploaded version, measured and reasoned in preview-urls-and-durable-objects. The degraded “upload, check by hand, promote” path recorded above is therefore history, not current behaviour.

The fourth drill, 31 August 2026: the sequence works, and the gate was wrong

The first drill since the preview-URL cause was established and the gate rebuilt around a version override. Run against the live Node with CLOUDFLARE_ACCOUNT_ID set, seven pending migrations (0045–0051, all expansion; 0048 swaps an index and recreates it in the same file).

Every step of the sequence ran, in order, and the mechanism works:

preflight                    account resolved, wrangler 4.118.0 above the 4.97 floor
workflow guard               mailda-butler-runs owned by mailda — ran, rather than being skipped
migrations                   0045–0051 applied
reading the serving version  d27a228d  (the last percentage line, parsed correctly)
canary upload                c7e7b917
canary at 0%                 SUCCESS: c7e7b917 at 0% and d27a228d at 100%
override probe               answered version: c7e7b917  ← the identity gate passed

That seventh line is the result the previous three drills could not reach. The version override does reach a 0% version on the production hostname, and the canary named itself, so the identity check, the thing standing between this gate and an assertion that cannot fail, works against a real account. The incumbent still carries no version field, which makes a fall-through unmistakable rather than ambiguous.

Then the gate refused, and the refusal was the defect. The canary reported degraded with one finding, signing_key: “No current signing key. One is generated on the next sign-in, so this self-heals”. The incumbent reported degraded with the same one finding. So a version neither better nor worse than the one already taking every request was withheld, and the operator was told to promote it by hand.

An unclaimed Node is in that state by construction until somebody signs in, so every deploy to one would have gone the same way, which is the weak “upload, check by hand, promote” path the earlier drills recorded, reached from a different direction and for a different reason.

The gate is differential now: a finding the canary has and the incumbent does not blocks; shared findings are reported as carried. refuse still refuses whatever the incumbent says, because two broken versions is a reason to stop rather than to proceed. Re-run against the same account, the gate answers promote: true, carried: ["signing_key"].

Left in a deliberate state: canary at 0%, incumbent serving 100%, schema advanced. Safe by design, since expansion is backward-compatible ahead of the code, and the promotion is an operator’s decision rather than a drill’s.

Completed on the operator’s word. c7e7b917 promoted to 100%. The Node now reports its own version, so every future canary gate can run, migrations_applied reads “All 52 expected tables present”, and the mailda-sending-events consumer was already attached from an earlier deploy, so the step the gate interrupted had nothing left to do. One finding remains: signing_key, self-healing on the next sign-in.

What the gate can actually see, measured rather than assumed. The canary check is unauthenticated, so it reads the reduced report. On this Node that is 9 findings of 21. The other 12 describe the organization’s mail and are withheld from an anonymous caller. The differential comparison therefore covers 9, and a regression confined to a data-disclosing finding would not block a promotion.

Fixable and deliberately not fixed: sessions are signed by the Node’s own key and that state is shared across versions, so signing in and then sending the cookie with the override header would reach the canary authenticated and compare all 21. That needs credentials in the deploy path, which is a decision about what mailda deploy may hold.

A finding-count change that looked alarming and was not. The report went from 20 findings to 9 across the promotion, which looks like checks disappearing. It is the opposite: the old route reduced only if (orgId !== null && !signedIn), so an unclaimed Node served its full report to anonymous callers. The current route reduces that case too. Nothing was removed (four checks were added since), and the tightening is an improvement. Worth recording because the first reading of a shrinking number is that coverage was lost.

Both findings above are closed. mailda deploy no longer inherits doctor’s exit code. A deploy is asked whether it happened, so a carried degradation exits 0 while a post-promotion refuse exits 2 and prints the rollback. And the canary check signs in when credentials are present, so the gate compares the whole report rather than the 9 findings an anonymous caller may see; both sides are asked with the same credentials, because authenticated-canary against anonymous-incumbent would compare 21 to 9 and block every deploy.

The restore drill, 1 September 2026: the escrow is load-bearing, measured

Not given a values key: what it establishes is a property, and the numbers in it describe one drill’s fixtures rather than a bound anybody should hold.

The first restore into a different Cloudflare account. Source: the claimed Node on Mystraits.ai@gmail.com, holding three drafts. Destination: a fresh Node on Admin@arbuilder.app, deployed by mailda deploy as a first install.

backup            52 tables, data only, 29,199 bytes; search index excluded; 3 objects inventoried
deploy            first install into the destination account, exit 0
restore           wrangler d1 execute --file  →  313 rows written
identity          the destination reports claimed: true — organization, users, tuples, drafts all present
objects           3 copied source → destination, each at the size the inventory recorded

And then it refuses.

/api/doctor   503
  refuse    signing_key   Could not use the current signing key:
                          E_EVIDENCE_AUTH_FAILED  frame 0 of 1 failed authentication
/api/auth/login  500

The signing key row came across in the dump, since it is an ordinary row, but it is wrapped under the source Node’s credential KEK, which lives in the KeyVault Durable Object and is not in a D1 export. The destination generated its own KEK at first install, so the unwrap fails AES-GCM authentication.

So a restore that carries every row and every object produces a Node that knows who everybody is and cannot let anybody in, and cannot read a single message. That is exactly what ADR 28 says, “Lose it and every message is permanently unreadable”, and why keyvault.ts said ADR 28 does not ship without ADR 29’s escrow.

The escrow is now measured as load-bearing rather than argued to be. The remaining step is redeeming one of the ten recovery codes against the destination, which installs the source’s keys; it needs a code, which is held by a person by design and is the one part of this drill a tool cannot do.

What the drill taught about the backup itself, each found by running it and each fixed:

wrangler d1 export refuses any database containing an fts5 virtual table, so no Mailda catalog could be exported at all. The command names its tables now.
The export must be data only. A destination is deployed before it is restored into, so its 53 tables already exist and CREATE TABLE fails on the first statement.
d1_migrations must be excluded, not restored. It describes the database rather than the organization, and overwriting the destination’s own rows was what made a restored Node believe its search index existed when it did not.

An unexplained D1 API failure during migration, 28 August 2026

Not given a values key, because it is a negative result whose cause is not established and a number would imply otherwise.

Applying migrations 0042–0044 through mailda deploy printed the three names twice and then:

✘ [ERROR] A request to the Cloudflare API
  (/accounts/…/d1/database/…/query) failed.

with no further detail. Re-running immediately reported “No migrations to apply!”, and the schema was verified correct afterwards: the ledger at 0044_body_index_state.sql, the four new body_index_* columns and msg_body_index_due present on messages, and both new columns on recovery_codes. So the migrations applied and the error arrived after them.

What is not established is which request failed or why. The candidates are the final UPDATE in 0044, some part of the deploy step that follows migration, or a transient API failure with no relationship to either. It did not reproduce, and it cannot now. The migrations are applied and the path is idempotent.

Why this Node cannot answer it. messages is empty here, so 0044’s classifying UPDATE had nothing to touch: it would have succeeded trivially whether or not it ran. The one place the question matters is a Node with mail, and this Node deliberately has none.

What would settle it, and is worth doing before this migration reaches a Node with an archive: apply 0044 to a scratch database seeded with messages in both states, and check the classification actually ran rather than leaving every row on the column default. A migration recorded as applied whose last statement silently did nothing is the shape of failure that shows up months later as “search never found old mail”, and D1 does not wrap a migration file in a transaction, so it is representable rather than theoretical.

Recorded as unknown rather than guessed at, for the reason the preview-URL section above gives: the next person to touch this needs to know the difference between “we measured this and it is broken” and “we measured this and do not know why”.

Addition, 15 September 2026: a second Node in one account, which `wrangler.jsonc` recorded as unmeasured

wrangler.jsonc said of its Workflow block: “a second install into one account gets a different Worker name and the same workflow name. What happens then is unmeasured. The queue case collided silently, and this one is not known to. It is the one thing about this block a second Node in one account should be checked against.”

Checked, with mailda deploy --plan against the live Swmengappdev account. A plan reads and changes nothing, so this cost a lookup rather than a deploy.

`deploy.second_node_per_account_supported: 1`

Renaming the Worker and the Workflow together produces a clean first install, with every other resource derived:

== plan for the Worker `mailda-drill`
   A first install. Nothing here yet, so the deploy runs directly.
       d1       mailda-drill-catalog          absent — the deploy provisions it
       r2       mailda-drill-evidence         absent — the deploy provisions it
       queue    mailda-drill-sending-events   absent — the deploy provisions it
       workflow mailda-drill-butler-runs      absent — the deploy provisions it
   unwind: nothing to remove.

So a second Node in one account is supported. What it costs is two edits rather than one, because the Workflow is the single name that does not derive from the Worker’s.

`deploy.workflow_collision_refused_by_plan: 1`, and it does not collide silently

Renaming only the Worker is the mistake somebody actually makes, since three of the four names derive themselves. It is refused, and the refusal names the owner:

   BLOCKED. What a deploy would do here is not what it looks like it would do.
     ! workflow `mailda-butler-runs`
         PRESENT and owned by another Worker. Deploying takes it — exit 0, no warning — and the
         other Node keeps a binding pointing at a Workflow now running this Node's code
         owner: mailda

This is the half that was unknown. The queue case (#72) collided silently. A second Node’s producer binding attached to the first Node’s queue and nothing looked wrong on either. The Workflow case does not: #99’s guard reads the owner and blocks, and the plan carries the unwind (wrangler workflows delete) because a Workflow survives its script’s deletion.

So the residual wrangler.jsonc stated is smaller than it feared. It is not “a second Node may silently steal the Workflow”; it is “the Workflow name is the one that must be edited by hand, and forgetting is caught.”

The fifth drill, 17 September 2026: a second Node is a name, and the gate reaches it

Two things the third restore drill (disaster-recovery.md) did by hand are done by the tool now, and both were measured on the live Swmengappdev account with a Node named mailda-drill beside the live mailda.

mailda deploy --name mailda-drill. The three places the Worker’s name lives (name, the Workflow’s name, vars.WORKER_NAME) are rewritten into a derived file beside wrangler.jsonc (wrangler.<name>.jsonc, git-ignored, comments kept) and every wrangler call takes --config to it. The checked-in config is not touched. deploy.workflow_name_derives_from_worker stays 0, since Cloudflare still requires the literal on the binding, but the edit is the tool’s, once, and the rule workflow-name-world.test.ts holds is applied rather than remembered. --plan --name reads as the 15 September plan above, unchanged.

step	measured
first install, `--name mailda-drill --url …` (deploy, migrations, consumer, doctor)	108 s
second deploy to the same Node: upload, canary at 0 %, gate, promote, consumer, doctor	76 s
override attempts before the canary answered as itself	1

The canary gate now reaches a Node that is not called mailda. The override header read mailda="<version>" as a literal, whatever the config said. On the third drill’s mailda-drill Node the override therefore never applied, the incumbent answered every time, and the gate refused, correctly, for the reason the 10 September correction gives, so the operator promoted by hand and wrote it down as a propagation race. It was the name. The header takes the Worker’s name from the config now (test/node/deploy-sequence.test.ts pins the literal’s absence), and on this run the first attempt was answered by the uploaded version: no retry, signing_key carried from the incumbent, traffic moved without a hand.

The consumer attach had the same defect from the other end. attach-queue-consumer.mjs already took --name; the deploy never passed it, so the first install above reported “mailda is already the consumer of mailda-sending-events. Nothing to do.”, the live Node’s queue, read for a deploy of the drill Node. Passed now, and re-run by hand for this drill: “Attached mailda-drill as the consumer of mailda-drill-sending-events.” A defect that reads as success is the kind the README’s account-abstraction row exists to stop; it is recorded here because it shipped.

Torn down afterwards, and the order is a finding: wrangler delete refuses a Worker that is a queue’s consumer (code: 10064), so the unwind is consumer off the queue → Worker → D1 → bucket (emptied first: R2 refuses a bucket with objects) → queue → Workflow (which survives its script’s deletion). The live Node was not touched; deployments list --name mailda read the same ten versions before and after.