wisewigdocs
GitHubWebsite

Operational resilience

WiseWig keeps operational metadata in D1 and mutable tenant content in one SQLite Durable Object per tenant. The reference Cloudflare deployment treats backups, recovery, retention, write suspension, deletion, and telemetry as private control-plane capabilities. None are exposed through the public site or public MCP Worker.

Backups and recovery

TenantWorkspaceDO.createBackup exports a versioned, structured tenant archive to an R2 key scoped under tenants/{tenantId}/backups/. Each object carries its SHA-256 digest and tenant ID as metadata. Verification recomputes the digest and rejects a cross-tenant key or malformed archive. Restore imports only into an empty tenant workspace, validates every table and column against a fixed allowlist, and runs in one SQLite transaction.

The daily tenant-workspace cron creates a backup before retention pruning. An owner can create and checksum-verify backups from the management UI. On managed hosting, usage, backup listing/creation/verification, maintenance, and Cloudflare Durable Object point-in-time recovery execute through fixed operations in the selected tenant Worker; the browser-facing Astro Worker has no tenant namespace or R2 backup binding. Every effect is permission-, confirmation-, quota-, idempotency-, and audit-gated. Bookmark restoration aborts the active Object session as required by the storage API. Operators should rehearse both archive restore into a fresh Object and bookmark restore before production launch.

Quotas, retention, and incident controls

  • Editor mutations are limited per site and user. Public forms and MCP tools retain their separate subject/tool rate limits.
  • SQLite writes enforce tenant storage and instance limits; media, submission, plugin, and site counts are included in operational usage.
  • Immutable document/resource versions are retained for 365 days, audits and submissions for 730 days, completed command claims for 30 days, and completed publication operations for 90 days.
  • read-only stops routine mutations but leaves authenticated recovery controls available. suspended removes the tenant from hostname routing.
  • Queue delivery and maintenance outcomes go to Analytics Engine; detailed bounded operational events remain in D1.

Suspension and erasure are platform control-plane effects, not tenant data-plane commands. The managed UI sends them to a fixed dispatcher endpoint with site.manage, exact confirmation, a digest-bound idempotency key, and D1 audit. Suspension immediately removes routing and purges known public cache entries. Erasure convergently upserts a pending deletion request, suspends routing, and leaves physical deletion to the private scheduled lifecycle Worker. The browser-facing Astro Worker never receives direct cross-tenant Durable Object or shared-platform mutation authority.

Tenant erasure

Erasure is deliberately asynchronous. A confirmed owner request records tenant_deletion_requests and suspends routing. The private scheduled Worker then removes the tenant’s media R2 prefix, backup objects, Durable Object rows, published KV snapshots, sessions, domains, roles, sites, and D1 tenant record. It processes only explicit requests, in bounded batches. Failures retain a failed request with a bounded error for retry; the public and editor Workers cannot invoke the deletion implementation.

Do not add a direct hard-delete endpoint. New tenant-owned stores must be added to both archive/retention policy where appropriate and the scheduled erasure workflow. Shared user identities are intentionally retained because one identity can belong to other agency tenants.

Managed tenant Worker lifecycle

Managed Workers for Platforms installations add a script/resource lifecycle around the data workflow. Suspension updates the ready tenant Worker record first, so hostname resolution stops immediately while the script-local Durable Object and R2 bucket remain intact. Upgrade and rollback accept only a trusted release identifier and preserve the previous ready release record until the replacement upload succeeds.

Permanent deletion requires DELETE <tenant-id> and follows this order: mark non-routable deleting; create and validate a content-addressed export receipt; copy that export to operator-controlled recovery storage; erase script-local Durable Object and R2 data; preflight exact provider identities; delete the user script, sole Queue consumer, Queue, and empty R2 bucket; then remove the control-plane mapping. A partial failure becomes a non-routable failed tombstone. Operators retry from the failed step and never substitute a resource name supplied by a browser or tenant Worker.

The lifecycle orchestrator, Cloudflare exact-resource deleter, managed archive copier/restorer, and script-local eraser are implemented. CloudflareTenantArchiveAdapter asks the pinned tenant Worker for a verified backup, enforces a 20 MiB stream limit, recomputes its SHA-256 digest, and copies it into an operator-owned ARCHIVES R2 bucket. Erasure and restore are authorized only after the adapter reloads that exact archive and verifies tenant ID, Worker identity, size, and checksum. Restore streams the verified bytes through a restore operator grant into the tenant-local backup bucket; the SQLite Durable Object validates the archive tenant, fixed table/column allowlist, and empty destination before one-transaction import. A rejected import deletes its temporary tenant-local R2 object so foreign bytes are not retained in the target bucket.

ARCHIVES, the Workers for Platforms dispatch binding, and Cloudflare API credentials belong only in the operator deployer. Never bind operator recovery storage to the request-facing dispatch Worker, public MCP Worker, editor Worker, plugin sandbox, or tenant user Worker. The tenant user Worker receives only its local BACKUPS bucket and an immutable operator dispatch argument. Production hard deletion remains disabled until an archive restore drill and a disposable two-tenant deletion drill succeed in the target account.

@wise-wig/hosting-deployer-worker is the reference operator composition. It has no public route or workers.dev hostname and requires a separate high-entropy service token even across a service binding. It exposes only deploy, schema-validated site initialization, upgrade, rollback, suspend, export, confirmed empty-target restore, and confirmed delete operations. Initialization accepts at most 2 MiB and pins the requested tenant/Worker/namespace/release to the ready D1 record before touching tenant state. Restore accepts the original exportId, sha256, and createdAt receipt plus RESTORE <tenant-id>; the production dispatch namespace comes from operator configuration and cannot be selected by request input.

Release artifacts live at releases/{release}/worker.mjs in the operator-owned TENANT_RELEASES bucket. Before a release can deploy, an independent release process must set custom metadata release={release}, trusted=true, and sha256={digest}. The deployer bounds the module to 10 MiB and recomputes the digest. Uploading bytes alone never makes a release trusted. Keep release signing/promotion credentials separate from the runtime deployer’s Cloudflare token.

Deployment checklist

  1. Configure separate BACKUPS and MEDIA R2 buckets, the OPERATIONS Analytics Engine dataset, the publish queue/DLQ, D1, KV, and private service bindings.
  2. Apply D1 migrations before deploying the tenant-workspace Worker.
  3. Set external alerts on publish-delivery failures, maintenance failures, deletion failures, storage growth, and queue/DLQ depth.
  4. Create a backup, verify its checksum, restore it into a fresh test Object, and record the drill.
  5. Exercise read-only mode and a disposable-tenant erasure before accepting production data.

The adapter is Cloudflare-specific, but the invariant is portable: a host implementation must provide tenant-scoped snapshots, checksum verification, empty-target restore, retention, observable async deletion, mutation throttling, and an authenticated incident-control surface.