10 · Sync protocol — Rasd Sync Protocol (RSP) v1, @rasd/sync and @rasd/server

Purpose: normative wire specification of RSP v1 (every route in 00 §8), the client sync-engine algorithm in @rasd/sync, and the design of the reference server @rasd/server, so that a host team can (a) drop the engine into a PWA/RN app, (b) implement RSP on their own backend, or (c) run the reference server. Audience: engineers building @rasd/sync / @rasd/server; backend developers at UN/NGO organisations implementing RSP on an existing stack; integrators mapping RSP onto ODK Central / KoboToolbox.

TL;DR

• RSP v1 is plain HTTPS + JSON under /v1, with tus 1.0.0 for attachments and an optional SSE channel. Auth is a host bearer token; every request carries X-Rasd-Device and X-Rasd-Client; every write carries Idempotency-Key. Replays are always safe.
• Submissions are append-only: POST /v1/submissions:batch (≤ 50 items, ≤ 5 MiB) returns a per-item verdict — accepted (with serverRev), duplicate, rejected ({code,message,field?}), conflict. HTTP 200 for the envelope even when items fail; 4xx only for envelope errors.
• Attachments are content-addressed (SHA-256), uploaded independently of their submission via tus (creation + HEAD + PATCH, 5 MiB chunks, checksum extension, 7-day expiry); a submission is complete on the server only when every referenced hash exists.
• Forms and datasets are pulled as deltas with opaque cursors and tombstones; published definitions are immutable, hash-checked and optionally JWS-signed. Unknown versions are quarantined, never dropped.
• Records/cases (optional) use per-field last-writer-wins by HLC with a server rev; true same-field collisions come back as conflict with both versions and land in a supervisor queue.
• The client engine is foreground-driven, single-leader (Web Locks), single-flight, priority-ordered (submissions → attachments → forms → datasets → records), with full-jitter exponential backoff 1 s → 5 min, per-error-class retry rules, dead-lettering, and lease-based resumption after an app kill. Background Sync / expo-background-task are accelerators only (research/04, research/05).
• Security: TLS mandatory, token refresh mid-sync without losing offsets, device_revoked halts network but never data, remote wipe via X-Rasd-Wipe = crypto-shred after an optional final sync. Synced data is purged from the device after a configurable window (default 7 days).
• @rasd/server = Hono + Postgres 15+ + S3-compatible tus store, multi-tenant by org_id, webhooks, Kobo/Central-style exports; sized for 10 k devices and 1 M submissions per tenant.

---

1. Scope, principles and wire conventions

1.1 What RSP is (and is not)

RSP is the device ⇄ host-backend contract. It carries five collections: form definitions (pull), datasets (pull), submissions (push), attachments (push), records (bidirectional, optional), plus device policy and events. It deliberately does not define end-user identity (delegated to the host's getAuthToken()), form authoring/publishing (an admin concern, §2.10), analytics, or user management. Anything a device needs to work offline for weeks is in RSP; anything a dashboard needs is not.

Design principles inherited from the spine and research/04 §4: outbox + idempotency for submissions (no merge), immutable versioned definitions, tus for bytes, LWW+HLC only for records, foreground-driven client, and “never lose data”: every failure path ends in retry, reject-with-reasons-and-keep-local, or server-side quarantine.

1.2 Base URL, versioning, content types

• Base URL is host-configured (createSyncEngine({ baseUrl })); all paths below are relative to it. Path prefix /v1 is the protocol major; within v1 the server may only add fields/endpoints. Clients ignore unknown response fields.
• Content-Type: application/json; charset=utf-8 for JSON; tus routes use tus content types; SSE uses text/event-stream.
• Time on the wire is ISO-8601 UTC (2026-08-15T10:01:02.123Z); IDs are UUID v7 strings; sizes are bytes; checksums are "sha256:<64 hex>".
• The client refuses http: base URLs except localhost, 127.0.0.1, 10.0.2.2 and *.local (dev) — see §7.

1.3 Request and response headers

Header	Direction	Required	Semantics
Authorization: Bearer <token>	req	yes (all routes except GET /v1/ping)	Token from host getAuthToken(). Server-side AuthAdapter (§9.4) resolves it to {orgId, userId, roles}.
X-Rasd-Device: <deviceId>	req	yes	UUID v7 generated once per install, stored in kv. Server binds it to the org on first POST /v1/devices. Mismatch between token org and device org ⇒ 403 device_mismatch.
X-Rasd-Client: <lib>/<version>	req	yes	e.g. @rasd/sync/1.4.0 (web; chrome/128); used for compat decisions and metrics only.
Accept-Rasd: 1.0	req	recommended	Highest RFD spec version the client understands (research/12 §7). Server omits definitions it cannot downgrade and lists them in unsupported[].
Idempotency-Key: <sha256 hex>	req	yes on POST …:batch	Batch hash (§2.6). Same key + same body ⇒ replayed response (Idempotency-Replayed: true); same key + different body ⇒ 422 idempotency_mismatch. Keys are retained ≥ 24 h.
Accept-Language	req	optional	Localises server messages in error bodies.
X-Rasd-Request-Id	resp	always	Server request id, echoed into the client sync log.
X-Rasd-Definition-Signature: <compact JWS>	resp	optional	Detached signature over a published RFD (§2.4). Distinct from the webhook HMAC header X-Rasd-Signature (§9.5, 20 §7), which is a server→downstream header and never appears on a device route.
X-Rasd-License: <RLT>	resp	optional	Fresh license token piggy-backed on any 2xx (00 §9); forwarded to @rasd/license.
X-Rasd-Wipe: <nonce>	resp	optional	Remote-wipe order (§7.4). Only honoured on an authenticated 2xx over TLS.
Retry-After	resp	on 429/503	Seconds; the client clamps to 3600 and overrides its backoff.
RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset	resp	recommended	IETF RateLimit header fields (§1.6).
ETag / If-None-Match	both	on GET /v1/forms/{id}/versions/{v}	ETag = "<definitionHash>"; 304 on match.
Date	resp	always	Used by @rasd/license monotonic clock guard and by the HLC (§5).

1.4 Error body

Every non-2xx response and every per-item rejection uses one shape:

{
  "error": {
    "code": "form_version_unknown",        // wire code, snake_case (table below)
    "message": "Form pdm-gfd-2026 has no published version 4",
    "details": { "formId": "pdm-gfd-2026", "formVersion": "4" },
    "field": "hh_size",                     // only for data-level rejections (element name / path)
    "retryable": false,
    "requestId": "01J5…"
  }
}

Wire codes are lower-case strings on the wire only; the client raises a RasdError whose code comes from the frozen RASD_* set (00 §12, consolidated in 17 §16) and keeps the wire code in details.reason:

HTTP	Wire code	Meaning	Client RasdError.code / action
400	bad_request, payload_too_large, batch_too_large	Malformed JSON, > 5 MiB, > 50 items	RASD_SYNC_REJECTED; split batch on batch_too_large/payload_too_large
401	unauthorized, token_expired	Bad/expired bearer	RASD_SYNC_AUTH; refresh token once, then pause (§4.6)
403	forbidden, device_mismatch, org_suspended	Policy	RASD_SYNC_AUTH; the affected phase is skipped
403	device_revoked	Device disabled by an admin	RASD_SYNC_REVOKED ⇒ engine revoked state (§7.3)
404	form_unknown, dataset_unknown, record_unknown, upload_unknown	Resource missing	item-level rejected (RASD_SYNC_REJECTED) or full resync of that collection
409	record_conflict, submission_conflict	Same id, different content	item conflict + RASD_SYNC_CONFLICT (§2.6, §2.8)
410	cursor_expired, upload_expired	Cursor older than retention / tus expiry	full resync of collection / recreate upload
412	checksum_mismatch, definition_hash_mismatch	Integrity	item rejected (client recomputes; if still mismatching ⇒ dead-letter §4.7); on an attachment ⇒ RASD_ATTACHMENT_FAILED
413	chunk_too_large	tus PATCH larger than Tus-Max-Size policy	halve chunk size
415	unsupported_media_type	Wrong content type	client bug ⇒ dead-letter (RASD_SYNC_REJECTED)
422	data_invalid, idempotency_mismatch, spec_unsupported	Validation	item rejected with field; spec_unsupported ⇒ RASD_UNSUPPORTED_SPEC
429	rate_limited	Quota	RASD_SYNC_NETWORK; honour Retry-After
460	tus_checksum_mismatch	tus chunk checksum failed	retry chunk from HEAD offset (§2.7)
5xx	internal, storage_unavailable	Server	RASD_SYNC_NETWORK (retryable); retry with backoff

Transport-level failures (DNS, TLS, timeout, fetch rejection) never reach this table; they surface as RASD_SYNC_NETWORK with details: { url }.

1.5 Cursors and pagination

Delta endpoints (/v1/forms, /v1/datasets/{name}, /v1/records) take ?since=<cursor> and return { items, cursor, hasMore }. A cursor is an opaque, URL-safe string (reference server: base64url of "<seq>:<lastId>" where seq is a commit-ordered BIGINT — one global BIGSERIAL in the reference server, a per-tenant sequence as an optimisation; §9.2 and Open questions). Rules:

• Omit since (or send since=0) for a full sync. Cursors are collection-specific; never reuse across collections.
• Page size: server chooses (limit query param is a hint; forms default 100 / max 500, dataset rows default 1 000 / max 5 000, records default 200 / max 1 000). Client loops while hasMore and persists the cursor only after applying the page in one storage transaction (storage.transaction(['forms','datasets','kv'], …)).
• Ordering is by commit sequence, so a row updated twice appears once with its latest state; deletes appear as tombstones ({ id, deleted: true, deletedAt }) for at least 90 days, after which the cursor may be reported 410 cursor_expired and the client performs a replacement full resync (drop local rows not present in the full pull) — the WatermelonDB-style escape hatch recommended in research/04 §4.2.
• Cursors MUST NOT depend on wall-clock updatedAt alone (clock skew between app servers causes missed rows); use a sequence or (updatedAt, id) with a lag guard.

1.6 Rate limits (reference server defaults)

Scope	Limit	Notes
Per device, JSON routes	120 req / min	Burst 240
Per device, POST /v1/submissions:batch	20 req / min	50 items each ⇒ 1 000 submissions/min/device is far above any enumerator
Per device, tus PATCH	600 req / min, 200 MB / min	Chunk cadence, not item count
Per org	5 000 req / min	Sync storm after an outage is spread by 429 + Retry-After + client jitter
SSE	1 connection per device	Second connection closes the first

Responses carry RateLimit-* fields; a 429 always carries Retry-After.

---

2. Endpoint specification

Every device-facing route in 00 §8, plus two small additions the spine table does not list (GET /v1/ping, tus OPTIONS/DELETE) — backwards-compatible extensions proposed by this document.

2.1 GET /v1/ping — connectivity probe

No auth. Returns 204 with Date, X-Rasd-Server: rasd-server/<semver>, Cache-Control: no-store. The client uses it as the reachability probe (2 s timeout) that complements navigator.onLine / NetInfo (§4.4). Servers behind captive portals return HTML/200 — the client treats any non-204 as offline.

2.2 POST /v1/devices — register device, receive policy

Called on first sync, after every app update, every 24 h, and after any 401/403. Idempotent on deviceId.

// request
{
  "deviceId": "0198c1a2-…", "platform": "android" | "ios" | "web", "appVersion": "2.3.0", "client": "@rasd/sync/1.4.0", "locale": "ar",
  "capabilities": { "tus": true, "sse": true, "backgroundSync": false, "storage": "sqlite" | "dexie", "encryptedAtRest": true },
  "storage": { "usageBytes": 183000000, "quotaBytes": 6400000000, "persisted": true },   // optional diagnostics
  "pending": { "submissions": 12, "attachments": 31 }, "wipeAck": null, "ext": {}
}
// response 200
{
  "device": { "id": "0198c1a2-…", "status": "active" | "revoked" }, "serverTime": "2026-08-15T10:00:00.000Z", "supportedRasd": ["1.0"],
  "policy": {
    "intervals": { "pullSec": 900, "probeSec": 60, "policyRefreshSec": 86400 },
    "limits": { "batchMaxItems": 50, "batchMaxBytes": 5242880, "tusChunkBytes": 5242880, "tusMaxBytes": 104857600 },
    "retention": { "purgeSyncedAfterDays": 7, "keepSentMetadataDays": 90, "draftMaxAgeDays": 30, "datasetTtlHours": 168, "enforce": true },
    "metered": { "attachments": "wifiOnly" | "always" | "ask", "maxAttachmentBytesOnMetered": 2097152, "datasetsOver": 1048576 },
    "background": { "web": "accelerator" | "off", "native": { "enabled": false, "minimumIntervalMinutes": 15 } },
    "features": { "records": false, "sse": true },
    "wipe": null | { "nonce": "…", "issuedAt": "…", "reason": "device_lost", "mode": "sync-first" | "immediate" }
  },
  "license": "eyJ…"      // optional RLT, same as X-Rasd-License
}

retention.enforce: true means server values override host SyncPolicy defaults; otherwise they are hints. A device with status: "revoked" receives 403 device_revoked on every other route.

2.3 GET /v1/forms?since=<cursor> — form manifest delta

Returns the definitions assigned to this device/user (assignment is the host's business; the reference server assigns by org + optional project membership).

// response 200
{
  "items": [
    { "id": "pdm-gfd-2026", "version": "3", "definitionHash": "sha256:9f…", "rasd": "1.0", "title": { "en": "PDM – GFD", "ar": "…" },
      "state": "published" | "closing" | "closed" | "retired", "publishedAt": "…", "sizeBytes": 48211,
      "datasets": ["geo_gov", "geo_dist"], "requires": { "rasd": ">=1.0", "features": [] },
      "url": "/v1/forms/pdm-gfd-2026/versions/3", "signed": true },
    { "id": "old-form", "version": "1", "deleted": true, "deletedAt": "…" }             // tombstone: stop offering; keep if drafts reference it
  ],
  "cursor": "MTIzNDU6MDE5OA", "hasMore": false,
  "unsupported": [ { "id": "x", "version": "9", "reason": "spec_unsupported", "requires": { "rasd": ">=2.0" } } ]
}

Client behaviour: for each item whose (id, version) is not stored, fetch it (2.4); state drives the Start new list (only published for new submissions; closing/retired accepted for existing drafts/outbox; closed blocks push — server replies rejected form_closed); tombstones hide the form but the definition is retained while any draft/outbox item references it (research/12 §5).

2.4 GET /v1/forms/{id}/versions/{version} — one RFD

Returns the immutable definition. Headers: ETag: "<definitionHash>", Cache-Control: private, max-age=31536000, immutable, optional X-Rasd-Definition-Signature: <compact JWS> (payload = canonical JSON, alg: EdDSA, kid resolvable via GET /v1/.well-known/jwks.json on the same origin). Body = the RFD document exactly as published ($schema, rasd, id, version, …). The client MUST recompute SHA-256 over canonical JSON (RFC 8785 JCS) and compare with the manifest hash before storing; mismatch ⇒ discard + RASD_SYNC_REJECTED{reason:'definition_hash_mismatch'} in the sync log; a present-but-invalid signature is treated the same way. Media referenced by element.media.* are relative URLs under /v1/forms/{id}/versions/{version}/media/<path> (immutable, cacheable by registerRasdRoutes → rasd-media-v1).

2.5 GET /v1/datasets/{name}?since=<cursor> — dataset delta with tombstones

// response 200
{
  "dataset": { "name": "geo_dist", "version": "2026-06", "hash": "sha256:…", "keyField": "code", "columns": ["code", "name", "gov_code"], "rowCount": 4123 },
  "items": [
    { "key": "SY0201", "row": { "code": "SY0201", "name": { "en": "Aleppo", "ar": "حلب" }, "gov_code": "SY02" } },
    { "key": "SY0299", "deleted": true, "deletedAt": "2026-07-01T…" }
  ],
  "cursor": "…", "hasMore": true
}

Rows are applied with storage.datasets.putRows(name, rows, { since }) in the same transaction as the cursor. A dataset larger than policy.metered.datasetsOver is deferred while metered. Only columns referenced by installed forms are requested (?columns=code,name,gov_code) — field-level projection is a data-minimisation MUST for beneficiary lists (research/11); the server may ignore the hint. If dataset.hash changes without a compatible cursor (re-import), the server answers 410 cursor_expired and the client replaces the table.

2.6 POST /v1/submissions:batch — push finalized submissions

Request: ≤ 50 submissions, ≤ 5 MiB, Idempotency-Key = hex SHA-256 of the canonical JSON array [[id, checksum], …] in batch order.

{
  "items": [
    {
      "id": "0198c1a2-…", "formId": "pdm-gfd-2026", "formVersion": "3", "definitionHash": "sha256:9f…", "rasd": "1.0",
      "data": { "consent": "yes", "hh_members": [ { "name": "…", "age": 34 } ], "photo": { "attachmentId": "0198c1a3-…", "mime": "image/jpeg", "bytes": 182334, "sha256": "…" } },
      "meta": { "startedAt": "…", "finalizedAt": "…", "deviceId": "…", "userId": "…", "locale": "ar", "appVersion": "…", "platform": "android", "instanceName": "HH-0231 – Site A", "geo": null, "ext": {} },
      "attachments": [ { "id": "0198c1a3-…", "field": "photo", "mime": "image/jpeg", "bytes": 182334, "sha256": "…" } ],
      "audit": [ { "t": "…", "event": "value", "field": "consent", "old": null, "new": "yes" } ],   // ≤ 256 KiB inline, else attachment "$audit"
      "clientRev": 12, "createdAt": "…", "updatedAt": "…", "checksum": "sha256:…", "ext": {}
    }
  ]
}

checksum = "sha256:" + hex SHA-256 over JCS-canonical (RFC 8785) JSON of { id, formId, formVersion, definitionHash, data, meta, attachments } — exactly those keys, with each attachment projected to { id, field, mime, bytes, sha256 } and the array sorted by id, so a re-serialised submission hashes identically and the hash does not move as status/remoteId change during upload. Normative definition: 00 §6.1. @rasd/core computes it in finalize().

Response 200 (partial success is normal):

{
  "results": [
    { "id": "0198c1a2-…", "status": "accepted", "serverRev": 1, "receivedAt": "…", "checksum": "sha256:…", "complete": false, "missingAttachments": ["0198c1a3-…"], "quarantined": false, "issues": [] },
    { "id": "0198c1b0-…", "status": "duplicate", "serverRev": 1, "receivedAt": "…" },
    { "id": "0198c1c4-…", "status": "rejected", "error": { "code": "data_invalid", "message": "hh_size must be an integer", "field": "hh_size", "retryable": false } },
    { "id": "0198c1d9-…", "status": "conflict", "serverRev": 1, "error": { "code": "submission_conflict", "message": "A different payload for this id was accepted at 2026-08-14T…" } }
  ],
  "serverTime": "…"
}

Verdict semantics:

status	Server condition	Client transition
accepted	New id; version published (or quarantined); checksum verified; schema-valid	sending → synced (serverRev, syncedAt); attachments queue continues; quarantined: true is recorded in meta.ext["dev.rasd.sync"].quarantined (reverse-DNS vendor key per 00 §4, round-tripped like any ext) and shown as “held for review”
duplicate	Same id and same checksum already stored (retry after lost response)	sending → synced (idempotent)
rejected	Item cannot be stored as-is (data_invalid, checksum_mismatch, form_closed, payload_too_large, spec_unsupported; form_unknown only when the server runs with quarantine.enabled = false, §3)	sending → rejected with RasdError RASD_SYNC_REJECTED{ field }; leaves outbox; enumerator fixes and re-finalizes (new clientRev, same id)
conflict	Same id, different checksum already accepted (client bug, tampering, or a re-finalize after acceptance)	sending → conflict + RASD_SYNC_CONFLICT; local copy kept, supervisor decides (§5 UX hooks); never auto-overwritten

Envelope-level failures (auth, batch_too_large, idempotency_mismatch, 5xx) return the error body of §1.4 and no item changes state; the client retries the whole batch per §4.5. Ordering within a batch is preserved; the server processes items independently (no all-or-nothing) but each item is atomic.

Audit trails: inline audit[] is accepted up to 256 KiB per submission; larger trails must be attached as audit.jsonl (field: "$audit"), which is also how ODK-style audit.csv is mapped (research/14 §2).

2.7 Attachments — tus 1.0.0 on /v1/attachments

Profile of tus 1.0.0 required by RSP: core + creation + checksum + expiration + termination; creation-with-upload optional; concatenation not used. Tus-Resumable: 1.0.0 on every request/response. Servers SHOULD also accept the IETF successor (draft-ietf-httpbis-resumable-upload-12, largely tus-compatible per research/04 §4.3) once stable; clients speak tus 1.0.0 only in v1.

Step	Request	Response
Discover	OPTIONS /v1/attachments	204, Tus-Version: 1.0.0, Tus-Extension: creation,checksum,expiration,termination, Tus-Max-Size: 104857600, Tus-Checksum-Algorithm: sha256,sha1
Create	POST /v1/attachments with Upload-Length: <bytes>, Upload-Metadata: attachmentId <b64>,submissionId <b64>,formId <b64>,field <b64>,sha256 <b64>,mime <b64>,filename <b64>	201, Location: /v1/attachments/{attachmentId}, Upload-Expires: <HTTP-date, +7 d>. Idempotent: an existing attachmentId returns 201 with the existing Location (and current Upload-Offset header) instead of a new upload
Resume	HEAD /v1/attachments/{id}	200, Upload-Offset, Upload-Length, Cache-Control: no-store; 404/410 if unknown/expired ⇒ recreate
Send	PATCH /v1/attachments/{id}, Content-Type: application/offset+octet-stream, Upload-Offset: <n>, Upload-Checksum: sha256 <b64 of chunk>	204 + new Upload-Offset; 409 on offset mismatch (re-HEAD); 460 on chunk checksum failure; 413 chunk_too_large
Abort	DELETE /v1/attachments/{id}	204; used when the enumerator retakes a photo after upload started

Completion: when Upload-Offset == Upload-Length the server computes the whole-file SHA-256, compares it with Upload-Metadata.sha256, stores the object under attachments/{orgId}/{sha256} (dedup by content hash), links it to (submissionId, attachmentId), and marks the submission complete when no referenced hashes are missing. Whole-file mismatch ⇒ upload deleted, 412 checksum_mismatch on the final PATCH; the client re-hashes the local blob: if it now differs from the recorded sha256 the blob is corrupt ⇒ attachment failed + RASD_ATTACHMENT_FAILED surfaced to the enumerator (retake). Attachments may arrive before their submission; unlinked uploads expire after 7 days (Upload-Expires) and are garbage-collected. Limits: tusMaxBytes 100 MiB per attachment (policy), client chunk 5 MiB default, adaptive 256 KiB … 8 MiB (§4.4), 2 uploads in parallel, PATCH timeout 120 s. Auth: bearer required on every tus request; the upload URL is a capability secret and is redacted in logs.

2.8 Records / cases — GET /v1/records, POST /v1/records:batch (optional feature)

Enabled by policy.features.records. A record is a longitudinal entity keyed by id, typed by form (the RFD that defines its properties; settings.ext["dev.rasd.record"] = { keyField, label }).

// GET /v1/records?form=hh_registry&since=<cursor>  → 200
{ "items": [ { "id": "0198…", "form": "hh_registry", "rev": 7, "label": "HH-0231", "data": { "head_name": "…", "size": 6 },
               "fields": { "size": { "hlc": "1755252062123-0003-a1b2c3d4", "deviceId": "…", "userId": "…" } },
               "updatedAt": "…", "deleted": false } ],
  "cursor": "…", "hasMore": false }

// POST /v1/records:batch  (≤ 100 ops, Idempotency-Key)
{ "ops": [ { "op": "upsert", "id": "0198…", "form": "hh_registry", "baseRev": 7,
             "changes": { "size": { "value": 7, "hlc": "1755252071000-0001-a1b2c3d4" }, "phone": { "value": "0999…", "hlc": "1755252071000-0002-a1b2c3d4" } },
             "submissionId": "0198c1a2-…" },                          // provenance: the submission that produced the change
           { "op": "delete", "id": "0198…", "form": "hh_registry", "baseRev": 3, "hlc": "…" } ] }
// → 200
{ "results": [
    { "id": "0198…", "status": "applied", "rev": 8, "fields": { "size": "applied", "phone": "applied" } },
    { "id": "0198…", "status": "conflict", "rev": 12,
      "fields": { "size": "superseded" },
      "conflict": { "field": "size", "mine": { "value": 7, "hlc": "…", "deviceId": "…" }, "theirs": { "value": 8, "hlc": "…", "deviceId": "…", "userId": "…" }, "queueId": "01J5…" },
      "current": { "rev": 12, "data": { "…": "…" }, "fields": { "…": {} } } } ] }

Server rules: per field, apply the change iff incoming.hlc > stored.hlc (applied), else superseded; rev increments per accepted op; an op is conflict when at least one field is superseded and the two HLCs are within the review window (default 5 days, mirroring Central's out-of-order hold research/12 §10) — older stragglers are superseded silently. Every conflict also creates a row in the supervisor conflict queue (§9.2). baseRev is informational (drives the current snapshot in the response), never a precondition — offline devices are always allowed to write. HLC format: "<unixMillis 13 digits>-<counter 4 hex>-<node 8 hex>", lexicographically ordered; the client advances its HLC from every serverTime/Date it sees, so a device with a wrong clock cannot win forever.

2.9 GET /v1/events — Server-Sent Events (optional)

Accept: text/event-stream, bearer + X-Rasd-Device as usual (EventSource cannot set headers, so the client uses fetch streaming; on RN the engine falls back to polling unless the host injects an SSE-capable fetch). Server sends retry: 15000, a : ping comment every 25 s, and events with id: (a cursor) so Last-Event-ID resumes:

event: formPublished
id: 8812
data: {"formId":"pdm-gfd-2026","version":"4","definitionHash":"sha256:…"}

event: datasetUpdated
data: {"name":"geo_dist","cursor":"…"}

event: recordChanged
data: {"form":"hh_registry","count":3}

event: policyUpdated
data: {}

The four SSE event names — formPublished, datasetUpdated, recordChanged, policyUpdated — are the wire names fixed by the event catalogue in 17 §17. They are deliberately not the engine's own event names: an SSE formPublished makes the engine run phase 3 and, once the definition is stored, emit the engine event formUpdated (§6).

Events are triggers only: the client never applies SSE payloads; it schedules the corresponding pull (syncNow('event') debounced 2 s). Missing SSE never causes data loss because timers and foreground triggers perform the same pulls.

2.10 Non-device routes of the reference server (not part of RSP v1)

POST /v1/forms/{id}/versions (publish; rejects duplicate version / unchanged definitionHash; stores changelog + MigrationPlan), POST /v1/datasets/{name}/import, GET /v1/export/forms/{id}/submissions.{csv,json,xlsx}, GET /v1/odata/forms/{id}.svc, /v1/webhooks, /v1/quarantine, /v1/conflicts — see §9.1 and §9.5. They require an admin-scoped token; devices never call them.

---

3. Server-side validation of a submission

For each item, in order (first failure decides; cost is bounded so a 50-item batch validates in < 250 ms p95 on the reference server):

1. Envelope: id is a UUID v7; formId slug; checksum well-formed; item ≤ 1 MiB (audit excluded).
2. Idempotency / dedupe: look up (org_id, id). Exists with same checksum ⇒ duplicate. Exists with different checksum ⇒ conflict (never overwrite; the enumerator's attestation is immutable — 00 §4.3b).
3. Form & version: (formId, formVersion) published for this org and form not closed ⇒ continue. Form closed ⇒ rejected form_closed. Form unknown / version never published / definitionHash ≠ stored hash ⇒ accept into quarantine (accepted, quarantined: true, stored with raw bytes and reason) — the fix for silent drops documented in research/12 §6. Quarantine is admin-visible and can be admitted later against a chosen version.
4. Checksum: recompute over JCS-canonical JSON of the seven keys; mismatch ⇒ rejected checksum_mismatch (client re-serialisation bug or tampering).
5. RFD data validation (@rasd/core engine loaded per definitionHash, cached): structural/type errors (unknown element names outside ext, wrong value types, repeat cardinality below min, select values outside a static choice list) ⇒ rejected data_invalid with field. Constraint / required / dataset-choice failures ⇒ accepted with issues[] and reviewState: "hasIssues" — server-side re-evaluation can legitimately differ (now(), today(), dataset versions, once()), so it must not block delivery. Irrelevant fields present in data are dropped (ODK semantics) and noted as a warning.
6. Attachments: every attachments[].sha256 referenced by data must appear in attachments[]; each bytes ≤ attachmentMaxBytes; presence in the store decides complete.
7. Store: insert submission row (JSONB), attachment links, per-item webhook event, in one transaction per item.

Reference-server policy switches: validation.mode = "flag" (default) | "strict" (strict turns constraint failures into rejected), quarantine.enabled = true, acceptClosedFormsFromOutbox = false.

---

4. Client sync engine (@rasd/sync)

4.1 Construction and policy

import { createSyncEngine } from '@rasd/sync';

const sync = createSyncEngine({
  storage,                                     // StorageAdapter (00 §7)
  baseUrl: 'https://forms.example.org/api',
  getAuthToken: async ({ forceRefresh }) => host.getAccessToken(forceRefresh),
  transport: undefined,                        // omitted ⇒ createRspTransport({ baseUrl, getAuthToken }); or createOpenRosaTransport(...) (phase 3) or a custom SyncTransport
  policy: {                                    // all values are the defaults
    batch: { maxItems: 50, maxBytes: 5 * 2 ** 20 },
    attachments: { chunkBytes: 5 * 2 ** 20, minChunkBytes: 256 * 1024, maxChunkBytes: 8 * 2 ** 20, parallel: 2, onMetered: 'wifiOnly', maxBytesOnMetered: 2 * 2 ** 20 },
    backoff: { baseMs: 1000, capMs: 300_000, deadAfterAttempts: 5 },
    timers: { pullIntervalMs: 900_000, probeIntervalMs: 60_000, requestTimeoutMs: 30_000, chunkTimeoutMs: 120_000, leaseMs: 120_000 },
    triggers: { online: true, foreground: true, finalize: true, timer: true, events: true },
    retention: { purgeSyncedAfterDays: 7, keepSentMetadataDays: 90, purgeAttachmentsWithSubmission: true },
    background: { web: 'accelerator', native: { enabled: false, minimumIntervalMinutes: 15 } },
    records: { enabled: false }, requireHttps: true, fetch: undefined /* inject for TLS pinning / SSE on RN */, log: { level: 'warn', maxEntries: 500 },
  },
});
sync.on('progress', p => …); sync.on('conflict', c => …);
await sync.syncNow('manual');

Server DevicePolicy (2.2) is merged over these values on every registration (server wins for retention, limits, intervals, metered, background when enforce: true).

getAuthToken keeps the spine signature getAuthToken(): Promise<string> (00 §8); the engine passes an optional { forceRefresh } hint after a 401 (§4.6), which hosts implementing the plain zero-argument signature can ignore.

4.2 State machine

stateDiagram-v2
  [*] --> idle
  idle --> acquiring: trigger fires
  acquiring --> idle: another tab holds the lock
  acquiring --> probing: lock acquired
  probing --> offline: probe failed
  offline --> probing: online event or probe timer
  probing --> running: probe returned 204
  running --> running: next phase
  running --> idle: all queues drained
  running --> backoff: retryable failure
  backoff --> probing: backoff timer or new trigger
  running --> authRequired: second 401
  authRequired --> probing: host resumed with a token
  running --> revoked: 403 device_revoked
  running --> wiping: wipe order received
  wiping --> [*]
  idle --> paused: paused by host
  running --> paused: paused by host
  paused --> idle: resumed by host

Reading the diagram: a trigger is online, foreground, finalize, timer, event or manual (SyncReason, 17 §7). running cycles through the six phases of §4.3 in order. running → idle records lastSuccessAt and resets the backoff attempt counter. Retryable failures are network errors and HTTP 408/425/429/500–504; the backoff timer is full-jitter capped at 5 min (§4.5). authRequired is entered only after a 401 that survived one forced token refresh. wiping is entered on X-Rasd-Wipe or policy.wipe (§7.4). pause() lets the in-flight request finish and releases the lock; resume() returns to idle.

Leadership: on web the engine holds navigator.locks.request('rasd-sync:<namespace>', { ifAvailable: true }) for the duration of a run; a follower that fails to take the lock falls back to idle and shows live status received over BroadcastChannel('rasd-sync:<namespace>'), forwarding syncNow() requests to the leader. On RN there is one engine per process. syncNow() is single-flight: a call during a run returns the in-progress promise (and sets a rerun flag so new outbox items are picked up before the run ends).

4.3 Phases and batching

Phase	Source	Batching	Done when
1 Submissions	outbox.peek(50, { kinds: ['submission'] }) (09 §3), FIFO by UUID v7	up to 50 items / 5 MiB per POST …:batch; oversize single item (> 1 MiB) is sent alone; audit > 256 KiB spun off as $audit attachment	outbox has no submission op that is not leased/dead
2 Attachments	attachments.listByStatus(['pending','uploading','failed']) whose submission is synced (or accepted-quarantined), skipping rows whose submissionId is the $asset sentinel (see below); ordered by submission age, smallest first inside a submission	2 concurrent tus uploads; chunk size adaptive	no eligible attachment left (metered-deferred ones are excluded and counted separately)
3 Forms	GET /v1/forms?since loop, then per-definition fetch (max 4 concurrent)	pages	hasMore=false and all fetches stored
4 Datasets	for each dataset referenced by installed forms: GET /v1/datasets/{name}?since	pages of ≤ 5 000 rows applied per transaction	as above
5 Records	push records:batch (≤ 100 ops) then pull GET /v1/records per form		queues drained
6 Purge & housekeeping	retention (§8), orphan blob GC, sync-log trim

Phases 3–5 run at most once per pullIntervalMs unless triggered by an event or manual; phase 1–2 run on every trigger. A phase failure with a retryable error aborts the run into backoff; a non-retryable failure of one item never stops the phase.

Local host assets are never uploaded. Phase 2 excludes every attachment whose submissionId is the reserved $asset sentinel: such rows belong to no submission and are host assets, today the parts of an offline basemap stored under the reserved basemap: id prefix (09 §3, 14 §4.4). They are never enqueued in the outbox, never counted in phase-2 progress (§4.9), have no remoteId and no server counterpart, and are exempt from retention and orphan GC in phase 6 — only the host deletes them.

4.4 Connectivity, probes and metered networks

• Signals: web navigator.onLine + online/offline events + document.visibilitychange; RN @react-native-community/netinfo (isConnected, isInternetReachable, details.isConnectionExpensive, type) + AppState. All are hints; the truth is GET /v1/ping (2 s timeout) before a run and after every network error.
• Metered: web navigator.connection.saveData || type === 'cellular' (Chromium only; unknown ⇒ not metered); RN isConnectionExpensive || type === 'cellular'. Policy attachments.onMetered: always | wifiOnly (defer attachments larger than maxBytesOnMetered; submissions and definitions always sync) | ask (emit progress{ waiting: 'metered', bytesTotal } and wait for resume({ allowMetered: true })).
• Adaptive chunks: start at chunkBytes; after a chunk failure halve (floor minChunkBytes); after 4 consecutive successes double (cap maxChunkBytes); persist the last good size in kv per network type. On 2G/3G tests a 5 MB photo with 256 KiB chunks survives repeated drops without re-sending more than one chunk.

4.5 Retry policy per error class and backoff schedule

Class	Examples	Action
Network	DNS/TLS/timeout/TypeError: Failed to fetch, probe fails	RASD_SYNC_NETWORK; run → backoff (table below); item stays sending with lease
Transient server	408, 425, 429 (Retry-After wins), 500–504	same as network (RASD_SYNC_NETWORK)
Auth	401	getAuthToken({ forceRefresh: true }) once, redo the same request; second 401 ⇒ RASD_SYNC_AUTH + authRequired (no backoff, no data change)
Policy	403 device_revoked	RASD_SYNC_REVOKED + revoked state (§7.3); 403 org_suspended / forbidden on one route ⇒ RASD_SYNC_AUTH, that phase skipped, error emitted
Item-level terminal	rejected verdicts, 412 checksum, 422 data	RASD_SYNC_REJECTED; submission → rejected; op acked; no retry until the user re-finalizes
Envelope shape	400 batch_too_large/payload_too_large	split batch in half and retry immediately (down to 1)
Cursor	410 cursor_expired	clear cursor, replacement resync of that collection
tus offset/checksum	409, 460	re-HEAD, continue from server offset; 3 consecutive 460 ⇒ re-hash local blob
tus expired	404/410 on HEAD	recreate upload (idempotent by attachmentId), start at 0
Client-side exception	serialization throws, storage read fails	attempt counter++; deadAfterAttempts ⇒ dead-letter (§4.7)

Backoff (full jitter, sleep = random(0, min(capMs, baseMs · 2^attempt)), attempt counter reset by any successful request; a new trigger bypasses the wait):

attempt	0	1	2	3	4	5	6	7	8	≥ 9
max sleep	1 s	2 s	4 s	8 s	16 s	32 s	64 s	128 s	256 s	300 s

Retry-After (clamped 1 s – 3600 s) replaces the computed sleep. While backing off the UI shows “Retrying in ~N s” from status().backoffUntil.

4.6 Token expiry mid-run

Tokens are fetched lazily per request (getAuthToken() result cached in memory for ≤ 60 s). A 401 in the middle of an attachment upload triggers one refresh and the PATCH is retried from the HEAD offset — no bytes are lost. If refresh fails the engine parks in authRequired, keeps all leases valid until they expire, and emits error{ code: 'RASD_SYNC_AUTH' }; the host shows a re-login UI and calls resume().

4.7 Poison messages and dead-lettering

An outbox op that (a) throws client-side deadAfterAttempts (5) times, or (b) receives a non-retryable envelope error the engine cannot fix (415, 422 idempotency_mismatch after re-hashing), is moved to the outbox dead state (outbox.fail(id, { code, message, terminal: true }, null) — 09 §3), the submission is set to rejected with RasdError{ code:'RASD_SYNC_REJECTED', details:{ reason:'client_error', attempts } }, and error is emitted. Dead ops never block the queue (the next op proceeds) and are visible in status().queues.submissions.dead and outbox.dead(); the host offers Retry (sync.retry(ids), which resets attempts and returns the ops to pending) and Export (storage.export()).

4.8 Resumption after an app kill or tab close

• Outbox ops are leased: peek marks leaseUntil = now + leaseMs; on start any expired lease is visible again. Submissions found in sending at start revert to queued. Because the batch is idempotent, a lost response simply yields duplicate next time.
• Attachments found uploading at start: HEAD → continue. Upload URL and offset are persisted on the attachment row after every successful PATCH (attachments.patch(id, { uploadUrl, uploadOffset }), 09 §3); the last good chunk size lives in kv per network type (§4.4), not on the row.
• Cursors are committed with the applied page; a kill mid-page re-applies the same page (idempotent putRows/forms.put).
• Web tab discard: the Web Lock is released automatically; another tab or the next visit takes over.

4.9 Progress accounting

interface SyncStatus {
  state: 'idle' | 'acquiring' | 'probing' | 'offline' | 'running' | 'backoff' | 'authRequired' | 'revoked' | 'paused' | 'wiping';
  phase?: 'submissions' | 'attachments' | 'forms' | 'datasets' | 'records' | 'purge';
  leader: boolean; network: { online: boolean; metered: boolean | null; type?: string };
  lastSyncAt?: string; lastSuccessAt?: string; backoffUntil?: string; attempt: number;
  queues: {
    submissions: { pending: number; sending: number; rejected: number; conflict: number; dead: number; oldestPendingAt?: string };
    attachments: { pending: number; uploading: number; failed: number; deferredMetered: number; bytesTotal: number; bytesSent: number };
    forms: { toFetch: number }; datasets: { toPull: number }; records: { pendingOps: number; conflicts: number };
  };
  lastError?: RasdError;
}
// event 'progress'
interface SyncProgress { phase: SyncStatus['phase']; done: number; total: number; bytesDone?: number; bytesTotal?: number; current?: { kind: 'submission' | 'attachment' | 'form' | 'dataset'; id: string; label?: string }; waiting?: 'metered' | 'backoff' | 'auth'; }

In queues.submissions, sending, rejected and conflict are spine submission statuses (00 §6); pending and dead are outbox op states (09 §3) — a pending op corresponds to a submission in queued. bytesSent counts acknowledged tus offsets (not request bytes), so a progress bar never moves backwards except after a 409 offset correction. useSync() in the renderers is a thin projection of SyncStatus (06 §4, 17 §7).

4.10 Background execution

• Web (Chromium only, ~76.7 % global; no Safari/Firefox — research/05 §3): the bridge is @rasd/pwa's connectBackgroundSync({ sync, registration }) in the page plus RasdOutboxPlugin (@rasd/pwa/sw) in the worker, specified in 11 §4. The page registers the tag rasd-outbox (feature-detected) whenever a submission enters queued while offline or a batch fails with a network error. On the sync event the worker posts { type: 'rasd:sync-wake' } to every window client; if no client is open it rejects, which keeps the tag pending so Chromium retries with its own backoff (it resolves on event.lastChance). The worker never drains the outbox itself: auth is host-delegated and getAuthToken() lives in the page, so a headless drain would require a bearer token inside the worker bundle — an explicitly rejected trade-off (Open questions). Workbox's BackgroundSyncPlugin is not used for RSP requests either: it queues only on network exceptions, ignores 4xx/5xx and replays without idempotency awareness, and a second queue beside storage.outbox would duplicate submissions (research/05 §3). Periodic Background Sync (installed Chromium PWAs, tag rasd-forms-pull) may wake a client to refresh forms/datasets opportunistically.
• React Native: expo-background-task (WorkManager / BGTaskScheduler; minimumInterval ≥ 15 min, OS-scheduled, skipped on low battery, not on simulators — research/04 §5). When policy.background.native.enabled, the registered task calls syncNow('background') with a 25 s wall budget (iOS gives ~30 s), submissions + attachments ≤ 5 MiB only, and returns Success/Failed. Foreground triggers remain the primary path; the UI must always show last sync and pending counts.

---

5. Records/cases on the client

• Local model: records live in storage.datasets under __records:<form> (rows keyed by record id with fields{} metadata) — no new storage table in v1; edits go through the outbox as record ops so they share leasing, retries and dead-lettering.
• Each field write stamps hlc = max(localHlc, lastServerHlc) + 1; the HLC node id is the first 8 hex of deviceId.
• Pull applies per-field LWW locally with the same rule as the server, so a device converges to the server state without a merge dialog. A pulled row that overwrites a locally pending change of the same field marks the local record conflict (spine status) and emits conflict.
• UX hooks: sync.on('conflict', c => …) receives a Conflict = { kind: 'record' | 'submission', id, form, field?, local, server, queueId, resolve(resolution) }, where resolution uses the same vocabulary as the imperative sync.resolveConflict(id, resolution) in 17 §7: 'keep-local' | 'keep-server' | { merged }. 'keep-local' re-enqueues the field with a fresh HLC; 'keep-server' clears the pending change; { merged } writes merged values. (local/server are the client-side names for the wire's mine/theirs in §2.8, matching RASD_SYNC_CONFLICT.details = { recordId, local, server }.) Enumerators never see a merge dialog by default: the renderer surfaces conflicts in the Sent/Records list for supervisors (research/04 §4.4); the host may hide them entirely and rely on the server queue.

---

6. Observability

• Events (the full SyncEvents set of 17 §7; sync.on(event, handler) returns an unsubscribe function, 00 §12): the six spine events progress, error(RasdError), conflict, formUpdated{ id, version, definitionHash }, datasetUpdated{ name, rows }, licenseRefreshed{ exp } (00 §8), plus stateChange(SyncStatus), remoteWipe{ reason, unsyncedCount } (§7.4) and policyUpdated(DevicePolicy) (§2.2). Engine event names are distinct from the SSE wire names of §2.9.
• Sync log: ring buffer in kv['sync.log'] (500 entries × ≤ 512 B), one entry per request/phase: { t, level, event, phase, httpStatus, wire, durationMs, count, bytes, requestId, attempt }. Never contains answers, tokens, or full tus URLs (/v1/attachments/0198… is logged as att:0198). Exposed via sync.getLog({ since, level }) and included in storage.export() diagnostics.
• Metrics for the host UI/dev panel: sync.getMetrics() → { runs24h, successRate24h, avgBatchMs, p95BatchMs, bytesUploaded24h, oldestPendingAgeMs, rejected, dead, conflicts, lastPolicyAt }. rasd doctor (CLI/dev panel) prints them with storage estimate and license state.
• Reference server emits OpenTelemetry spans per route, and Prometheus counters rsp_submissions_total{verdict}, rsp_batch_seconds, rsp_tus_bytes_total, rsp_quarantine_total, rsp_conflicts_total, rsp_devices_active.

---

7. Security

1. Transport: TLS only (requireHttps), no redirects followed for POST/PATCH; hosts may inject fetch for SPKI pinning (documented trade-off: pinning failures cause field outages, default off — research/11 §6). Bearer tokens live in memory on web and in SecureStore on RN when the host chooses to persist them; the engine never persists them.
2. Token expiry mid-sync: §4.6. The server MUST accept a fresh token on a PATCH continuing an upload created under an older token of the same device/org.
3. Device revocation: admin sets devices.status = 'revoked' → next request 403 device_revoked → engine revoked: no further network, all data retained, error{ code: 'RASD_SYNC_REVOKED', details: { reason: 'device_revoked' } } (17 §16), storage.export() remains available (P6). Re-activation on the server + resume() continues where it stopped.
4. Remote wipe: X-Rasd-Wipe: <nonce> on any authenticated 2xx, or policy.wipe. Client verifies the nonce is new (kv['wipe.seen']), then: mode: 'sync-first' ⇒ one final run with a 60 s budget for submissions only; then storage.wipe() (crypto-shred keys, delete DB/blobs, clear profiles), emit the remoteWipe{ reason, unsyncedCount } event (17 §7); on web the server additionally sends Clear-Site-Data: "storage" on the device's next navigation. Wipe orders are logged server-side with actor and reason.
5. Integrity: SHA-256 per submission and attachment, echoed in acks; JWS-signed definitions verified against the server's JWKS (cached 7 days, refreshed opportunistically); definitionHash on every submission lets auditors reproduce the exact form.
6. Server: parameterised SQL, per-org row-level security, request body limits (JSON 6 MiB, tus chunk 8 MiB), rate limits (§1.6), audit log of admin actions, tus URLs unguessable (UUID v7 + org scoping) and auth-required.

---

8. Data retention on the device

Item	Default	Configurable via
Synced submissions (answers + attachments)	purged 7 days after syncedAt and complete ack of every attachment (0 = immediately after ack, mirroring Collect “delete after send” research/01 §4)	retention.purgeSyncedAfterDays (server-enforceable)
“Sent” metadata (id, instanceName, formId/version, syncedAt, serverRev)	kept 90 days for the enumerator's Sent list	retention.keepSentMetadataDays
Rejected / conflict submissions	never auto-purged (user action)	—
Drafts	warn at 30 days (draftMaxAgeDays), never auto-deleted	retention.draftMaxAgeDays
Datasets	TTL 168 h then re-pulled; rows for uninstalled forms dropped	retention.datasetTtlHours
Form definitions	referenced by any draft/outbox item + last 3 versions per form	forms.keepVersions
Sync log	500 entries	log.maxEntries

Purge runs in phase 6 and once per day; it is skipped while storage.estimate() cannot be read; it never deletes an attachment whose submission is not synced.

---

9. Reference server @rasd/server

9.1 Runtime and layout

Node ≥ 20, Hono app (createRspApp({ db, blobs, auth, hooks })) mountable in Node, Bun or a serverless adapter; Postgres 15+ (pg + postgres.js allowed; SQL migrations in migrations/*.sql); S3-compatible object store via @tus/server + @tus/s3-store (tus-node-server) behind /v1/attachments; optional Redis for rate limits/SSE fan-out (single-node falls back to in-memory). Ships as a Docker image + docker compose (Postgres, MinIO) + Helm chart. Routes:

GET  /v1/ping                          POST /v1/devices
GET  /v1/forms                          GET  /v1/forms/:id/versions/:version[/media/*]
GET  /v1/datasets/:name                 POST /v1/submissions:batch
OPTIONS|POST /v1/attachments            HEAD|PATCH|DELETE /v1/attachments/:id
GET  /v1/records                        POST /v1/records:batch
GET  /v1/events                         GET  /v1/.well-known/jwks.json
-- admin (scope admin:*): POST /v1/forms/:id/versions · POST /v1/datasets/:name/import · GET /v1/quarantine · POST /v1/quarantine/:id/admit
--                        GET /v1/conflicts · POST /v1/conflicts/:id/resolve · CRUD /v1/webhooks · GET /v1/export/forms/:id/submissions.{csv,json,xlsx} · GET /v1/odata/forms/:id.svc

9.2 Postgres schema (DDL excerpt)

CREATE TABLE orgs (id uuid PRIMARY KEY, slug text UNIQUE, settings jsonb NOT NULL DEFAULT '{}');
CREATE TABLE devices (org_id uuid, id uuid, platform text, app_version text, status text NOT NULL DEFAULT 'active',
  policy jsonb NOT NULL DEFAULT '{}', wipe jsonb, last_seen_at timestamptz, PRIMARY KEY (org_id, id));
CREATE TABLE forms (org_id uuid, id text, state text NOT NULL DEFAULT 'published', PRIMARY KEY (org_id, id));
CREATE TABLE form_versions (org_id uuid, form_id text, version text, definition_hash text NOT NULL, rasd text NOT NULL,
  definition jsonb NOT NULL, jws text, changelog jsonb, migration_plan jsonb, published_at timestamptz DEFAULT now(),
  seq bigint NOT NULL, PRIMARY KEY (org_id, form_id, version), UNIQUE (org_id, form_id, definition_hash));
CREATE TABLE form_assignments (org_id uuid, form_id text, device_id uuid, seq bigint NOT NULL, deleted_at timestamptz,
  PRIMARY KEY (org_id, form_id, device_id));
CREATE TABLE datasets (org_id uuid, name text, version text, hash text, key_field text, columns jsonb, PRIMARY KEY (org_id, name));
CREATE TABLE dataset_rows (org_id uuid, dataset text, key text, row jsonb, seq bigint NOT NULL, deleted_at timestamptz,
  PRIMARY KEY (org_id, dataset, key));
CREATE TABLE submissions (org_id uuid, id uuid, form_id text NOT NULL, form_version text NOT NULL, definition_hash text NOT NULL,
  device_id uuid, user_id text, checksum text NOT NULL, data jsonb NOT NULL, meta jsonb NOT NULL, audit jsonb,
  client_rev int, server_rev int NOT NULL DEFAULT 1, received_at timestamptz DEFAULT now(),
  complete boolean NOT NULL DEFAULT false, review_state text, issues jsonb,
  quarantined boolean NOT NULL DEFAULT false, quarantine_reason text, raw bytea, seq bigint NOT NULL,
  PRIMARY KEY (org_id, id)) PARTITION BY LIST (org_id);
CREATE INDEX ON submissions (org_id, form_id, seq);
CREATE TABLE attachments (org_id uuid, sha256 text, bytes bigint, mime text, storage_key text NOT NULL, PRIMARY KEY (org_id, sha256));
CREATE TABLE submission_attachments (org_id uuid, submission_id uuid, attachment_id uuid, field text, sha256 text, bytes bigint,
  mime text, uploaded boolean NOT NULL DEFAULT false, PRIMARY KEY (org_id, submission_id, attachment_id));
CREATE TABLE tus_uploads (org_id uuid, id uuid, sha256 text, length bigint, offset_bytes bigint NOT NULL DEFAULT 0, metadata jsonb,
  storage_key text, expires_at timestamptz NOT NULL, completed_at timestamptz, PRIMARY KEY (org_id, id));
CREATE TABLE records (org_id uuid, form_id text, id uuid, rev int NOT NULL, label text, data jsonb NOT NULL, fields jsonb NOT NULL,
  deleted_at timestamptz, updated_at timestamptz, seq bigint NOT NULL, PRIMARY KEY (org_id, form_id, id));
CREATE TABLE record_conflicts (id uuid PRIMARY KEY, org_id uuid, form_id text, record_id uuid, field text, mine jsonb, theirs jsonb,
  status text NOT NULL DEFAULT 'open', resolved_by text, resolved_at timestamptz);
CREATE TABLE idempotency_keys (org_id uuid, device_id uuid, key text, request_hash text NOT NULL, response jsonb NOT NULL,
  created_at timestamptz DEFAULT now(), PRIMARY KEY (org_id, device_id, key));
CREATE TABLE webhooks (id uuid PRIMARY KEY, org_id uuid, url text, secret text, events text[], active boolean DEFAULT true);
CREATE TABLE webhook_deliveries (id uuid PRIMARY KEY, webhook_id uuid, event jsonb, attempts int DEFAULT 0, next_at timestamptz,
  delivered_at timestamptz, last_status int);
-- `seq` columns come from one global BIGSERIAL (per-org sequences are an optimisation);
-- every table has ROW LEVEL SECURITY: org_id = current_setting('rasd.org_id')::uuid

9.3 Storage of bytes

Objects are keyed attachments/{orgId}/{sha256} (dedup: two enumerators photographing the same poster upload once). tus temp objects live under tus/{orgId}/{uploadId} and are promoted on completion; a nightly job deletes expired uploads and unreferenced objects older than 30 days. Server-side encryption (SSE-S3/KMS) is enabled by default; hosts on air-gapped infrastructure may configure the filesystem store.

9.4 Auth adapter interface

export interface AuthAdapter {
  verify(token: string, ctx: { deviceId: string; ip: string; route: string }): Promise<
    | { ok: true; orgId: string; userId: string; roles: string[]; scopes?: string[]; expiresAt?: string }
    | { ok: false; code: 'unauthorized' | 'token_expired' | 'forbidden' }>;
  issueLicense?(orgId: string): Promise<string | null>;                 // returns an RLT to attach as X-Rasd-License
  onDeviceRegistered?(orgId: string, device: DeviceInfo): Promise<Partial<DevicePolicy> | void>;
}

Bundled adapters: jwtAuth({ jwksUrl | secret, issuer, audience, claims: { org: 'org_id', user: 'sub', roles: 'roles' } }), staticApiKeyAuth() (dev), centralAppUserAuth() (token-in-path style for Collect-like enrolment). Rasd never stores end-user credentials.

9.5 Multi-tenancy, webhooks, exports

• Multi-tenant: org_id on every row, RLS enforced via SET LOCAL rasd.org_id; per-org quotas (settings.limits); partitions per org for submissions when > 100 k rows; per-org JWKS key for signing definitions.
• Webhooks: events submission.accepted, submission.completed (all attachments present), submission.quarantined, record.updated, record.conflict, form.published, device.registered; payload = the entity + event, orgId, occurredAt; header X-Rasd-Signature: t=<unix>,v1=<hex HMAC-SHA256(secret, t + "." + body)>; delivery via webhook_deliveries worker with backoff up to 24 h and manual redelivery. Downstream targets: Power BI dataflows, DHIS2 bridges, RapidPro, data warehouses.
• Exports: Kobo/Ona-family CSV/XLSX by default (group/question headers, q + q/choice 0/1 columns, _q_latitude…, repeat sheets with _index/_parent_table_name/_parent_index, system columns incl. __version__), mode=central switch, versions=all|latest|<v> with union-of-columns and alias merge, always __version + __definitionHash; an OData 4.0 minimal feed shaped like Central's (Submissions, Submissions.{repeat}, __id, __system, $expand=*, $skiptoken) so Power BI templates work unchanged (research/14 §7). Exports never include quarantined rows unless includeQuarantined=true.

9.6 Scaling notes (10 k devices, 1 M submissions per tenant)

• Load shape: 10 k devices polling /v1/forms every 15 min ≈ 11 req/s; a synchronous field day of 2 000 enumerators × 30 submissions ≈ 60 k submissions ≈ 1 200 batches — trivial for one Postgres. The risk is the post-outage storm: 10 k devices reconnecting within a minute → 429 + Retry-After (server-computed spread of 0–120 s) plus client jitter flattens it.
• Storage: 1 M submissions × ~10 KiB JSONB ≈ 10 GiB + indexes; keep raw bytes only for quarantined rows; partition by org (and by month above 5 M); seq-indexed cursors keep delta pulls O(page). Attachments dominate: 1 M × 3 × 300 KiB ≈ 900 GiB in S3 — never in Postgres.
• tus: stateless PATCH handling with S3 multipart under the hood; sticky sessions not required because offsets are in the DB; chunk size ≤ 8 MiB keeps memory per request bounded; 200 concurrent uploads per node is a comfortable default.
• SSE: 10 k idle connections need a dedicated node (or polling only); fan-out via Postgres LISTEN/NOTIFY (single node) or Redis pub/sub.
• Connections: pgbouncer transaction pooling; batch inserts inside one transaction per item; idempotency table pruned after 48 h; validation engines cached per definitionHash (LRU 500).
• Ops SLOs: batch p95 < 500 ms, HEAD p95 < 50 ms, definition fetch served from CDN with immutable caching.

---

10. Interoperability: RSP ⇄ OpenRosa / ODK Central / KoboToolbox

The openrosa transport (phase 3, @rasd/sync + @rasd/xlsform serializer) maps the engine's operations onto OpenRosa 1.0 and vendor REST; the table shows where RSP concepts land (research/14 §1, §5).

RSP	Generic OpenRosa	ODK Central v2026.2	KoboToolbox (kc/kpi)
POST /v1/devices + policy	— (Collect QR settings)	app-user token in path /v1/key/{token}/…	Data-Collector token /collector/{token}/…
GET /v1/forms?since (delta, hash)	formList (hash md5:, version) — full list, client diffs	formList or /v1/projects/{id}/forms	formList on kc; kpi assets/
GET …/versions/{v} (RFD, ETag)	XForm XML → @rasd/xlsform import	.xml/.xlsx	asset ?format=xml
GET /v1/datasets (delta, tombstones)	manifest mediaFile (full CSV)	entities.csv with ETag	manifest CSV / paired-data (full)
POST /v1/submissions:batch (per-item verdicts)	one multipart POST /submission per submission (HEAD first, ACL chunking, XML repeated per chunk); 201/202	same, or XML body POST …/forms/{xmlFormId}/submissions	multipart or JSON {"id","submission"} on kc
duplicate / conflict	202 / 409 (same instanceID, different XML)	409	202 “Duplicate Instance” / 409
Idempotency key	meta/instanceID (uuid: + our id)	same	same
tus attachments	media parts in the multipart POST (split by X-OpenRosa-Accept-Content-Length: 10 MB Kobo, 100 MB Central)	per-file POST …/submissions/{id}/attachments/{filename}	multipart only
Records	—	Entities (PATCH …/entities/{uuid}?baseVersion=, string values, branchId)	none
SSE, X-Rasd-Wipe, license header, quarantine, issues[]	—	—	—

Consequences: OpenRosa transports collapse phases 1–2 into one multipart per submission (no resumable attachments), map formList hashes onto definitionHash, treat 401/403 as authRequired, and never see quarantine, records or wipe. Browser PWAs usually need the relay transport (Rasd server proxying Kobo/Central with the user's token) because of CORS.

---

11. Conformance test list for third-party RSP servers

@rasd/testing ships rspConformance({ baseUrl, getAuthToken }) (Vitest); a server is RSP v1 conformant when all pass:

1. GET /v1/ping → 204 with Date, no auth.
2. POST /v1/devices twice with the same id → same device, policy present, serverTime within ±5 s.
3. Missing/invalid bearer → 401 body per §1.4; X-Rasd-Device from another org → 403 device_mismatch.
4. GET /v1/forms without since returns all assigned forms; the returned cursor yields nothing new; publishing yields exactly one delta item; deleting yields a tombstone.
5. Definition fetch: ETag = "<definitionHash>"; If-None-Match → 304; body hash matches the manifest; when X-Rasd-Definition-Signature is present it verifies against the org's JWKS.
6. Dataset delta: paging with hasMore, tombstone appears, 410 cursor_expired after re-import.
7. Batch of 3 valid submissions → 3 accepted with serverRev; replay with the same Idempotency-Key → identical body + Idempotency-Replayed: true; same key, different body → 422.
8. Re-send under a new key → duplicate; same id, different checksum → conflict; unknown version → accepted, quarantined: true; type-invalid data → rejected with field; constraint failure → accepted with issues[]; closed form → rejected form_closed.
9. 51 items → 400 batch_too_large; > 5 MiB → 400 payload_too_large.
10. tus: OPTIONS advertises the required extensions; creation → 201 + Location + Upload-Expires; re-creation with the same attachmentId → existing Location; HEAD offset; two PATCH chunks with Upload-Checksum; wrong offset → 409; bad chunk checksum → 460; whole-file mismatch → 412 and upload removed; completion marks the submission complete; an attachment uploaded before its submission still links.
11. PATCH after token rotation (new bearer, same device) continues.
12. Records: older HLC → superseded; conflict payload includes mine, theirs, current, queueId; pull reflects rev.
13. 429 carries Retry-After; RateLimit-* fields present.
14. X-Rasd-Wipe appears only on 2xx and repeats the nonce until acknowledged (wipeAck in POST /v1/devices).
15. Revoked device → 403 device_revoked on every route except /v1/ping.
16. SSE (if features.sse): formPublished within 5 s of publish; Last-Event-ID resumes; heartbeat ≤ 30 s.
17. All error bodies match the §1.4 shape (validated against the error-body JSON Schema shipped with rspConformance in @rasd/testing); all timestamps are UTC ISO-8601.

---

12. Acceptance criteria

• @rasd/sync passes the engine suite in @rasd/testing (fake storage + fault-injecting transport): every row of the §4.5 retry table, backoff schedule with seeded RNG, lease expiry, dead-lettering, single-flight, leader election with two tabs (Playwright), and metered deferral.
• Killing the app/tab at any point (between finalize and first batch, mid-batch, mid-chunk, mid-page-apply) loses no data and produces no server duplicates (Playwright offline + Maestro; assertions on server row counts).
• A 5 MB photo uploads over a throttled 3G profile with two forced disconnects and one app restart, re-sending ≤ 1 chunk (measured via bytesSent).
• 401 mid-upload triggers exactly one token refresh and the upload continues from the server offset.
• Post-outage storm test: 1 000 simulated devices reconnecting within 10 s against the reference server → no 5xx, all batches accepted within 5 min, p95 batch latency < 500 ms.
• Reference server passes §11 and the OWASP ASVS 5.0 API checks in 16 · Security; RLS prevents cross-org reads in an automated test.
• Quarantine, issues[], conflict and remote wipe are visible end-to-end in apps/playground-web and apps/example-expo; useSync() always shows last sync and pending counts within 250 ms of a change.
• Retention: synced submissions and their blobs are gone 7 days after ack (fake clock), Sent metadata remains, drafts untouched; storage.export() still lists Sent metadata.
• Bundle: @rasd/sync (engine + rsp transport incl. minimal tus client) ≤ 18 kB min+gzip; form-runner total stays ≤ 120 kB.
• Sync log never contains answers, bearer tokens or full attachment URLs (grep test over 10 k log lines from the fault-injection suite).
• Every row of the failure-mode table (§13) has a fault-injection test that asserts the stated device behaviour, the stated RasdError.code and no data loss.
• Sync chrome is axe-clean in en and ar: status messages are polite live regions, no focus steal during a run, the retry countdown is announced at most once per 10 s, and every string resolves from the locale catalogue (§14, 13 §11).
• Performance (§14): a 50-item batch serialises in < 150 ms on a 2019 mid-range Android, blob hashing never blocks the main thread for > 16 ms, and status() costs no table scan.
• The openrosa transport (phase 3) passes the recorded-fixture contract tests for formList/HEAD/POST 201/202/409/413 paths.

---

13. Failure modes and how each surfaces

Consolidated view of what breaks, how the engine notices, and what the enumerator sees. §4.5 gives the retry mechanics; this table gives the outcomes. The invariant behind every row: a failure ends in retry, reject-with-reasons-and-keep-local, or server-side quarantine — never in a silent drop.

#	Failure	Detected by	Device behaviour	Enumerator sees	Recovery
F1	Backend unreachable, DNS/TLS failure, captive portal	GET /v1/ping ≠ 204 (2 s timeout, §4.4)	state offline; no requests; queues untouched	"Offline · N to send", last sync time	Automatic on the next online/foreground/probe trigger
F2	Token expired mid-run	401 on any route	one forced getAuthToken({ forceRefresh: true }), same request retried; second 401 ⇒ authRequired	"Sign in to keep syncing"	Host re-authenticates, calls resume(); tus uploads continue from the HEAD offset (§4.6)
F3	Device revoked by an admin	403 device_revoked	state revoked, RASD_SYNC_REVOKED; all local data retained	"This device was disabled — data is kept and can be exported"	Admin re-activates + resume(); storage.export() works throughout (P6)
F4	Data rejected by server validation	item verdict rejected (§2.6, §3)	submission sending → rejected, RASD_SYNC_REJECTED{ field }, op acked	Item in Needs attention with the offending question	Enumerator fixes and re-finalizes (same id, new clientRev)
F5	Unknown form version on the server	§3 step 3	accepted, quarantined: true; nothing to do on the device	Item marked "held for review"	Admin admits the row against a version in /v1/quarantine
F6	Same id, different checksum already on the server	409 / verdict conflict	sending → conflict, RASD_SYNC_CONFLICT; local copy never overwritten	Item in the supervisor list	Supervisor resolves (§5); server row is immutable
F7	Local blob corrupt (whole-file checksum fails twice)	412 on final PATCH + local re-hash	attachment failed, RASD_ATTACHMENT_FAILED; the submission stays synced but complete: false	"Photo could not be sent — retake"	Retake the media; a new attachmentId is queued
F8	Poison outbox op (client-side throw, 415, unfixable 422)	deadAfterAttempts (5) or terminal envelope error	op → outbox dead; the queue keeps moving	Count in Needs attention; item never blocks others	sync.retry(ids) or storage.export() (§4.7)
F9	Storage quota exhausted while applying a pulled page	RASD_STORAGE_QUOTA from the page transaction	transaction rolls back, cursor not advanced, phase aborts	"Device storage full — free space to receive forms"	Purge (§8), drop an unused dataset, or grant persistence; the same page re-applies
F10	Cursor older than the tombstone window	410 cursor_expired	cursor cleared, replacement resync of that collection only	Brief "Updating reference data"	Automatic; local rows absent from the full pull are dropped
F11	Definition hash or signature mismatch	client re-hash after §2.4	definition discarded, not stored, logged; the form is simply not offered	Form missing from Start new	Automatic retry next pull; drafts keep their pinned version
F12	App/tab killed mid-run	lease expiry + sending rows at start (§4.8)	leases expire, sending → queued, tus resumes from HEAD	Nothing (progress bar restarts at the server offset)	Idempotency turns any lost response into duplicate
F13	Two tabs racing	Web Locks (§4.2)	follower stays idle and mirrors status; even if locks were unavailable, Idempotency-Key + id dedupe absorb the race	One consistent status across tabs	Automatic
F14	SSE connection dropped or unsupported	stream error / no EventSource-capable fetch	falls back to the pullIntervalMs timer; no data depends on SSE	Slightly later form/dataset updates	Automatic reconnect with Last-Event-ID
F15	Post-outage sync storm	429 + Retry-After (§1.6, §9.6)	backoff replaced by Retry-After, spread by full jitter	"Retrying in ~N s"	Automatic; server-computed spread of 0–120 s
F16	Device clock badly wrong	HLC compare against Date/serverTime (§2.8)	HLC advances from server time, so a wrong clock cannot win forever; @rasd/license freezes on backwards time (00 §9)	Nothing	Automatic

14. Accessibility, i18n and performance of the sync surface

@rasd/sync is headless, so these are requirements on the status UI that hosts and the bundled renderer components build from SyncStatus/SyncProgress (§4.9), plus the engine-side costs that make that UI cheap.

Accessibility (WCAG 2.2 AA, mapped in 13 §11):

• Sync status is a status message, not an alert: the pending/sending counter lives in an aria-live="polite" region (accessibilityLiveRegion="polite" / announceForAccessibility on native). role="alert" is reserved for outcomes the enumerator must act on — a rejected item or authRequired.
• A sync run never moves focus and never blocks input; finalizing, editing a draft and navigating stay available while phase 1 runs (P1: nothing in the renderer awaits the network).
• Progress announcements are debounced to at most one per 2 s, and the backoff countdown is re-announced at most once per 10 s — render status().backoffUntil as a static string rather than a live-updating timer inside the live region, or screen readers become unusable during a long outage.
• Needs attention (rejected / conflict / dead) is an ordinary keyboard-reachable list with 48 px targets, not a modal: conflicts are supervisor work, not an interruption at the household door (§5).
• Colour is never the only channel for sync state — pair every chip with text and an icon (Sent, Pending, Needs attention).

Internationalisation:

• Every sync string comes from the chrome catalogues (@rasd/react/locales/<lc>, @rasd/native/locales/<lc>) through useLocale().t(key, vars) (13 §7); the engine itself emits no user-facing prose, only codes and numbers.
• Counts, byte sizes and times are formatted with Intl under the active locale, honouring settings.numbering and the locale calendar (00 §4.3a) — never string-concatenated digits, so Arabic-Indic digits and RTL bidi isolation work.
• Server error.message is localised via Accept-Language (§1.3), but it is a diagnostic: the UI keys its own message off the wire code, so an unlocalised server never leaks English into an Arabic field UI.
• RTL: progress bars, byte counters and the Sent list use logical properties and mirror with dir (13 §8).

Performance and battery:

• checksum is computed once at finalize() and stored on the submission (00 §6); retries and re-batching never re-canonicalise JSON. Attachment SHA-256 is computed at capture and verified by the store, not per attempt.
• Blob hashing and chunk slicing for anything > 1 MiB run off the main thread (Web Worker on web, native module on RN); the main thread must never be blocked > 16 ms by sync work.
• Batches are built from submissions.list({ select: 'summary' }) plus per-item reads, so a full outbox is never materialised in memory; adaptive chunk sizes (§4.4) bound tus memory to one chunk per upload (≤ 8 MiB, 2 in parallel).
• Radio discipline: no probe more often than probeIntervalMs, no timer-triggered run while offline, one SSE connection per device, and attachments deferred on metered links by default — the battery cost of sync is dominated by radio wake-ups, not CPU.
• status() is O(1) from maintained counters (submissions.countByStatus, 09 §3), never a table scan, so useSync() can re-render at UI cadence.
• Budget: @rasd/sync (engine + rsp transport incl. the minimal tus client) ≤ 18 kB min+gzip inside the ≤ 120 kB form-runner total (00 §12); the openrosa transport and the XForm serialiser are a separate lazy chunk.

---

Open questions

• Should quarantined submissions be reported to the device as accepted{quarantined:true} (chosen here: nothing is actionable on the device) or as a distinct verdict, so hosts can show a different status chip? Needs UX validation with an M&E team.
• Is a per-org sequence worth the complexity over a single global BIGSERIAL for cursors on very large tenants (hot sequence contention vs. simpler DDL)?
• SSE on React Native requires an SSE-capable fetch (streaming); do we ship a documented recipe (react-native-sse) or drop SSE from RN in v1?
• Records: is a 5-day review window the right default for humanitarian case work where devices sync monthly, or should the window be per form (settings.ext["dev.rasd.record"].reviewWindowDays)?
• Should the publish route (POST /v1/forms/{id}/versions) become part of RSP v1 so builders can target any conformant server, or stay a reference-server admin API (chosen here)?
• Web background sync is a wake-up accelerator only (§4.10, 11 §4): a headless service-worker drain would need a bearer token inside the worker bundle, which we refuse. Is the resulting behaviour on a phone whose browser is closed for days acceptable to M&E teams, or do we need an opt-in swSync escape hatch with a documented threat model?
• Client-side envelope encryption of submissions (ODK-style RSA/AES) would make server-side validation impossible; decide whether RSP v1.1 adds an encrypted batch mode that skips steps 4–5 of §3.

Related documents

00 · Decisions & conventions · 03 · Architecture · 04 · Form schema spec · 05 · Logic & expressions · 06 · Renderer (React) · 07 · Renderer (native) · 09 · Offline storage · 11 · PWA & embedding · 13 · i18n, RTL & accessibility · 14 · Media & field capture · 15 · Licensing & billing · 16 · Security & data protection · 17 · API reference · 18 · Engineering practices · 20 · Interoperability · 21 · Getting started

Research: research/04 · Offline storage & sync · research/05 · PWA & embedding · research/11 · Security threat model · research/12 · Form versioning · research/14 · ODK/Kobo interop