Skip to main content

14 — Interoperating with ODK Central, KoboToolbox, Ona/WFP MoDa and OpenRosa back-ends; XLSForm/XPath → Rasd mapping fidelity

Research note for Rasd Forms (gap #4). Accessed 2026-08-15. Web search quota was exhausted for this session, so every fact below was taken directly from primary pages (official docs, source code on GitHub, npm/PyPI registries) fetched on 2026-08-15; items that could not be verified are flagged [unverified].

Summary

  1. All four targets speak OpenRosa 1.0 (formList / manifest / multipart submission), and it is the only write path that works everywhere: ODK Central (/v1/projects/{id}/submission or /v1/key/{token}/…), Kobo (https://kc.kobotoolbox.org/{username}/submission, /submission, new /collector/{token}/submission), Ona/MoDa (/{username}/submission, /projects/{id}/submission) [2][4][13][17]. Send X-OpenRosa-Version: 1.0, multipart with a xml_submission_file part, resend the XML on every chunk, treat 201/202 as success, 409 as "instanceID reused with different XML" [2][4][13].
  2. JSON submissions are accepted only by the formhub family (Ona and Kobo/kc): body {"id": "<id_string>", "submission": {..., "meta": {"instanceID": "uuid:…"}}}; the server converts it to XML (dict2xform, lists joined into space-separated strings). ODK Central is XML-only (POST …/forms/{xmlFormId}/submissions with Content-Type: text/xml) [5][13][17]. Rasd therefore needs one canonical RFD-JSON → XForm-instance serializer; the JSON path is an optimisation, not a substitute.
  3. Central's REST layer is the richest (v2026.2, changelog fetched): app-user tokens in the URL path, XLSX upload with X-XlsForm-FormId-Fallback, draft/publish, PUT edits keyed by deprecatedID, per-attachment POST, entities (POST/PATCH …/datasets/{name}/entities, all values strings, baseVersion, branchId), entities.csv with ETag, OData 4.0 "minimal" for Power BI/Tableau, review states, comments, versions/diffs, 100 MB default body [4][5][7][8][9][10][11].
  4. Kobo (kpi main, 2026) now ships "Data Collectors": token-in-path OpenRosa endpoints /collector/{token}/formList|xformManifest/{id}|submission with add-submissions-only permission — the equivalent of Central app users; [unverified whether it is live on kf/eu yet] [13][14]. kpi v2 (/api/v2/assets/{uid}/data/) is read/validate/edit only; default page 100, MAX_API_PAGE_SIZE 1000; X-OpenRosa-Accept-Content-Length defaults to 10,000,000 bytes [15][16].
  5. Ona (MoDa) = same lineage as Kobo: Token/temp-token/Basic/Digest auth, POST /api/v1/forms (xls_file), form.json = pyxform JSON, POST /api/v1/submissions JSON or XML, edits with deprecatedID and per-form conflict strategy reject (409) or last_write_wins, GET /api/v1/data/{pk} with Mongo-style query, page_size ≤ 10,000 [17][18][19]. One "formhub-family" adapter with a host profile covers Kobo + Ona/MoDa.
  6. Instance serialization is fully specified by the ODK XForms spec: select_multiple/rank = space-separated names, geopoint = "lat lon alt acc", geotrace/geoshape = ;-separated geopoints (shape closed), date yyyy-mm-dd, time/dateTime carry a numeric TZ offset (never Z), binary = bare filename, meta/instanceID = uuid: + v4, deprecatedID for edits, entity block in meta for datasets, non-relevant nodes removed on submission [20][21].
  7. pyxform 4.5.0 (2026-06-25, BSD-2-Clause) is the reference XLSForm→XForm compiler; its --json CLI flag prints a status JSON (code 100/101/999), not the survey JSON — the survey dict is ConvertResult._pyxform/workbook_to_json() and documented by pyxform/json_form_schema.json [22][23]. There is no maintained JS XLSForm compiler: xform-to-json 1.2.1 (2017), xls2xform 1.1.0 (2016) are dead; enketo-transformer 4.2.0 (Dec 2024) only produces Enketo HTML/model; @getodk/xforms-engine 1.0.2 (2026-08-11) parses XForms XML, not XLSForm [24].
  8. XPath ≠ REL in four places that break real forms: (a) regex() is anchored Pattern.matches in JavaRosa/Collect but unanchored RegExp.test in Enketo and ODK Web Forms [25][26][27]; (b) boolean('false') = true() because node values are strings [20]; (c) empty numbers → NaN, dates coerce to decimal days; (d) node-set semantics (${q} inside sum()/count() from outside a repeat, current()/.., instance() predicates). Everything else is a mechanical rename.
  9. Export conventions differ by family: Kobo/Ona flatten with group/question, split geopoints into _q_latitude/_longitude/_altitude/_precision, select_multiple as q + q/choice 0/1, repeats as extra sheets with _index, _parent_table_name, _parent_index, system columns _id,_uuid,_submission_time,_validation_status,_notes,_status,_submitted_by,__version__,_tags [16][28][29]; Central uses group-question, KEY/PARENT_KEY, SubmissionDate, SubmitterID, ReviewState, …-Latitude/-Longitude/-Altitude/-Accuracy [30].
  10. Recommended order: Kobo (largest UN/NGO footprint, JSON write path, Data Collector tokens) → ODK Central (entities, OData, WFP/UNICEF growth, Web Forms) → Ona/MoDa (WFP; ~90 % shared with Kobo) → generic OpenRosa (SurveyCTO etc.). Ship a CI that round-trips ~140 public XLSForms through pyxform and compares canonical XML.

1. Back-end capability matrix

1.1 Protocol facts (verified)

OpenRosa 1.0 (approved Dec 2011) [1][2][3]

  • GET …/formList[?formID=&verbose=true&listAllVersions=&deviceID=]<xforms xmlns="http://openrosa.org/xforms/xformsList"><xform><formID/><name/><version/><hash>md5:…</hash><downloadUrl/><manifestUrl/></xform></xforms>; Accept-Language localises names [3].
  • Manifest → <manifest xmlns="http://openrosa.org/xforms/xformsManifest"><mediaFile [type="entityList"]><filename/><hash/><downloadUrl/><integrityUrl/></mediaFile></manifest> [3].
  • Submission: HEAD first → 204 + X-OpenRosa-Version + X-OpenRosa-Accept-Content-Length (bytes); then POST multipart/form-data with exactly one xml_submission_file part (text/xml) plus media parts named by filename; if larger than the accepted length, split into several POSTs each repeating the XML; success 201 or 202; 401/403/404/413/500; body <OpenRosaResponse xmlns="http://openrosa.org/http/response"><message/></OpenRosaResponse>; digest auth + https "without redirects" [2].

ODK Central (API v2026.2) [4]–[11]

  • Auth: POST /v1/sessions → Bearer token; Basic; cookie; app-user token in path /v1/key/{token}/projects/{id}/formList|submission|forms/{xmlFormId}/manifest; draft test tokens /v1/test/{token}/…/draft/…; app users can only read forms/create submissions; actor properties (v2026.2) [4][6].
  • Forms: POST /v1/projects/{id}/forms (XML, or XLSX with X-XlsForm-FormId-Fallback, ?publish=true, ?ignoreWarnings=true), GET …/forms/{xmlFormId}.xml|.xlsx, /versions, /draft, /draft/publish?version=, /attachments/{name} (PATCH {"dataset":true} links an entity list), /fields?odata=true, state open|closing|closed, hash = md5 of XML [11].
  • Submissions: OpenRosa POST /v1/projects/{id}/submission (multipart) or REST POST …/forms/{xmlFormId}/submissions (XML body only, ?deviceID=); attachments POST …/submissions/{instanceId}/attachments/{filename} (binary) ; identical instanceId + different XML → 409; identical XML re-POSTs allowed (that is how multi-POST works); edit = new XML with deprecatedID via PUT …/submissions/{instanceId} (rejected if deprecatedID ≠ latest version; media auto-copied); reviewState null|edited|hasIssues|rejected|approved; versions, diffs, comments; default max body 100 MB [4][5].
  • Entities: POST /v1/projects/{id}/datasets/{name}/entities ({uuid?, label, data:{…}} or {entities:[…], source:{…}}), PATCH …/entities/{uuid}?baseVersion=N[&force|resolve], soft delete/restore, entities.csv (ETag/304, $filter, $search), .svc OData; property values are always strings [8][9][10].
  • OData: /v1/projects/{id}/forms/{xmlFormId}.svcSubmissions + Submissions.{repeat} tables, __id, __system/submissionDate|submitterId|reviewState|updatedAt|deletedAt, $filter/$select/$expand=*/$top/$skip/$count/$orderby/$skiptoken/$wkt, JSON only, "4.0 Minimal Conformance … targeting Power BI and Tableau" [10].
  • Encryption: XLSForm public_key (RSA-2048); envelope <data xmlns="http://www.opendatakit.org/xforms/encrypted" encrypted="yes"> with base64EncryptedKey, encryptedXmlFile (submission.xml.enc), <media><file>*.enc, base64EncryptedElementSignature; AES-256-CFB/PKCS5, RSA-OAEP-SHA256-MGF1, IV = MD5(instanceID+key) with per-file counter; Central "managed encryption" decrypts on export (?keyId=passphrase) [5][31][32].
  • Rate limits: none documented in the API pages fetched [unverified].

KoboToolbox [12]–[16][28][29][33]

  • Hosts: kpi kf.kobotoolbox.org (global) / eu.kobotoolbox.org (EU); OpenRosa only on kc.kobotoolbox.org / kc-eu.kobotoolbox.org [12][13][33].
  • Auth: Authorization: Token <key> (/token/?format=json), Basic; Digest for Collect; new DataCollector tokens (secrets.token_urlsafe(20).lower(), groups of assets, permission = add_submissions only) used as /collector/{token}/formList, /collector/{token}/xformManifest/{id}, /collector/{token}/submission [13][14]. Anonymous: /{username}/formList|submission when "Allow submissions without username and password" is on [33].
  • kpi v2 (OpenAPI at /api/v2/schema/, docs /api/v2/docs/): /api/v2/assets/ (list/create/clone), /api/v2/assets/{uid}/ (?format=json|xml|xls|ssjson; content = Kobo's own survey JSON: survey[], choices[], settings, translations, translated, $kuid/$autoname/$xpath), asset_snapshots/ (XML/Enketo preview), deployment/, versions/, files/ (media), data/ (query, sort, fields, limit, start; JSON/XML/GeoJSON; default 100, max 1000), data/{id|rootUuid}/, data/{id}/validation_status (PATCH {"validation_status.uid": "validation_status_approved"} — also …_not_approved, …_on_hold [values partly unverified]), data/bulk, data/{id}/enketo/edit|view, data/{id}/duplicate, exports/, paired-data/, hooks/ [15][16][34].
  • Submissions: OpenRosa multipart or JSON ({"id": id_string, "submission": {…,"meta":{"instanceID":"uuid:…"}}}) on kc; 201; 202 Duplicate Instance for exact duplicates; 409 for same UUID with different content; X-OpenRosa-Accept-Content-Length default 10,000,000 [13][16]. Edits: Enketo edit URL from kpi, or resubmission with deprecatedID (formhub lineage) [direct API edit unverified].
  • Export: _id, _uuid, _submission_time, _submitted_by, _validation_status, _index, _status, _notes, _tags, __version__, _xform_id_string, _attachments, _geolocation, meta/instanceID, meta/rootUuid, formhub/uuid; repeats _index/_parent_table_name/_parent_index [28][29].
  • No entities; "dynamic data attachments" (paired-data) and select_one_from_file media instead. Encrypted forms accepted but not decryptable server-side (Briefcase) [31].

Ona Data / WFP MoDa (api.ona.io, api.moda.wfp.org [prior note 01]) [17]–[19]

  • Auth: Token, temporary token, Basic, Digest. Forms: POST /api/v1/forms (xls_file|xls_url|dropbox_xls_url, owner), GET /api/v1/forms/{pk}/form.json (pyxform JSON), form.xml, form.xls, PATCH to replace, /versions, media via /api/v1/metadata (data_type: media), csv_import, async exports (export_async?format=csv|xls|savzip|csvzip|kml|osm|gsheets), Enketo URLs, encrypted flag [18].
  • Submissions: POST /api/v1/submissions JSON {"id": id_string, "submission": {…}} (arrays allowed, e.g. ["ambulance","bicycle"]), multipart XML, FLOIP; meta.instanceID uuid: required; edits = new instanceID + deprecatedID; conflict strategy reject (409, default) or last_write_wins; 201 [17]. OpenRosa: /{username}/formList, /{username}/submission, /projects/{id}/submission [17][18].
  • Data: GET /api/v1/data/{pk} (query={"age":{"$gt":"21"}}, sort, fields, start/limit, page/page_size ≤ 10,000, .xml/.csv), system fields _id,_uuid,_submission_time,_xform_id_string,_attachments[{download_url,mimetype,filename}],_geolocation,_status,_tags,_notes,_version,_submitted_by,_date_modified,_media_all_received,_media_count,_total_media [19].

1.2 Matrix

CapabilityGeneric OpenRosaODK Central v2026.2KoboToolbox (kpi/kc 2026)Ona / MoDaRasd native (REST+tus)
AuthDigest/Basic (+https) [2]Bearer session, Basic, app-user token in URL, draft token [6]API Token, Basic, Digest (kc), Data-Collector token in URL, anonymous [12][14][33]Token, temp token, Basic, Digest [17]Bearer/API key + license token
Form pullformList + XForm XML + manifest [3]formList/XML/XLSX/versions/fields; app-user or Bearer [11]formList/XML on kc; kpi asset JSON, ?format=xls, snapshots [15][34]formList/XML on api; form.json/.xml/.xls, versions [18]RFD JSON, ETag
Media / datasetsmanifest mediaFile [3]attachments, entity lists as type="entityList" + integrityUrl, entities.csv ETag [3][8]manifest (media, paired-data CSV/XML) [16]manifest, /api/v1/metadata [18]tus/HTTP, hashed
Submission formatXML multipart, chunked by ACL [2]XML multipart or XML body + per-file POST [4][5]XML multipart or JSON (kc) [13]XML multipart or JSON [17]JSON + tus attachments
Size limitserver ACL header [2]100 MB default body [4]ACL 10,000,000 B default [16]ACL (Ona default 10 MB [unverified])tus (resumable)
Idempotency / duplicatesinstanceID; 201/202same instanceID+same XML ok, +different XML 409 [4][5]202 "Duplicate Instance", 409 conflict [13]201; 409 on edit conflict (reject) [17]client submissionId UUID
Edit after submitdeprecatedID [20]PUT new XML w/ deprecatedID; versions/diffs; reviewState [5]Enketo edit; deprecatedID resubmit [unverified]deprecatedID; last_write_wins option [17]versioned PATCH
Encryptionenvelope spec [32]managed keys, decrypt on export [5][31]accepted, opaqueencrypted flag [18]TLS + optional client envelope
Entities / casesfull (create/update/offline branchId) [8][9]none (paired-data only)noneRasd datasets (roadmap)
Form versioningversion attr; listAllVersions [3]draft/publish, must bump version [11]asset versions; deployment __version__ in data [28]/versions [18]RFD version + hash
Pull submissionsREST list, CSV zip, OData, GeoJSON [5][10]kpi data/ JSON/XML/GeoJSON, exports [15]/api/v1/data/{pk} JSON/CSV/XML [19]REST/OData
ImpossibleJSON, entities, edit UIJSON submissions; unknown attachment names → 400 [4]server-side decrypt; entities; write via kpi data/entities

2. Serialization spec — RFD submission JSON → XForm instance (and back)

Rule 0 (root and order come from the pulled XForm, never from RFD alone). Kobo roots are the asset uid, Ona/pyxform roots default to data or the settings name; Kobo/Ona forms also inject <formhub><uuid>…</uuid></formhub> as first child. @rasd/xlsform must record ext.xform.rootName, id, version, formhubUuid and the depth-first node order (/fields-style path list) at import and the serializer must emit exactly that order (Central computes expected attachments and OData tables from it) [5][11].

Envelope:

<{root} xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" id="{form_id}" version="{version}">
<formhub><uuid>{server form uuid}</uuid></formhub> <!-- only if present in pulled XForm -->
…survey nodes in form order…
<meta>
<instanceID>uuid:{v4}</instanceID>
<instanceName>{eval(instance_name)}</instanceName> <!-- optional -->
<deprecatedID>uuid:{previous}</deprecatedID> <!-- edits only -->
<entity dataset="{list}" id="{uuid}" create="1"|update="1" baseVersion="" trunkVersion="" branchId=""><label></label></entity>
</meta>
</{root}>

Meta children live in the orx namespace when the form declares them so; copy whatever the pulled form uses [20][21].

RFD type (JSON value)XForm bind typeInstance text (write)Read back (pull)
text (string)stringXML-escaped as isas is
integer (number)intString(Math.trunc(v)); null→emptyparseInt
decimal / range (number)decimal/intshortest round-trip, no exponent (0.0000001, not 1e-7)parseFloat
select_one (string)string/select1choice namestring
select_multiple (string[])string/selectnames joined by one space, in selection order; []→emptysplit(/\s+/)
rank (string[])odk:rankspace-separated (all options) [20]split
date ("YYYY-MM-DD")dateas isas is
timetimeHH:mm:ss.SSS±HH:mm (device offset) [20]keep offset
datetime (ISO)dateTimeYYYY-MM-DDTHH:mm:ss.SSS±HH:mm, not normalised to Z [20]keep offset
geopoint ({lat,lng,alt,accuracy})geopoint"lat lng alt acc"; missing alt/acc → 0.0 (4 tokens like Collect) [20]2–4 tokens
geotrace (point[])geotracepoints joined by ;split ;
geoshape (point[], closed)geoshapesame; ensure first == last [20]split
image/audio/video/file/signature/draw/annotate (attachment ref)binarybare filename, unique in the submission; part name = filename, Content-Type per file [2][20]Kobo/Ona _attachments[].download_url; Central …/attachments/{filename}
barcodebarcodestringstring
acknowledge (bool)stringOK / empty=== 'OK'
notestringempty element (keeps order)ignore
calculate / hiddenper bindstring of valuestring
RFD boolean (no XLSForm type)stringtrue/false (author-declared)boolean-from-string
group (RFD object)wrapper element only if the XForm has one; RFD "transparent" groups need ext.xform.wrap=falsenest
repeat (object[])N sibling elements named after the repeat; []→none; never emit jr:template [20]array
meta types start/end/today/deviceid/username/email/phonenumber/auditdateTime/date/string/binaryfrom Rasd device/user context; audit → audit.csv attachmentsystem columns
non-relevant elementsomitted per spec ("removed on submission") [20]; option emitEmpty for servers that want stable shapetreat missing = empty

Chunking: sort media by size, fill POSTs up to X-OpenRosa-Accept-Content-Length (10 MB Kobo, 100 MB Central), always include the XML part; on 413 halve the chunk; on Central REST use per-file POST instead [2][4][16].

Reverse (pulled data): Kobo/Ona JSON is flat with group/question keys and repeats as arrays of objects whose keys are full paths (rep/q); Central OData is nested by group with $expand=* for repeats and geodata as GeoJSON unless $wkt=true [10][19][28]. Map both into RFD values with the table above; keep server system fields under submission.server (_id/_uuid/_submission_time/_validation_status or __id/__system).


3. XLSForm → RFD mapping

3.1 Types [35]

XLSForm typeRFD typeNotes
text / integer / decimal / rangetext / integer / decimal / rangeparameters start,end,stepparams
select_one L / select_multiple Lselect_one / select_multiple (choices: 'L')or_otherallowOther:true (Kobo _or_other column) [29]
select_one_from_file f.csv/.xml/.geojson, select_multiple_from_filesame with choices: {source:'f.csv', value:'name', label:'label'}parameters value=,label=
select_one_externalselect_one + choiceFilter from itemsets.csv (pyxform emits it) [23]
rank Lranknative ordered array
notenote${} label templates preserved
geopoint / geotrace / geoshapesameparameters allow-mock-accuracy, capture-accuracy, warning-accuracy
date / time / dateTimedate / time / datetimeappearances month-year/year/no-calendar/ethiopian…
image / audio / video / file / background-audioimage/audio/video/file/background_audiomax-pixels, quality, appearances signature/draw/annotate/new/selfie
barcodebarcode
calculatecalculatecalculation → REL
acknowledgeacknowledge
hiddenhidden
xml-external / csv-externalform-level datasets[] (external instance)
begin_group/end_groupgroup (wrap:true)appearance field-list/table-list → layout
begin_repeat/end_repeatrepeat (repeatCount)
start,end,today,deviceid,phonenumber,username,email,auditmeta.* configaudit parameters (location-priority…)
trigger, background-geopoint, start-geopointtrigger element (odk:setvalue/odk:setgeopoint) [21]
entities sheetentity objectsee 3.4

3.2 Survey columns [35]

namename; label[::lang]label (i18n map, keep original header strings in ext.xlsform.languages); hint, guidance_hint; required (bool or expr) + required_message; constraint + constraint_messagevalidation:[{expr,message}]; relevantrelevant; calculationcalculate; defaultdefault (static; expressions become once(...) dynamic default); read_only; appearanceappearance:string[]; choice_filter→REL predicate; parameters (k=v pairs, space/comma) → params; repeat_count; media::image|audio|video[::lang]; trigger; save_toentity.saveTo; body::*, bind::*, instance::*, attribute::*verbatim ext.xlsform.body|bind|instance. Any unknown column → ext.xlsform.row[column].

3.3 Appearances → RFD widget hints (Collect ignores unknown appearances, so unmapped ones are safe to keep verbatim)

minimal, quick, quickcompact/quick_compact, compact, autocomplete, columns, columns-pack, columns-n, likert, label, list-nolabel, image-map, map, field-list, table-list, signature, draw, annotate, new, new-front, selfie, thousands-sep, numbers, multiline, url, masked, counter, printer, ex:*, bearing, vertical, distress, rating, no-calendar, month-year, year, ethiopian, coptic, islamic, bikram-sambat, myanmar, persian, placement-map, hidden-answer, no-buttons [35]. Rasd maps ~20 of these to first-class widget props (layout: 'grid'|'columns', variant:'minimal'|'autocomplete', calendar:'islamic', capture:'signature', mask, thousandsSep) and stores the raw list.

3.4 Settings, choices, entities, media

  • settings: form_title→title, form_id→id, version→version, instance_name→instanceName (REL), default_language, public_key→encryption.publicKey, submission_url→ext.xlsform.settings, style (pages→RFD pages; theme-grid→layout), allow_choice_duplicates, auto_send/auto_delete/client_editableext.collect, nameext.xform.rootName, namespaces/attribute::*→ext [35].
  • choices: list_name,name,label[::lang],media::image,geometry + arbitrary filter columns → choices[list] = [{name,label,media,attrs:{…}}]; allow_choice_duplicates [35].
  • entities: list_name(dataset), label, entity_id, create_if, update_if (+ entities:version 2022.1.0/2023.1.0/2024.1.0; offline metadata trunkVersion/branchId) → entity:{dataset,label,id,createIf,updateIf} [21]; a Rasd form that references an entity list gets datasets:[{name, kind:'entityList'}] and the adapter fetches entities.csv (ETag) [8].
  • media: jr://images|audio|video|file|file-csv/… → RFD media refs by filename; manifest resolves them [20].

3.5 Kobo extensions (from kpi source) [36][37][38][39]

Kobo constructKobo's own expansionRFD importExport
begin_scoreend_score, score__row, col kobo--score-choicesbegin_group appearance field-list + header row appearance label; each row select_one appearance list-nolabel [36]group{layout:'score-grid', ext.kobo.score:{choices}} with select_one childrenfold back to begin_score when target = Kobo, else expanded standard XLSForm
begin_rankend_rank, rank__level, cols kobo--rank-items, kobo--rank-constraint-messagebegin_group field-list + note; each level select_one (required, appearance minimal) with constraint ${level} != ${prev} and … [37]RFD rank (native array) + ext.kobo.rankKobo target: begin_rank; other: standard rank
begin_kobomatrixend_kobomatrix, col kobo--matrix_listheader group with w7-style widths, notes ##### {}; per list item a begin_group {matrix}_{item} with children {item}_{q}; appearances w2 horizontal-compact / w2 no-label [38]matrix element (rows = list, columns = child questions) storing values as {row:{col:val}}; serializer expands to {matrix}_{item}/{item}_{q} nodesfold back for Kobo
kobo--locking-profile (survey/settings), kobo--lock_all, sheet kobo--locking-profiles [39]n/aext.kobo.locking (profiles + restriction names)verbatim
select_one x or_other, _or_other column [29]pyxform or_otherallowOther_or_other for Kobo
asset-content keys $kuid,$autoname,$autovalue,$xpath,$given_name, translations:[null,…], translated:[label,hint] [29]n/aext.kobo.kuid etc.needed for kpi content PATCH round-trip

3.6 Lossless storage

Every RFD element carries ext: { xlsform: { row:{unmapped columns}, raw:{relevant,constraint,calculation,choice_filter,required,default}, rowIndex }, xform:{ path, rootName, bind:{…verbatim attrs} }, kobo:{…} }. Exporter rule: if the REL AST hash still equals the hash computed at import, re-emit ext.xlsform.raw.* verbatim; otherwise serialize REL→XPath. Form-level ext.xlsform.settings, ext.xlsform.extraSheets (osm, kobo--locking-profiles, external_choices) and column order are preserved so an unchanged import/export is byte-comparable in pyxform output.


4. XPath → REL translation rules and gap list

XPath / XLSFormRELRule
${q}${q}resolve in nearest scope; inside repeats = current instance
. / ... / .... = enclosing repeat/group instance
/data/g/q, ../q, ../../q${q} or path ref @/g/qresolve via ext.xform.path map; REL needs absolute-path refs
current()/../q (choice_filter, itemsets)${q} relative to current questionmost common in cascading selects inside repeats
instance('L')/root/item[name=${q}]/labelchoice-label('L', ${q}) (= jr:choice-name)
instance('csv')/root/item[k=${x}]/vlookup('csv','v','k',${x}) (alias pulldata)pulldata is defined as exactly this shortcut [20]
count(${rep}), sum(${rep_q}), max/min/join over repeatcount(${rep}), sum(${rep}.q)${q} inside an aggregate from outside the repeat = collection
position(..)position() (1-based)
indexed-repeat(${q}, ${r}, i[, ${r2}, j])same builtin (≤3 levels) [20]
selected/selected-at/count-selectedsame, on arraysKobo/Ona JSON path also joins arrays [13]
jr:itext('id')itext('id')
regex(., 'p')regex(., 'p', mode)JavaRosa = anchored Pattern.matches; Enketo & Web Forms = unanchored RegExp.test [25][26][27] → import lint: warn when pattern lacks ^…$; default mode:'full' for Collect parity
format-date(%Y %y %m %n %b %d %e %a), format-date-time(+%H %h %M %S %3)same tokens [20]
today(), now() (local, with offset), decimal-date-time, decimal-time, date()samedates in arithmetic → decimal days since epoch [20]
once(), uuid([n]), random(), randomize(nodeset[,seed]), `digest(s,'SHA-256'[, 'hex''base64'])`same
if, coalesce, boolean-from-string, checklist, weighted-checklist, int, round(n,d), pow, log10, math setsame
distance, area, geofence, intersectssame on RFD geo objects
substr (0-based), string-length (arg required), concat (node-sets ok), joinsame
coercionXPath rulesboolean('false')=true; ''→NaN in arithmetic; number-vs-string = compares numerically [20]

Not translatable (kept verbatim in ext, evaluated only in the optional Node @getodk/xpath import pipeline): XPath axes (ancestor::, following-sibling::), //, *, union |, name()/local-name()/lang()/id(), predicates with positional filters on the primary instance other than the indexed-repeat shape, count(/data/rep[q='yes'])-style filtered aggregates (recommend adding count-if(${rep}, q = 'yes') to REL), instance() node-sets fed to count()/sum(), dynamic jr:itext(concat(...)).

Coverage estimate (to be measured — see §6): on typical humanitarian XLSForms authored in Kobo/XLSForm style, ≥97 % of individual expressions are ${}-refs + the ~60 functions above and parse 1:1; 90–95 % of forms import with zero unmapped expressions; residual losses cluster in cascading-select choice_filters using current()/../, repeat aggregates, and hand-written /data/... paths. ODK's own JS engine reports 98 % XPath-function coverage and 91 % question-type coverage after two years — a realistic ceiling [prior note 09].

Round-trip loss classes: (1) expression text formatting (mitigated by raw re-emit); (2) Rasd-only widgets/themes/pages (Collect ignores unknown appearances; pages ↔ style: pages); (3) transparent groups vs XForm groups (instance nesting); (4) Rasd types with no XLSForm type (boolean, matrix, ratingselect_one + appearance, email/urltext + constraint); (5) language keys (label::Arabic (ar) vs ar); (6) Kobo/Ona-only columns; (7) numeric formatting (1.0 vs 1); (8) default vs once() normalisation; (9) media filename case; (10) entities sheet ordering.


5. Adapter architecture and roadmap

interface RasdTransport {
capabilities(): { write:'openrosa-xml'|'json'|'rasd', chunking:boolean, maxBody:number,
edit:'deprecatedID'|'put'|'none', pull:'odata'|'kpi'|'ona'|'rasd'|'none',
entities:boolean, encryption:'envelope'|'none', auth:('bearer'|'basic'|'digest'|'token-path'|'api-token')[] };
listForms(): Promise<FormRef[]>; // formList → {formID,name,version,hash,downloadUrl,manifestUrl}
pullForm(ref): Promise<{ xform:string, rfd:RFD, media:MediaRef[], datasets:DatasetRef[] }>;
pullDataset(ref, etag?): Promise<{ rows|csv, etag }>; // entities.csv / paired-data / manifest CSV
push(sub:RasdSubmission, o?): Promise<PushResult>; // serialize → HEAD → chunked POST(s) → 201/202
edit?(sub, previousInstanceId): Promise<PushResult>; // deprecatedID
pullSubmissions?(since): AsyncIterable<PulledSubmission>;
}

Profiles: openrosa (base: HEAD/ACL, multipart, digest/basic), central (adds /v1/key/{token} URL builder, REST attachments, OData/entities, review states), kobo (kc/kf host pair, Token auth, JSON fast path, validation_status, Data-Collector tokens), ona (/{username} paths, /api/v1/submissions, data/{pk}, last_write_wins). Also rasd (REST+tus).

Error → Rasd status model: 401/403auth_required (pause queue, keep local); 409 identical hash → sent; 409 different → conflict (needs user/edit); 413 → re-chunk, else too_large; 400 (Central: unknown attachment name / bad XML) → rejected (keep, surface); 404 form closed/deleted → blocked; 5xx/network → queued with exponential backoff + jitter; 202sent (accepted). Idempotency key = meta/instanceID; safe to retry on all four servers [4][13][17].

Two blockers to test early: CORS for browser PWAs hitting kc.kobotoolbox.org / api.ona.io / Central directly [unverified] — plan a relay transport (Rasd server proxies with the user's token) and Digest auth in React Native (implement RFC 7616 MD5 for generic OpenRosa; use Token/Bearer/URL-token everywhere else).

Roadmap: P1 Kobo (biggest UN/NGO base — UNHCR, IOM, OCHA, REACH; JSON write path; Data-Collector tokens; kpi pull) → P2 ODK Central (WFP/UNICEF adoption, entities, OData, Web Forms default in v2026.2) → P3 Ona/MoDa (WFP corporate; ~90 % code shared with Kobo profile) → P4 generic OpenRosa (SurveyCTO, self-hosted formhub) → P5 encryption envelope, offline entities (branchId), edits/reviewState sync.


6. Test corpus and CI plan

CorpusWhatLicence / URL
pyxform 4.5.0 tests/fixtures/example_forms (36 files: xlsform_spec_test.xlsx, widgets.xlsx, pull_data.xlsx, repeat_date_test.xlsx, choice_filter_test.xlsx, or_other.xlsx, attribute_columns_test.xlsx, flat_xlsform_test.xlsx, field-list.xlsx, loop.xlsx, group.xlsx, case_insensitivity, utf_csv.csv…) + bug_example_forms (9) + test_expected_output XML goldensXLSForm→XForm goldenBSD-2 — https://github.com/XLSForm/pyxform/tree/master/tests/fixtures [22][23]
pyxform unit tests (~90 files: test_repeat.py, test_rank.py, test_external_instances*.py, test_translations.py, test_unicode_rtl.py, test_settings.py, test_trigger.py, test_set_geopoint.py, entities/)markdown-table XLSForms embedded in tests — cheap to harvestsame
ODK Web Forms packages/common/src/fixtures/{computations,controls,date-and-time,entities,geolocation,groups,itext,notes,rank,repeats,select,upload,validation,value-types,xpath-fns,test-javarosa,test-scenario} + packages/scenario (JavaRosa-ported tests: initial 120 pass/193 fail/28 todo)engine conformance, XPath fn semanticsApache-2.0 — https://github.com/getodk/web-forms [40][41]
@getodk/xpath 1.0.0 test suiteXPath 1.0 + ODK function oracle for RELApache-2.0 [24]
JavaRosa src/test/resources (~45: template-repeat.xml, repeat-secondary-instance.xml, nigeria_wards_external*.xml, rank-form.xml, two-secondary-instances.xml, regression/, smoketests/)reference behaviour of CollectApache-2.0 — https://github.com/getodk/javarosa [42]
enketo-transformer test/forms (43 XForms: itemset.xml, rank.xml, setvalue*.xml, setgeopoint*.xml, external.xml, widgets.xml…) + enketo-core formsweb-form parityApache-2.0 — https://github.com/enketo/enketo [43]
ODK Collect test forms (collect_app/src/test/resources/forms) and ODK docs sample formswidgets/appearancesApache-2.0 [44]
XLSForm.org examples; Kobo public library/templates; humanitarian XLSForms published by REACH/IMPACT (MSNA tools), IOM DTM, UNHCR, WFP mVAM [URLs to be catalogued; check redistribution terms]realism: Arabic RTL, cascading admin selects, big repeatsmixed

CI: (1) pyxform round-trip — Python job runs pyxform 4.5.0 on each XLSForm → XForm A; Rasd imports XLSForm→RFD→exports XLSForm→pyxform → XForm B; compare canonicalised XML (sorted attrs, whitespace-normalised, ignore version); fail on any diff outside an allow-list. (2) Expression conformance — evaluate every corpus expression with @getodk/xpath (Node/jsdom) and REL over generated contexts; assert equality (with the documented regex/NaN caveats). (3) Submission goldens — fill fixture answers → RFD serializer → XML; validate against the pulled form's field list; nightly post to dockerised ODK Central (getodk/central) and kobo-install, read back via OData/kpi and diff. (4) Transport contract tests — recorded HTTP fixtures (msw) for formList/manifest/HEAD/POST/409/413/202 paths per profile. (5) Track a coverage dashboard: % expressions parsed, % forms lossless, per-function pass rate.


7. Export conventions for Rasd's own server

Adopt the Kobo/Ona family as default (most UN Excel/Power BI pipelines are built on it), with a Central-compatible switch:

  • Headers: group/question (/ separator; option - for Central mode); labels or XML names; multi-language labels [28].
  • select_multiple: q (space-separated names) and q/choice 0/1 columns (multiple_select='both'|'summary'|'details') [29].
  • geopoint: q, _q_latitude, _q_longitude, _q_altitude, _q_precision (Central mode: q-Latitude/-Longitude/-Altitude/-Accuracy) [29][30].
  • repeats: one sheet/CSV per repeat with _index, _parent_table_name, _parent_index (+ Kobo's _submission__uuid-style copies [naming unverified]); Central mode: KEY (uuid:…/rep[1]), PARENT_KEY [29][30].
  • system columns: _id, _uuid, _submission_time, _validation_status, _notes, _status, _submitted_by, __version__, _tags, _index (+ _xform_id_string, _attachments, _geolocation in JSON) [28]; Central mode: SubmissionDate, KEY, SubmitterID, SubmitterName, AttachmentsPresent, AttachmentsExpected, Status, ReviewState, DeviceID, Edits, FormVersion [30].
  • Media: filename column + {q}_URL when include_media_url [29].
  • Also expose an OData 4.0 minimal feed shaped like Central's (Submissions, Submissions.{repeat}, __id, __system, $expand=*, $skiptoken) so existing Power BI templates for Central work unchanged [10].

8. Implications & recommendations for Rasd Forms

  1. Build one serializer (@rasd/xform-io): RFD → XForm instance XML per §2, driven by the pulled XForm's node order/root/formhub block; reverse parser for Kobo/Ona JSON and Central OData. This is the load-bearing piece for all three servers.
  2. Ship @rasd/adapter-openrosa first (HEAD/ACL, chunked multipart, X-OpenRosa-Version: 1.0, 201/202/409/413 handling), then thin profiles kobo, central, ona on top; keep the JSON write path for Kobo/Ona as an opt-in fast path.
  3. Support token-in-URL enrolment (Central app users, Kobo Data Collectors, Central draft tokens) and Collect-style QR settings import (zlib+base64 JSON with general.server_url), so field teams keep their existing onboarding [6][14][45].
  4. Treat meta/instanceID as the universal idempotency key; retries are safe on all servers; map 409 with identical local hash to sent.
  5. Implement Central specifics: per-file attachment POST, PUT edits with deprecatedID, reviewState read-back, entities create/update (string values only, baseVersion, offline branchId), OData pull with $skiptoken, ETag on entities.csv [5][8][9][10].
  6. In @rasd/xlsform: import every column/sheet with ext passthrough (§3.6); implement Kobo begin_score/rank/kobomatrix folding both ways; keep language header strings; record rootName/path per element.
  7. REL: add absolute-path refs, collection refs for repeat aggregates, lookup()/choice-label() sugar, count-if, XPath coercion mode (empty→NaN, string booleans), and a regex anchoring switch with an import lint. Publish a conformance table against @getodk/xpath.
  8. Do not attempt a JS XLSForm→XForm compiler; call pyxform (Python) in the Node import pipeline/CI and cache results; use @getodk/xpath only server-side for unmappable expressions.
  9. Provide a relay transport and verify CORS/Digest early; PWAs may need the Rasd server (or the org's proxy) between browser and Kobo/Ona/Central.
  10. Exports: Kobo-family default, Central-mode switch, Central-shaped OData feed (§7).
  11. Encryption: implement the ODK envelope (AES-256-CFB + RSA-OAEP) as an optional client feature for Central managed encryption; document that Kobo cannot decrypt.
  12. CI as in §6, with a public coverage dashboard — this is also a sales artefact for UN procurement ("imports N of M REACH/IOM/UNHCR forms losslessly").

Unverified / to re-check

Kobo Data Collectors live status on kf/eu and its UI; Kobo direct-API edit via deprecatedID; Kobo validation_status.uid value list; Ona ACL default; CORS headers on kc/api.ona/Central; Central API rate limiting; ODK Collect QR JSON key values (form_update_mode, autosend enumerations); Kobo repeat-sheet _submission__* column names; licences/URLs of the humanitarian XLSForm sets.

Sources (accessed 2026-08-15)

  1. ODK Docs — OpenRosa protocol overview. https://docs.getodk.org/openrosa/
  2. ODK Docs — OpenRosa Form Submission API. https://docs.getodk.org/openrosa-form-submission/
  3. ODK Docs — OpenRosa Form List API. https://docs.getodk.org/openrosa-form-list/
  4. ODK Central API — OpenRosa endpoints. https://docs.getodk.org/central-api-openrosa-endpoints/
  5. ODK Central API — Submission management. https://docs.getodk.org/central-api-submission-management/
  6. ODK Central API — Accounts, users, app users. https://docs.getodk.org/central-api-accounts-and-users/
  7. ODK Central API — Changelog (v2026.2 newest). https://docs.getodk.org/central-api-changelog/
  8. ODK Central API — Dataset management. https://docs.getodk.org/central-api-dataset-management/
  9. ODK Central API — Entity management. https://docs.getodk.org/central-api-entity-management/
  10. ODK Central API — OData endpoints. https://docs.getodk.org/central-api-odata-endpoints/
  11. ODK Central API — Form management. https://docs.getodk.org/central-api-form-management/
  12. KoboToolbox — Using the API (kf/eu hosts, token). https://support.kobotoolbox.org/api.html
  13. kpi source — OpenRosa submission viewset & docs (XML/JSON, 201/202/409, /collector/{token}/submission). https://github.com/kobotoolbox/kpi/blob/main/kobo/apps/openrosa/apps/api/viewsets/xform_submission_api.py and …/kobo/apps/openrosa/docs/api/openrosa/submission/{authenticated,anonymous,data_collector}.md
  14. kpi source — Data Collectors (models, token auth) and formList viewset. https://github.com/kobotoolbox/kpi/tree/main/kobo/apps/data_collectors ; …/kobo/apps/openrosa/apps/api/viewsets/xform_list_api.py
  15. kpi source — /api/v2/assets/{uid}/data/ viewset docstrings. https://github.com/kobotoolbox/kpi/blob/main/kpi/views/v2/data.py
  16. kpi source — settings (OPENROSA_DEFAULT_CONTENT_LENGTH, DEFAULT_API_PAGE_SIZE, MAX_API_PAGE_SIZE) and paginators. https://github.com/kobotoolbox/kpi/blob/main/kobo/settings/base.py ; …/kpi/paginators.py
  17. Ona API — Submissions. https://api.ona.io/static/docs/submissions.html
  18. Ona API — Forms. https://api.ona.io/static/docs/forms.html
  19. Ona API — Data. https://api.ona.io/static/docs/data.html
  20. ODK XForms Specification (types, meta, functions, coercion). https://getodk.github.io/xforms-spec/
  21. ODK XForms Specification — Entities. https://getodk.github.io/xforms-spec/entities
  22. PyPI — pyxform 4.5.0 (2026-06-25). https://pypi.org/project/pyxform/
  23. pyxform source — xls2xform.py (--json = status JSON, ConvertResult), xform2json.py, json_form_schema.json, tests. https://github.com/XLSForm/pyxform
  24. npm registry — enketo-transformer 4.2.0, enketo-core 9.0.1, openrosa-xpath-evaluator 3.2.0, @getodk/xforms-engine 1.0.2, @getodk/xpath 1.0.0, xform-to-json 1.2.1, xls2xform 1.1.0. https://registry.npmjs.org/
  25. JavaRosa — XPathFuncExpr.regex() uses Pattern.matches. https://github.com/getodk/javarosa/blob/master/src/main/java/org/javarosa/xpath/expr/XPathFuncExpr.java
  26. Enketo — openrosa-xpath-evaluator regex() uses new RegExp().test(). https://github.com/enketo/enketo/blob/main/packages/openrosa-xpath-evaluator/src/openrosa-extensions.js
  27. ODK Web Forms — @getodk/xpath regex(). https://github.com/getodk/web-forms/blob/main/packages/xpath/src/functions/xforms/string.ts
  28. kpi source — common_tags.py (submission metadata field names). https://github.com/kobotoolbox/kpi/blob/main/kobo/apps/openrosa/libs/utils/common_tags.py
  29. kobotoolbox/formpack — export options (group_sep, multiple_select, _index/_parent_*, _latitude…_precision, _URL, _or_other, translations). https://github.com/kobotoolbox/formpack
  30. ODK Docs — Central submissions/CSV export columns. https://docs.getodk.org/central-submissions/
  31. ODK Docs — Encrypted forms. https://docs.getodk.org/encrypted-forms/
  32. ODK XForms Specification — Encryption. https://getodk.github.io/xforms-spec/encryption.html
  33. KoboToolbox — KoboCollect on Android (kc / kc-eu URLs, QR, anonymous). https://support.kobotoolbox.org/kobocollect_on_android_latest.html
  34. KoboToolbox — OpenAPI schema. https://kf.kobotoolbox.org/api/v2/schema/?format=json
  35. XLSForm.org — reference (types, columns, settings, appearances, parameters). https://xlsform.org/en/
  36. kpi — koboscore_handler.py. https://github.com/kobotoolbox/kpi/blob/main/kpi/utils/xlsform_preprocessors/koboscore_handler.py
  37. kpi — koborank_handler.py. https://github.com/kobotoolbox/kpi/blob/main/kpi/utils/xlsform_preprocessors/koborank_handler.py
  38. kpi — kobomatrix_handler.py. https://github.com/kobotoolbox/kpi/blob/main/kpi/utils/xlsform_preprocessors/kobomatrix_handler.py
  39. KoboToolbox — Library locking (kobo--locking-profile(s), kobo--lock_all). https://support.kobotoolbox.org/library_locking.html
  40. ODK Web Forms — fixtures. https://github.com/getodk/web-forms/tree/main/packages/common/src/fixtures
  41. ODK Web Forms — packages/scenario README (JavaRosa-ported suite). https://github.com/getodk/web-forms/blob/main/packages/scenario/README.md
  42. JavaRosa — test resources. https://github.com/getodk/javarosa/tree/master/src/test/resources
  43. Enketo — enketo-transformer test forms. https://github.com/enketo/enketo/tree/main/packages/enketo-transformer/test/forms
  44. ODK Collect — test forms. https://github.com/getodk/collect/tree/master/collect_app/src/test/resources/forms
  45. ODK Docs — Collect settings import/export (QR JSON: general/admin/project, zlib+base64). https://docs.getodk.org/collect-import-export/
  46. Kobo — Export downloads (system columns). https://support.kobotoolbox.org/export_download.html
  47. Prior Rasd research notes 01 (MoDa = Ona at api.moda.wfp.org) and 09 (REL design, ODK Web Forms coverage). /Users/huthaifakhreshi/Downloads/metals-analyzer/rasd-forms/docs/research/01-un-field-data-collection-landscape.md ; …/09-field-features-and-expression-engine.md