14 — Interoperating with ODK Central, KoboToolbox, Ona/WFP MoDa and OpenRosa back-ends; XLSForm/XPath → Rasd mapping fidelity
Research note for Rasd Forms (gap #4). Accessed 2026-08-15. Web search quota was exhausted for this session, so every fact below was taken directly from primary pages (official docs, source code on GitHub, npm/PyPI registries) fetched on 2026-08-15; items that could not be verified are flagged [unverified].
Summary
- All four targets speak OpenRosa 1.0 (formList / manifest / multipart
submission), and it is the only write path that works everywhere: ODK Central (/v1/projects/{id}/submissionor/v1/key/{token}/…), Kobo (https://kc.kobotoolbox.org/{username}/submission,/submission, new/collector/{token}/submission), Ona/MoDa (/{username}/submission,/projects/{id}/submission) [2][4][13][17]. SendX-OpenRosa-Version: 1.0, multipart with axml_submission_filepart, resend the XML on every chunk, treat 201/202 as success, 409 as "instanceID reused with different XML" [2][4][13]. - JSON submissions are accepted only by the formhub family (Ona and Kobo/kc): body
{"id": "<id_string>", "submission": {..., "meta": {"instanceID": "uuid:…"}}}; the server converts it to XML (dict2xform, lists joined into space-separated strings). ODK Central is XML-only (POST …/forms/{xmlFormId}/submissionswithContent-Type: text/xml) [5][13][17]. Rasd therefore needs one canonical RFD-JSON → XForm-instance serializer; the JSON path is an optimisation, not a substitute. - Central's REST layer is the richest (v2026.2, changelog fetched): app-user tokens in the URL path, XLSX upload with
X-XlsForm-FormId-Fallback, draft/publish,PUTedits keyed bydeprecatedID, per-attachmentPOST, entities (POST/PATCH …/datasets/{name}/entities, all values strings,baseVersion,branchId),entities.csvwith ETag, OData 4.0 "minimal" for Power BI/Tableau, review states, comments, versions/diffs, 100 MB default body [4][5][7][8][9][10][11]. - Kobo (kpi
main, 2026) now ships "Data Collectors": token-in-path OpenRosa endpoints/collector/{token}/formList|xformManifest/{id}|submissionwith add-submissions-only permission — the equivalent of Central app users; [unverified whether it is live on kf/eu yet] [13][14]. kpi v2 (/api/v2/assets/{uid}/data/) is read/validate/edit only; default page 100,MAX_API_PAGE_SIZE1000;X-OpenRosa-Accept-Content-Lengthdefaults to 10,000,000 bytes [15][16]. - Ona (MoDa) = same lineage as Kobo: Token/temp-token/Basic/Digest auth,
POST /api/v1/forms(xls_file),form.json= pyxform JSON,POST /api/v1/submissionsJSON or XML, edits withdeprecatedIDand per-form conflict strategyreject(409) orlast_write_wins,GET /api/v1/data/{pk}with Mongo-stylequery,page_size≤ 10,000 [17][18][19]. One "formhub-family" adapter with a host profile covers Kobo + Ona/MoDa. - Instance serialization is fully specified by the ODK XForms spec:
select_multiple/rank= space-separated names, geopoint ="lat lon alt acc", geotrace/geoshape =;-separated geopoints (shape closed), dateyyyy-mm-dd, time/dateTime carry a numeric TZ offset (neverZ), binary = bare filename,meta/instanceID=uuid:+ v4,deprecatedIDfor edits,entityblock inmetafor datasets, non-relevant nodes removed on submission [20][21]. - pyxform 4.5.0 (2026-06-25, BSD-2-Clause) is the reference XLSForm→XForm compiler; its
--jsonCLI flag prints a status JSON (code100/101/999), not the survey JSON — the survey dict isConvertResult._pyxform/workbook_to_json()and documented bypyxform/json_form_schema.json[22][23]. There is no maintained JS XLSForm compiler:xform-to-json1.2.1 (2017),xls2xform1.1.0 (2016) are dead;enketo-transformer4.2.0 (Dec 2024) only produces Enketo HTML/model;@getodk/xforms-engine1.0.2 (2026-08-11) parses XForms XML, not XLSForm [24]. - XPath ≠ REL in four places that break real forms: (a)
regex()is anchoredPattern.matchesin JavaRosa/Collect but unanchoredRegExp.testin Enketo and ODK Web Forms [25][26][27]; (b)boolean('false') = true()because node values are strings [20]; (c) empty numbers →NaN, dates coerce to decimal days; (d) node-set semantics (${q}insidesum()/count()from outside a repeat,current()/..,instance()predicates). Everything else is a mechanical rename. - Export conventions differ by family: Kobo/Ona flatten with
group/question, split geopoints into_q_latitude/_longitude/_altitude/_precision, select_multiple asq+q/choice0/1, repeats as extra sheets with_index,_parent_table_name,_parent_index, system columns_id,_uuid,_submission_time,_validation_status,_notes,_status,_submitted_by,__version__,_tags[16][28][29]; Central usesgroup-question,KEY/PARENT_KEY,SubmissionDate,SubmitterID,ReviewState,…-Latitude/-Longitude/-Altitude/-Accuracy[30]. - Recommended order: Kobo (largest UN/NGO footprint, JSON write path, Data Collector tokens) → ODK Central (entities, OData, WFP/UNICEF growth, Web Forms) → Ona/MoDa (WFP; ~90 % shared with Kobo) → generic OpenRosa (SurveyCTO etc.). Ship a CI that round-trips ~140 public XLSForms through pyxform and compares canonical XML.
1. Back-end capability matrix
1.1 Protocol facts (verified)
OpenRosa 1.0 (approved Dec 2011) [1][2][3]
GET …/formList[?formID=&verbose=true&listAllVersions=&deviceID=]→<xforms xmlns="http://openrosa.org/xforms/xformsList"><xform><formID/><name/><version/><hash>md5:…</hash><downloadUrl/><manifestUrl/></xform></xforms>;Accept-Languagelocalises names [3].- Manifest →
<manifest xmlns="http://openrosa.org/xforms/xformsManifest"><mediaFile [type="entityList"]><filename/><hash/><downloadUrl/><integrityUrl/></mediaFile></manifest>[3]. - Submission:
HEADfirst →204+X-OpenRosa-Version+X-OpenRosa-Accept-Content-Length(bytes); thenPOST multipart/form-datawith exactly onexml_submission_filepart (text/xml) plus media parts named by filename; if larger than the accepted length, split into several POSTs each repeating the XML; success201or202;401/403/404/413/500; body<OpenRosaResponse xmlns="http://openrosa.org/http/response"><message/></OpenRosaResponse>; digest auth + https "without redirects" [2].
ODK Central (API v2026.2) [4]–[11]
- Auth:
POST /v1/sessions→ Bearer token; Basic; cookie; app-user token in path/v1/key/{token}/projects/{id}/formList|submission|forms/{xmlFormId}/manifest; draft test tokens/v1/test/{token}/…/draft/…; app users can only read forms/create submissions; actor properties (v2026.2) [4][6]. - Forms:
POST /v1/projects/{id}/forms(XML, or XLSX withX-XlsForm-FormId-Fallback,?publish=true,?ignoreWarnings=true),GET …/forms/{xmlFormId}.xml|.xlsx,/versions,/draft,/draft/publish?version=,/attachments/{name}(PATCH {"dataset":true}links an entity list),/fields?odata=true, stateopen|closing|closed,hash= md5 of XML [11]. - Submissions: OpenRosa
POST /v1/projects/{id}/submission(multipart) or RESTPOST …/forms/{xmlFormId}/submissions(XML body only,?deviceID=); attachmentsPOST …/submissions/{instanceId}/attachments/{filename}(binary) ; identicalinstanceId+ different XML →409; identical XML re-POSTs allowed (that is how multi-POST works); edit = new XML withdeprecatedIDviaPUT …/submissions/{instanceId}(rejected ifdeprecatedID≠ latest version; media auto-copied);reviewStatenull|edited|hasIssues|rejected|approved; versions, diffs, comments; default max body 100 MB [4][5]. - Entities:
POST /v1/projects/{id}/datasets/{name}/entities({uuid?, label, data:{…}}or{entities:[…], source:{…}}),PATCH …/entities/{uuid}?baseVersion=N[&force|resolve], soft delete/restore,entities.csv(ETag/304,$filter,$search),.svcOData; property values are always strings [8][9][10]. - OData:
/v1/projects/{id}/forms/{xmlFormId}.svc→Submissions+Submissions.{repeat}tables,__id,__system/submissionDate|submitterId|reviewState|updatedAt|deletedAt,$filter/$select/$expand=*/$top/$skip/$count/$orderby/$skiptoken/$wkt, JSON only, "4.0 Minimal Conformance … targeting Power BI and Tableau" [10]. - Encryption: XLSForm
public_key(RSA-2048); envelope<data xmlns="http://www.opendatakit.org/xforms/encrypted" encrypted="yes">withbase64EncryptedKey,encryptedXmlFile(submission.xml.enc),<media><file>*.enc,base64EncryptedElementSignature; AES-256-CFB/PKCS5, RSA-OAEP-SHA256-MGF1, IV = MD5(instanceID+key) with per-file counter; Central "managed encryption" decrypts on export (?keyId=passphrase) [5][31][32]. - Rate limits: none documented in the API pages fetched [unverified].
KoboToolbox [12]–[16][28][29][33]
- Hosts: kpi
kf.kobotoolbox.org(global) /eu.kobotoolbox.org(EU); OpenRosa only onkc.kobotoolbox.org/kc-eu.kobotoolbox.org[12][13][33]. - Auth:
Authorization: Token <key>(/token/?format=json), Basic; Digest for Collect; newDataCollectortokens (secrets.token_urlsafe(20).lower(), groups of assets, permission =add_submissionsonly) used as/collector/{token}/formList,/collector/{token}/xformManifest/{id},/collector/{token}/submission[13][14]. Anonymous:/{username}/formList|submissionwhen "Allow submissions without username and password" is on [33]. - kpi v2 (OpenAPI at
/api/v2/schema/, docs/api/v2/docs/):/api/v2/assets/(list/create/clone),/api/v2/assets/{uid}/(?format=json|xml|xls|ssjson;content= Kobo's own survey JSON:survey[],choices[],settings,translations,translated,$kuid/$autoname/$xpath),asset_snapshots/(XML/Enketo preview),deployment/,versions/,files/(media),data/(query,sort,fields,limit,start; JSON/XML/GeoJSON; default 100, max 1000),data/{id|rootUuid}/,data/{id}/validation_status(PATCH {"validation_status.uid": "validation_status_approved"}— also…_not_approved,…_on_hold[values partly unverified]),data/bulk,data/{id}/enketo/edit|view,data/{id}/duplicate,exports/,paired-data/,hooks/[15][16][34]. - Submissions: OpenRosa multipart or JSON (
{"id": id_string, "submission": {…,"meta":{"instanceID":"uuid:…"}}}) on kc;201;202 Duplicate Instancefor exact duplicates;409for same UUID with different content;X-OpenRosa-Accept-Content-Lengthdefault 10,000,000 [13][16]. Edits: Enketo edit URL from kpi, or resubmission withdeprecatedID(formhub lineage) [direct API edit unverified]. - Export:
_id, _uuid, _submission_time, _submitted_by, _validation_status, _index, _status, _notes, _tags, __version__,_xform_id_string,_attachments,_geolocation,meta/instanceID,meta/rootUuid,formhub/uuid; repeats_index/_parent_table_name/_parent_index[28][29]. - No entities; "dynamic data attachments" (
paired-data) andselect_one_from_filemedia instead. Encrypted forms accepted but not decryptable server-side (Briefcase) [31].
Ona Data / WFP MoDa (api.ona.io, api.moda.wfp.org [prior note 01]) [17]–[19]
- Auth: Token, temporary token, Basic, Digest. Forms:
POST /api/v1/forms(xls_file|xls_url|dropbox_xls_url,owner),GET /api/v1/forms/{pk}/form.json(pyxform JSON),form.xml,form.xls,PATCHto replace,/versions, media via/api/v1/metadata(data_type: media),csv_import, async exports (export_async?format=csv|xls|savzip|csvzip|kml|osm|gsheets), Enketo URLs,encryptedflag [18]. - Submissions:
POST /api/v1/submissionsJSON{"id": id_string, "submission": {…}}(arrays allowed, e.g.["ambulance","bicycle"]), multipart XML, FLOIP;meta.instanceIDuuid:required; edits = newinstanceID+deprecatedID; conflict strategyreject(409, default) orlast_write_wins;201[17]. OpenRosa:/{username}/formList,/{username}/submission,/projects/{id}/submission[17][18]. - Data:
GET /api/v1/data/{pk}(query={"age":{"$gt":"21"}},sort,fields,start/limit,page/page_size≤ 10,000,.xml/.csv), system fields_id,_uuid,_submission_time,_xform_id_string,_attachments[{download_url,mimetype,filename}],_geolocation,_status,_tags,_notes,_version,_submitted_by,_date_modified,_media_all_received,_media_count,_total_media[19].
1.2 Matrix
| Capability | Generic OpenRosa | ODK Central v2026.2 | KoboToolbox (kpi/kc 2026) | Ona / MoDa | Rasd native (REST+tus) |
|---|---|---|---|---|---|
| Auth | Digest/Basic (+https) [2] | Bearer session, Basic, app-user token in URL, draft token [6] | API Token, Basic, Digest (kc), Data-Collector token in URL, anonymous [12][14][33] | Token, temp token, Basic, Digest [17] | Bearer/API key + license token |
| Form pull | formList + XForm XML + manifest [3] | formList/XML/XLSX/versions/fields; app-user or Bearer [11] | formList/XML on kc; kpi asset JSON, ?format=xls, snapshots [15][34] | formList/XML on api; form.json/.xml/.xls, versions [18] | RFD JSON, ETag |
| Media / datasets | manifest mediaFile [3] | attachments, entity lists as type="entityList" + integrityUrl, entities.csv ETag [3][8] | manifest (media, paired-data CSV/XML) [16] | manifest, /api/v1/metadata [18] | tus/HTTP, hashed |
| Submission format | XML multipart, chunked by ACL [2] | XML multipart or XML body + per-file POST [4][5] | XML multipart or JSON (kc) [13] | XML multipart or JSON [17] | JSON + tus attachments |
| Size limit | server ACL header [2] | 100 MB default body [4] | ACL 10,000,000 B default [16] | ACL (Ona default 10 MB [unverified]) | tus (resumable) |
| Idempotency / duplicates | instanceID; 201/202 | same instanceID+same XML ok, +different XML 409 [4][5] | 202 "Duplicate Instance", 409 conflict [13] | 201; 409 on edit conflict (reject) [17] | client submissionId UUID |
| Edit after submit | deprecatedID [20] | PUT new XML w/ deprecatedID; versions/diffs; reviewState [5] | Enketo edit; deprecatedID resubmit [unverified] | deprecatedID; last_write_wins option [17] | versioned PATCH |
| Encryption | envelope spec [32] | managed keys, decrypt on export [5][31] | accepted, opaque | encrypted flag [18] | TLS + optional client envelope |
| Entities / cases | — | full (create/update/offline branchId) [8][9] | none (paired-data only) | none | Rasd datasets (roadmap) |
| Form versioning | version attr; listAllVersions [3] | draft/publish, must bump version [11] | asset versions; deployment __version__ in data [28] | /versions [18] | RFD version + hash |
| Pull submissions | — | REST list, CSV zip, OData, GeoJSON [5][10] | kpi data/ JSON/XML/GeoJSON, exports [15] | /api/v1/data/{pk} JSON/CSV/XML [19] | REST/OData |
| Impossible | JSON, entities, edit UI | JSON submissions; unknown attachment names → 400 [4] | server-side decrypt; entities; write via kpi data/ | entities | — |
2. Serialization spec — RFD submission JSON → XForm instance (and back)
Rule 0 (root and order come from the pulled XForm, never from RFD alone). Kobo roots are the asset uid, Ona/pyxform roots default to data or the settings name; Kobo/Ona forms also inject <formhub><uuid>…</uuid></formhub> as first child. @rasd/xlsform must record ext.xform.rootName, id, version, formhubUuid and the depth-first node order (/fields-style path list) at import and the serializer must emit exactly that order (Central computes expected attachments and OData tables from it) [5][11].
Envelope:
<{root} xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" id="{form_id}" version="{version}">
<formhub><uuid>{server form uuid}</uuid></formhub> <!-- only if present in pulled XForm -->
…survey nodes in form order…
<meta>
<instanceID>uuid:{v4}</instanceID>
<instanceName>{eval(instance_name)}</instanceName> <!-- optional -->
<deprecatedID>uuid:{previous}</deprecatedID> <!-- edits only -->
<entity dataset="{list}" id="{uuid}" create="1"|update="1" baseVersion="" trunkVersion="" branchId=""><label>…</label></entity>
</meta>
</{root}>
Meta children live in the orx namespace when the form declares them so; copy whatever the pulled form uses [20][21].
| RFD type (JSON value) | XForm bind type | Instance text (write) | Read back (pull) |
|---|---|---|---|
text (string) | string | XML-escaped as is | as is |
integer (number) | int | String(Math.trunc(v)); null→empty | parseInt |
decimal / range (number) | decimal/int | shortest round-trip, no exponent (0.0000001, not 1e-7) | parseFloat |
select_one (string) | string/select1 | choice name | string |
select_multiple (string[]) | string/select | names joined by one space, in selection order; []→empty | split(/\s+/) |
rank (string[]) | odk:rank | space-separated (all options) [20] | split |
date ("YYYY-MM-DD") | date | as is | as is |
time | time | HH:mm:ss.SSS±HH:mm (device offset) [20] | keep offset |
datetime (ISO) | dateTime | YYYY-MM-DDTHH:mm:ss.SSS±HH:mm, not normalised to Z [20] | keep offset |
geopoint ({lat,lng,alt,accuracy}) | geopoint | "lat lng alt acc"; missing alt/acc → 0.0 (4 tokens like Collect) [20] | 2–4 tokens |
geotrace (point[]) | geotrace | points joined by ; | split ; |
geoshape (point[], closed) | geoshape | same; ensure first == last [20] | split |
image/audio/video/file/signature/draw/annotate (attachment ref) | binary | bare filename, unique in the submission; part name = filename, Content-Type per file [2][20] | Kobo/Ona _attachments[].download_url; Central …/attachments/{filename} |
barcode | barcode | string | string |
acknowledge (bool) | string | OK / empty | === 'OK' |
note | string | empty element (keeps order) | ignore |
calculate / hidden | per bind | string of value | string |
RFD boolean (no XLSForm type) | string | true/false (author-declared) | boolean-from-string |
group (RFD object) | — | wrapper element only if the XForm has one; RFD "transparent" groups need ext.xform.wrap=false | nest |
repeat (object[]) | — | N sibling elements named after the repeat; []→none; never emit jr:template [20] | array |
meta types start/end/today/deviceid/username/email/phonenumber/audit | dateTime/date/string/binary | from Rasd device/user context; audit → audit.csv attachment | system columns |
| non-relevant elements | — | omitted per spec ("removed on submission") [20]; option emitEmpty for servers that want stable shape | treat missing = empty |
Chunking: sort media by size, fill POSTs up to X-OpenRosa-Accept-Content-Length (10 MB Kobo, 100 MB Central), always include the XML part; on 413 halve the chunk; on Central REST use per-file POST instead [2][4][16].
Reverse (pulled data): Kobo/Ona JSON is flat with group/question keys and repeats as arrays of objects whose keys are full paths (rep/q); Central OData is nested by group with $expand=* for repeats and geodata as GeoJSON unless $wkt=true [10][19][28]. Map both into RFD values with the table above; keep server system fields under submission.server (_id/_uuid/_submission_time/_validation_status or __id/__system).
3. XLSForm → RFD mapping
3.1 Types [35]
| XLSForm type | RFD type | Notes |
|---|---|---|
| text / integer / decimal / range | text / integer / decimal / range | parameters start,end,step → params |
| select_one L / select_multiple L | select_one / select_multiple (choices: 'L') | or_other → allowOther:true (Kobo _or_other column) [29] |
| select_one_from_file f.csv/.xml/.geojson, select_multiple_from_file | same with choices: {source:'f.csv', value:'name', label:'label'} | parameters value=,label= |
| select_one_external | select_one + choiceFilter from itemsets.csv (pyxform emits it) [23] | |
| rank L | rank | native ordered array |
| note | note | ${} label templates preserved |
| geopoint / geotrace / geoshape | same | parameters allow-mock-accuracy, capture-accuracy, warning-accuracy |
| date / time / dateTime | date / time / datetime | appearances month-year/year/no-calendar/ethiopian… |
| image / audio / video / file / background-audio | image/audio/video/file/background_audio | max-pixels, quality, appearances signature/draw/annotate/new/selfie |
| barcode | barcode | |
| calculate | calculate | calculation → REL |
| acknowledge | acknowledge | |
| hidden | hidden | |
| xml-external / csv-external | form-level datasets[] (external instance) | |
| begin_group/end_group | group (wrap:true) | appearance field-list/table-list → layout |
| begin_repeat/end_repeat | repeat (repeatCount) | |
| start,end,today,deviceid,phonenumber,username,email,audit | meta.* config | audit parameters (location-priority…) |
| trigger, background-geopoint, start-geopoint | trigger element (odk:setvalue/odk:setgeopoint) [21] | |
| entities sheet | entity object | see 3.4 |
3.2 Survey columns [35]
name→name; label[::lang]→label (i18n map, keep original header strings in ext.xlsform.languages); hint, guidance_hint; required (bool or expr) + required_message; constraint + constraint_message → validation:[{expr,message}]; relevant→relevant; calculation→calculate; default→default (static; expressions become once(...) dynamic default); read_only; appearance→ appearance:string[]; choice_filter→REL predicate; parameters (k=v pairs, space/comma) → params; repeat_count; media::image|audio|video[::lang]; trigger; save_to→entity.saveTo; body::*, bind::*, instance::*, attribute::* → verbatim ext.xlsform.body|bind|instance. Any unknown column → ext.xlsform.row[column].
3.3 Appearances → RFD widget hints (Collect ignores unknown appearances, so unmapped ones are safe to keep verbatim)
minimal, quick, quickcompact/quick_compact, compact, autocomplete, columns, columns-pack, columns-n, likert, label, list-nolabel, image-map, map, field-list, table-list, signature, draw, annotate, new, new-front, selfie, thousands-sep, numbers, multiline, url, masked, counter, printer, ex:*, bearing, vertical, distress, rating, no-calendar, month-year, year, ethiopian, coptic, islamic, bikram-sambat, myanmar, persian, placement-map, hidden-answer, no-buttons [35]. Rasd maps ~20 of these to first-class widget props (layout: 'grid'|'columns', variant:'minimal'|'autocomplete', calendar:'islamic', capture:'signature', mask, thousandsSep) and stores the raw list.
3.4 Settings, choices, entities, media
- settings:
form_title→title,form_id→id,version→version,instance_name→instanceName (REL),default_language,public_key→encryption.publicKey,submission_url→ext.xlsform.settings,style(pages→RFD pages;theme-grid→layout),allow_choice_duplicates,auto_send/auto_delete/client_editable→ext.collect,name→ext.xform.rootName,namespaces/attribute::*→ext [35]. - choices:
list_name,name,label[::lang],media::image,geometry+ arbitrary filter columns →choices[list] = [{name,label,media,attrs:{…}}];allow_choice_duplicates[35]. - entities:
list_name(dataset),label,entity_id,create_if,update_if(+entities:version2022.1.0/2023.1.0/2024.1.0;offlinemetadatatrunkVersion/branchId) →entity:{dataset,label,id,createIf,updateIf}[21]; a Rasd form that references an entity list getsdatasets:[{name, kind:'entityList'}]and the adapter fetchesentities.csv(ETag) [8]. - media:
jr://images|audio|video|file|file-csv/…→ RFDmediarefs by filename; manifest resolves them [20].
3.5 Kobo extensions (from kpi source) [36][37][38][39]
| Kobo construct | Kobo's own expansion | RFD import | Export |
|---|---|---|---|
begin_score…end_score, score__row, col kobo--score-choices | begin_group appearance field-list + header row appearance label; each row select_one appearance list-nolabel [36] | group{layout:'score-grid', ext.kobo.score:{choices}} with select_one children | fold back to begin_score when target = Kobo, else expanded standard XLSForm |
begin_rank…end_rank, rank__level, cols kobo--rank-items, kobo--rank-constraint-message | begin_group field-list + note; each level select_one (required, appearance minimal) with constraint ${level} != ${prev} and … [37] | RFD rank (native array) + ext.kobo.rank | Kobo target: begin_rank; other: standard rank |
begin_kobomatrix…end_kobomatrix, col kobo--matrix_list | header group with w7-style widths, notes ##### {}; per list item a begin_group {matrix}_{item} with children {item}_{q}; appearances w2 horizontal-compact / w2 no-label [38] | matrix element (rows = list, columns = child questions) storing values as {row:{col:val}}; serializer expands to {matrix}_{item}/{item}_{q} nodes | fold back for Kobo |
kobo--locking-profile (survey/settings), kobo--lock_all, sheet kobo--locking-profiles [39] | n/a | ext.kobo.locking (profiles + restriction names) | verbatim |
select_one x or_other, _or_other column [29] | pyxform or_other | allowOther | _or_other for Kobo |
asset-content keys $kuid,$autoname,$autovalue,$xpath,$given_name, translations:[null,…], translated:[label,hint] [29] | n/a | ext.kobo.kuid etc. | needed for kpi content PATCH round-trip |
3.6 Lossless storage
Every RFD element carries ext: { xlsform: { row:{unmapped columns}, raw:{relevant,constraint,calculation,choice_filter,required,default}, rowIndex }, xform:{ path, rootName, bind:{…verbatim attrs} }, kobo:{…} }. Exporter rule: if the REL AST hash still equals the hash computed at import, re-emit ext.xlsform.raw.* verbatim; otherwise serialize REL→XPath. Form-level ext.xlsform.settings, ext.xlsform.extraSheets (osm, kobo--locking-profiles, external_choices) and column order are preserved so an unchanged import/export is byte-comparable in pyxform output.
4. XPath → REL translation rules and gap list
| XPath / XLSForm | REL | Rule |
|---|---|---|
${q} | ${q} | resolve in nearest scope; inside repeats = current instance |
. / .. | . / .. | .. = enclosing repeat/group instance |
/data/g/q, ../q, ../../q | ${q} or path ref @/g/q | resolve via ext.xform.path map; REL needs absolute-path refs |
current()/../q (choice_filter, itemsets) | ${q} relative to current question | most common in cascading selects inside repeats |
instance('L')/root/item[name=${q}]/label | choice-label('L', ${q}) (= jr:choice-name) | |
instance('csv')/root/item[k=${x}]/v | lookup('csv','v','k',${x}) (alias pulldata) | pulldata is defined as exactly this shortcut [20] |
count(${rep}), sum(${rep_q}), max/min/join over repeat | count(${rep}), sum(${rep}.q) | ${q} inside an aggregate from outside the repeat = collection |
position(..) | position() (1-based) | |
indexed-repeat(${q}, ${r}, i[, ${r2}, j]) | same builtin (≤3 levels) [20] | |
selected/selected-at/count-selected | same, on arrays | Kobo/Ona JSON path also joins arrays [13] |
jr:itext('id') | itext('id') | |
regex(., 'p') | regex(., 'p', mode) | JavaRosa = anchored Pattern.matches; Enketo & Web Forms = unanchored RegExp.test [25][26][27] → import lint: warn when pattern lacks ^…$; default mode:'full' for Collect parity |
format-date(%Y %y %m %n %b %d %e %a), format-date-time(+%H %h %M %S %3) | same tokens [20] | |
today(), now() (local, with offset), decimal-date-time, decimal-time, date() | same | dates in arithmetic → decimal days since epoch [20] |
once(), uuid([n]), random(), randomize(nodeset[,seed]), `digest(s,'SHA-256'[, 'hex' | 'base64'])` | same |
if, coalesce, boolean-from-string, checklist, weighted-checklist, int, round(n,d), pow, log10, math set | same | |
distance, area, geofence, intersects | same on RFD geo objects | |
substr (0-based), string-length (arg required), concat (node-sets ok), join | same | |
| coercion | XPath rules | boolean('false')=true; ''→NaN in arithmetic; number-vs-string = compares numerically [20] |
Not translatable (kept verbatim in ext, evaluated only in the optional Node @getodk/xpath import pipeline): XPath axes (ancestor::, following-sibling::), //, *, union |, name()/local-name()/lang()/id(), predicates with positional filters on the primary instance other than the indexed-repeat shape, count(/data/rep[q='yes'])-style filtered aggregates (recommend adding count-if(${rep}, q = 'yes') to REL), instance() node-sets fed to count()/sum(), dynamic jr:itext(concat(...)).
Coverage estimate (to be measured — see §6): on typical humanitarian XLSForms authored in Kobo/XLSForm style, ≥97 % of individual expressions are ${}-refs + the ~60 functions above and parse 1:1; 90–95 % of forms import with zero unmapped expressions; residual losses cluster in cascading-select choice_filters using current()/../, repeat aggregates, and hand-written /data/... paths. ODK's own JS engine reports 98 % XPath-function coverage and 91 % question-type coverage after two years — a realistic ceiling [prior note 09].
Round-trip loss classes: (1) expression text formatting (mitigated by raw re-emit); (2) Rasd-only widgets/themes/pages (Collect ignores unknown appearances; pages ↔ style: pages); (3) transparent groups vs XForm groups (instance nesting); (4) Rasd types with no XLSForm type (boolean, matrix, rating → select_one + appearance, email/url → text + constraint); (5) language keys (label::Arabic (ar) vs ar); (6) Kobo/Ona-only columns; (7) numeric formatting (1.0 vs 1); (8) default vs once() normalisation; (9) media filename case; (10) entities sheet ordering.
5. Adapter architecture and roadmap
interface RasdTransport {
capabilities(): { write:'openrosa-xml'|'json'|'rasd', chunking:boolean, maxBody:number,
edit:'deprecatedID'|'put'|'none', pull:'odata'|'kpi'|'ona'|'rasd'|'none',
entities:boolean, encryption:'envelope'|'none', auth:('bearer'|'basic'|'digest'|'token-path'|'api-token')[] };
listForms(): Promise<FormRef[]>; // formList → {formID,name,version,hash,downloadUrl,manifestUrl}
pullForm(ref): Promise<{ xform:string, rfd:RFD, media:MediaRef[], datasets:DatasetRef[] }>;
pullDataset(ref, etag?): Promise<{ rows|csv, etag }>; // entities.csv / paired-data / manifest CSV
push(sub:RasdSubmission, o?): Promise<PushResult>; // serialize → HEAD → chunked POST(s) → 201/202
edit?(sub, previousInstanceId): Promise<PushResult>; // deprecatedID
pullSubmissions?(since): AsyncIterable<PulledSubmission>;
}
Profiles: openrosa (base: HEAD/ACL, multipart, digest/basic), central (adds /v1/key/{token} URL builder, REST attachments, OData/entities, review states), kobo (kc/kf host pair, Token auth, JSON fast path, validation_status, Data-Collector tokens), ona (/{username} paths, /api/v1/submissions, data/{pk}, last_write_wins). Also rasd (REST+tus).
Error → Rasd status model: 401/403 → auth_required (pause queue, keep local); 409 identical hash → sent; 409 different → conflict (needs user/edit); 413 → re-chunk, else too_large; 400 (Central: unknown attachment name / bad XML) → rejected (keep, surface); 404 form closed/deleted → blocked; 5xx/network → queued with exponential backoff + jitter; 202 → sent (accepted). Idempotency key = meta/instanceID; safe to retry on all four servers [4][13][17].
Two blockers to test early: CORS for browser PWAs hitting kc.kobotoolbox.org / api.ona.io / Central directly [unverified] — plan a relay transport (Rasd server proxies with the user's token) and Digest auth in React Native (implement RFC 7616 MD5 for generic OpenRosa; use Token/Bearer/URL-token everywhere else).
Roadmap: P1 Kobo (biggest UN/NGO base — UNHCR, IOM, OCHA, REACH; JSON write path; Data-Collector tokens; kpi pull) → P2 ODK Central (WFP/UNICEF adoption, entities, OData, Web Forms default in v2026.2) → P3 Ona/MoDa (WFP corporate; ~90 % code shared with Kobo profile) → P4 generic OpenRosa (SurveyCTO, self-hosted formhub) → P5 encryption envelope, offline entities (branchId), edits/reviewState sync.
6. Test corpus and CI plan
| Corpus | What | Licence / URL |
|---|---|---|
pyxform 4.5.0 tests/fixtures/example_forms (36 files: xlsform_spec_test.xlsx, widgets.xlsx, pull_data.xlsx, repeat_date_test.xlsx, choice_filter_test.xlsx, or_other.xlsx, attribute_columns_test.xlsx, flat_xlsform_test.xlsx, field-list.xlsx, loop.xlsx, group.xlsx, case_insensitivity, utf_csv.csv…) + bug_example_forms (9) + test_expected_output XML goldens | XLSForm→XForm golden | BSD-2 — https://github.com/XLSForm/pyxform/tree/master/tests/fixtures [22][23] |
pyxform unit tests (~90 files: test_repeat.py, test_rank.py, test_external_instances*.py, test_translations.py, test_unicode_rtl.py, test_settings.py, test_trigger.py, test_set_geopoint.py, entities/) | markdown-table XLSForms embedded in tests — cheap to harvest | same |
ODK Web Forms packages/common/src/fixtures/{computations,controls,date-and-time,entities,geolocation,groups,itext,notes,rank,repeats,select,upload,validation,value-types,xpath-fns,test-javarosa,test-scenario} + packages/scenario (JavaRosa-ported tests: initial 120 pass/193 fail/28 todo) | engine conformance, XPath fn semantics | Apache-2.0 — https://github.com/getodk/web-forms [40][41] |
@getodk/xpath 1.0.0 test suite | XPath 1.0 + ODK function oracle for REL | Apache-2.0 [24] |
JavaRosa src/test/resources (~45: template-repeat.xml, repeat-secondary-instance.xml, nigeria_wards_external*.xml, rank-form.xml, two-secondary-instances.xml, regression/, smoketests/) | reference behaviour of Collect | Apache-2.0 — https://github.com/getodk/javarosa [42] |
enketo-transformer test/forms (43 XForms: itemset.xml, rank.xml, setvalue*.xml, setgeopoint*.xml, external.xml, widgets.xml…) + enketo-core forms | web-form parity | Apache-2.0 — https://github.com/enketo/enketo [43] |
ODK Collect test forms (collect_app/src/test/resources/forms) and ODK docs sample forms | widgets/appearances | Apache-2.0 [44] |
| XLSForm.org examples; Kobo public library/templates; humanitarian XLSForms published by REACH/IMPACT (MSNA tools), IOM DTM, UNHCR, WFP mVAM [URLs to be catalogued; check redistribution terms] | realism: Arabic RTL, cascading admin selects, big repeats | mixed |
CI: (1) pyxform round-trip — Python job runs pyxform 4.5.0 on each XLSForm → XForm A; Rasd imports XLSForm→RFD→exports XLSForm→pyxform → XForm B; compare canonicalised XML (sorted attrs, whitespace-normalised, ignore version); fail on any diff outside an allow-list. (2) Expression conformance — evaluate every corpus expression with @getodk/xpath (Node/jsdom) and REL over generated contexts; assert equality (with the documented regex/NaN caveats). (3) Submission goldens — fill fixture answers → RFD serializer → XML; validate against the pulled form's field list; nightly post to dockerised ODK Central (getodk/central) and kobo-install, read back via OData/kpi and diff. (4) Transport contract tests — recorded HTTP fixtures (msw) for formList/manifest/HEAD/POST/409/413/202 paths per profile. (5) Track a coverage dashboard: % expressions parsed, % forms lossless, per-function pass rate.
7. Export conventions for Rasd's own server
Adopt the Kobo/Ona family as default (most UN Excel/Power BI pipelines are built on it), with a Central-compatible switch:
- Headers:
group/question(/separator; option-for Central mode); labels or XML names; multi-language labels [28]. select_multiple:q(space-separated names) andq/choice0/1 columns (multiple_select='both'|'summary'|'details') [29].- geopoint:
q,_q_latitude,_q_longitude,_q_altitude,_q_precision(Central mode:q-Latitude/-Longitude/-Altitude/-Accuracy) [29][30]. - repeats: one sheet/CSV per repeat with
_index,_parent_table_name,_parent_index(+ Kobo's_submission__uuid-style copies [naming unverified]); Central mode:KEY(uuid:…/rep[1]),PARENT_KEY[29][30]. - system columns:
_id, _uuid, _submission_time, _validation_status, _notes, _status, _submitted_by, __version__, _tags, _index(+_xform_id_string,_attachments,_geolocationin JSON) [28]; Central mode:SubmissionDate, KEY, SubmitterID, SubmitterName, AttachmentsPresent, AttachmentsExpected, Status, ReviewState, DeviceID, Edits, FormVersion[30]. - Media: filename column +
{q}_URLwheninclude_media_url[29]. - Also expose an OData 4.0 minimal feed shaped like Central's (
Submissions,Submissions.{repeat},__id,__system,$expand=*,$skiptoken) so existing Power BI templates for Central work unchanged [10].
8. Implications & recommendations for Rasd Forms
- Build one serializer (
@rasd/xform-io): RFD → XForm instance XML per §2, driven by the pulled XForm's node order/root/formhubblock; reverse parser for Kobo/Ona JSON and Central OData. This is the load-bearing piece for all three servers. - Ship
@rasd/adapter-openrosafirst (HEAD/ACL, chunked multipart,X-OpenRosa-Version: 1.0, 201/202/409/413 handling), then thin profileskobo,central,onaon top; keep the JSON write path for Kobo/Ona as an opt-in fast path. - Support token-in-URL enrolment (Central app users, Kobo Data Collectors, Central draft tokens) and Collect-style QR settings import (zlib+base64 JSON with
general.server_url), so field teams keep their existing onboarding [6][14][45]. - Treat
meta/instanceIDas the universal idempotency key; retries are safe on all servers; map 409 with identical local hash tosent. - Implement Central specifics: per-file attachment
POST,PUTedits withdeprecatedID,reviewStateread-back, entities create/update (string values only,baseVersion, offlinebranchId), OData pull with$skiptoken, ETag onentities.csv[5][8][9][10]. - In
@rasd/xlsform: import every column/sheet withextpassthrough (§3.6); implement Kobobegin_score/rank/kobomatrixfolding both ways; keep language header strings; recordrootName/pathper element. - REL: add absolute-path refs, collection refs for repeat aggregates,
lookup()/choice-label()sugar,count-if, XPath coercion mode (empty→NaN, string booleans), and aregexanchoring switch with an import lint. Publish a conformance table against@getodk/xpath. - Do not attempt a JS XLSForm→XForm compiler; call pyxform (Python) in the Node import pipeline/CI and cache results; use
@getodk/xpathonly server-side for unmappable expressions. - Provide a
relaytransport and verify CORS/Digest early; PWAs may need the Rasd server (or the org's proxy) between browser and Kobo/Ona/Central. - Exports: Kobo-family default, Central-mode switch, Central-shaped OData feed (§7).
- Encryption: implement the ODK envelope (AES-256-CFB + RSA-OAEP) as an optional client feature for Central managed encryption; document that Kobo cannot decrypt.
- CI as in §6, with a public coverage dashboard — this is also a sales artefact for UN procurement ("imports N of M REACH/IOM/UNHCR forms losslessly").
Unverified / to re-check
Kobo Data Collectors live status on kf/eu and its UI; Kobo direct-API edit via deprecatedID; Kobo validation_status.uid value list; Ona ACL default; CORS headers on kc/api.ona/Central; Central API rate limiting; ODK Collect QR JSON key values (form_update_mode, autosend enumerations); Kobo repeat-sheet _submission__* column names; licences/URLs of the humanitarian XLSForm sets.
Sources (accessed 2026-08-15)
- ODK Docs — OpenRosa protocol overview. https://docs.getodk.org/openrosa/
- ODK Docs — OpenRosa Form Submission API. https://docs.getodk.org/openrosa-form-submission/
- ODK Docs — OpenRosa Form List API. https://docs.getodk.org/openrosa-form-list/
- ODK Central API — OpenRosa endpoints. https://docs.getodk.org/central-api-openrosa-endpoints/
- ODK Central API — Submission management. https://docs.getodk.org/central-api-submission-management/
- ODK Central API — Accounts, users, app users. https://docs.getodk.org/central-api-accounts-and-users/
- ODK Central API — Changelog (v2026.2 newest). https://docs.getodk.org/central-api-changelog/
- ODK Central API — Dataset management. https://docs.getodk.org/central-api-dataset-management/
- ODK Central API — Entity management. https://docs.getodk.org/central-api-entity-management/
- ODK Central API — OData endpoints. https://docs.getodk.org/central-api-odata-endpoints/
- ODK Central API — Form management. https://docs.getodk.org/central-api-form-management/
- KoboToolbox — Using the API (kf/eu hosts, token). https://support.kobotoolbox.org/api.html
- kpi source — OpenRosa submission viewset & docs (XML/JSON, 201/202/409,
/collector/{token}/submission). https://github.com/kobotoolbox/kpi/blob/main/kobo/apps/openrosa/apps/api/viewsets/xform_submission_api.py and …/kobo/apps/openrosa/docs/api/openrosa/submission/{authenticated,anonymous,data_collector}.md - kpi source — Data Collectors (models, token auth) and formList viewset. https://github.com/kobotoolbox/kpi/tree/main/kobo/apps/data_collectors ; …/kobo/apps/openrosa/apps/api/viewsets/xform_list_api.py
- kpi source —
/api/v2/assets/{uid}/data/viewset docstrings. https://github.com/kobotoolbox/kpi/blob/main/kpi/views/v2/data.py - kpi source — settings (
OPENROSA_DEFAULT_CONTENT_LENGTH,DEFAULT_API_PAGE_SIZE,MAX_API_PAGE_SIZE) and paginators. https://github.com/kobotoolbox/kpi/blob/main/kobo/settings/base.py ; …/kpi/paginators.py - Ona API — Submissions. https://api.ona.io/static/docs/submissions.html
- Ona API — Forms. https://api.ona.io/static/docs/forms.html
- Ona API — Data. https://api.ona.io/static/docs/data.html
- ODK XForms Specification (types, meta, functions, coercion). https://getodk.github.io/xforms-spec/
- ODK XForms Specification — Entities. https://getodk.github.io/xforms-spec/entities
- PyPI — pyxform 4.5.0 (2026-06-25). https://pypi.org/project/pyxform/
- pyxform source —
xls2xform.py(--json= status JSON,ConvertResult),xform2json.py,json_form_schema.json, tests. https://github.com/XLSForm/pyxform - npm registry — enketo-transformer 4.2.0, enketo-core 9.0.1, openrosa-xpath-evaluator 3.2.0, @getodk/xforms-engine 1.0.2, @getodk/xpath 1.0.0, xform-to-json 1.2.1, xls2xform 1.1.0. https://registry.npmjs.org/
- JavaRosa —
XPathFuncExpr.regex()usesPattern.matches. https://github.com/getodk/javarosa/blob/master/src/main/java/org/javarosa/xpath/expr/XPathFuncExpr.java - Enketo — openrosa-xpath-evaluator
regex()usesnew RegExp().test(). https://github.com/enketo/enketo/blob/main/packages/openrosa-xpath-evaluator/src/openrosa-extensions.js - ODK Web Forms —
@getodk/xpathregex(). https://github.com/getodk/web-forms/blob/main/packages/xpath/src/functions/xforms/string.ts - kpi source —
common_tags.py(submission metadata field names). https://github.com/kobotoolbox/kpi/blob/main/kobo/apps/openrosa/libs/utils/common_tags.py - kobotoolbox/formpack — export options (
group_sep,multiple_select,_index/_parent_*,_latitude…_precision,_URL,_or_other, translations). https://github.com/kobotoolbox/formpack - ODK Docs — Central submissions/CSV export columns. https://docs.getodk.org/central-submissions/
- ODK Docs — Encrypted forms. https://docs.getodk.org/encrypted-forms/
- ODK XForms Specification — Encryption. https://getodk.github.io/xforms-spec/encryption.html
- KoboToolbox — KoboCollect on Android (kc / kc-eu URLs, QR, anonymous). https://support.kobotoolbox.org/kobocollect_on_android_latest.html
- KoboToolbox — OpenAPI schema. https://kf.kobotoolbox.org/api/v2/schema/?format=json
- XLSForm.org — reference (types, columns, settings, appearances, parameters). https://xlsform.org/en/
- kpi —
koboscore_handler.py. https://github.com/kobotoolbox/kpi/blob/main/kpi/utils/xlsform_preprocessors/koboscore_handler.py - kpi —
koborank_handler.py. https://github.com/kobotoolbox/kpi/blob/main/kpi/utils/xlsform_preprocessors/koborank_handler.py - kpi —
kobomatrix_handler.py. https://github.com/kobotoolbox/kpi/blob/main/kpi/utils/xlsform_preprocessors/kobomatrix_handler.py - KoboToolbox — Library locking (
kobo--locking-profile(s),kobo--lock_all). https://support.kobotoolbox.org/library_locking.html - ODK Web Forms — fixtures. https://github.com/getodk/web-forms/tree/main/packages/common/src/fixtures
- ODK Web Forms —
packages/scenarioREADME (JavaRosa-ported suite). https://github.com/getodk/web-forms/blob/main/packages/scenario/README.md - JavaRosa — test resources. https://github.com/getodk/javarosa/tree/master/src/test/resources
- Enketo — enketo-transformer test forms. https://github.com/enketo/enketo/tree/main/packages/enketo-transformer/test/forms
- ODK Collect — test forms. https://github.com/getodk/collect/tree/master/collect_app/src/test/resources/forms
- ODK Docs — Collect settings import/export (QR JSON:
general/admin/project, zlib+base64). https://docs.getodk.org/collect-import-export/ - Kobo — Export downloads (system columns). https://support.kobotoolbox.org/export_download.html
- Prior Rasd research notes 01 (MoDa = Ona at api.moda.wfp.org) and 09 (REL design, ODK Web Forms coverage). /Users/huthaifakhreshi/Downloads/metals-analyzer/rasd-forms/docs/research/01-un-field-data-collection-landscape.md ; …/09-field-features-and-expression-engine.md