# ShEx — Deep dive: Inside the ShEx Coder: The IR Map, the Extras Ledger, and What Round-Trips

Most schema formats name things locally: a SQL column is `full_name`, an Avro field is `full_name`, and any connection to the wider world has to be bolted on with annotations. ShEx arrives different. Its predicates and shape names are IRIs - `schema:name` *is* `https://schema.org/name` - which means a ShEx document carries its cross-standard identity in its bones. Our ShEx coder in CoreModels, by ARAMAI, is built around that fact, and this article is the full technical account of it: exactly what maps into our intermediate representation (IR), what rides in the preserved-extras ledger, every lossiness record the coder can emit, and precisely what survives a round trip.

# Inside the ShEx Coder: The IR Map, the Extras Ledger, and What Round-Trips

Most schema formats name things locally: a SQL column is `full_name`, an Avro field is `full_name`, and any connection to the wider world has to be bolted on with annotations. ShEx arrives different. Its predicates and shape names are IRIs - `schema:name` *is* `https://schema.org/name` - which means a ShEx document carries its cross-standard identity in its bones. Our ShEx coder in CoreModels, by ARAMAI, is built around that fact, and this article is the full technical account of it: exactly what maps into our intermediate representation (IR), what rides in the preserved-extras ledger, every lossiness record the coder can emit, and precisely what survives a round trip.

The coder is bidirectional under the format key `shex`. Its external representation is the ShExC (compact syntax) document as a string, and the subset it models is: shapes, triple constraints, `xsd:` datatypes, shape references (`@Shape`), inline value sets (`[ ... ]`), the cardinality operators (`?`, `*`, `+`, `{n,m}`), `EXTENDS` inheritance, and `PREFIX` declarations.

That entire subset fits in one small document:

```text
PREFIX schema: <https://schema.org/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

schema:PersonShape {
  schema:name xsd:string ;
  schema:email xsd:string * ;
  schema:birthDate xsd:date ?
}

schema:EmployeeShape EXTENDS @schema:PersonShape {
  schema:jobTitle xsd:string ;
  schema:workStatus [ "active" "leave" "terminated" ]
}
```

Decoded, that is two Types (`Person` and `Employee`, suffixes stripped), five Elements, one Taxonomy from the value set, a single parent link from `EXTENDS`, and a prefix map riding in extras - the mechanics the rest of this article specifies table by table.

## The structural map

The IR describes schemas as Types (record-like structures) containing Elements (fields), plus Taxonomies (controlled term lists), with every node carrying an annotation bag: `mapsTo` links to external standards, free-form `Extras`, and provenance (ShEx-decoded nodes are stamped with source format `ShEx` and the originating shape or predicate).

| ShExC construct | IR construct |
|---|---|
| Shape (`schema:PersonShape { ... }`) | Type; id is the shape name, label is the local name with the `Shape` suffix stripped (`Person`) |
| Triple constraint (`schema:name xsd:string ;`) | Element; id is `<shapeName>\|<predicate>`, label is the predicate's local name (`name`) |
| Datatype constraint (`xsd:integer`) | Primitive value type (see the datatype tables) |
| Shape reference (`@schema:PersonShape`) | Type reference - a real link to the target Type, not a string |
| Value set (`[ "draft" "active" ]`) | Taxonomy with one term per entry; id is `<elementId>\|enum` |
| `EXTENDS @Parent` | The Type's parent - single inheritance |
| Cardinality symbols | Required flag + single/collection cardinality (next table) |
| `PREFIX p: <iri>` | Schema-level prefix map, preserved in extras |

Cardinality is a two-axis decode - is the element required, and is it collection-valued:

| ShExC | Required | Cardinality |
|---|---|---|
| *(none)* | yes | single (ShEx's default is exactly one) |
| `?` | no | single |
| `+` | yes | collection |
| `*` | no | collection |
| `{n,m}` | yes when n ≥ 1 | collection when m > 1, with min/max kept in the IR |

The numeric form is the one asymmetry: `{1,4}` decodes faithfully (required, collection, min 1, max 4), but the encoder writes only the four symbols, so it re-emits as `+`. The bounds live on in the IR - formats that can express them receive them - but ShEx output rounds to the nearest symbol. `{0,1}` is recognized as exactly `?`.

## Datatypes, both directions

Decoding `xsd:` datatypes to IR primitive kinds:

| ShEx datatype | IR kind |
|---|---|
| `xsd:integer`, `xsd:int`, `xsd:long`, `xsd:short` | Integer |
| `xsd:decimal`, `xsd:double`, `xsd:float` | Double |
| `xsd:boolean` | Boolean |
| `xsd:dateTime`, `xsd:date`, `xsd:time` | DateTime |
| `xsd:string`, `xsd:anyURI`, anything unknown | String |

Encoding fresh IR back to ShEx:

| IR kind | ShEx datatype |
|---|---|
| Integer | `xsd:integer` |
| Double | `xsd:decimal` |
| Boolean | `xsd:boolean` |
| DateTime | `xsd:dateTime` |
| String, RichText | `xsd:string` |

Read those two tables together and you see a many-to-one funnel: `xsd:short` and `xsd:long` both become Integer. That would be lossy - except it isn't on a round trip, because of the extras ledger below.

A non-`xsd` node constraint - `.`, `LITERAL`, `NONLITERAL`, a bare IRI, or a class IRI used as a datatype - has no IR primitive. The decoder approximates it as String and records a `TypeApproximation`: *"ShEx value constraint '.' has no IR primitive; approximated as String."* The original token is still preserved, so ShEx-to-ShEx re-emits it exactly; the approximation is real only for other target formats, and the ledger says so.

## The extras ledger

Everything the IR's type system cannot hold natively, but the original document needs back, rides in dotted, coder-prefixed `Extras` keys - namespaced so no two coders ever collide:

| Key | On | Holds |
|---|---|---|
| `shex.prefix.<p>` | the schema | each prefix's namespace IRI (the full prefix map) |
| `shex.shapeName` | a type | the original prefixed shape name (`schema:PersonShape`) |
| `shex.predicate` | an element | the original prefixed predicate (`schema:email`) |
| `shex.datatype` | an element | the verbatim datatype token (`xsd:short`, `xsd:anyURI`, even `.`) |

On encode, preserved values win over synthesis: a ShEx-born element re-emits its exact predicate and its exact datatype, whatever the IR kind funneled to. This is how `xsd:date` survives being represented as DateTime, and `xsd:anyURI` survives being represented as String. The prefix map is re-emitted sorted, filtered to the prefixes actually used (with `xsd` always retained). Five prefixes are built in as defaults on both sides: `xsd`, `rdf`, `rdfs`, `schema` (`https://schema.org/`), and our own `cm` (written here as `https://coremodels.example.com/ns/`).

## The mapsTo lift - no annotation syntax required

Because predicates are IRIs, the decoder lifts every one into the element's `mapsTo` annotation automatically: standard = the prefix (`schema`), URI = the expansion (`https://schema.org/name`). Shapes get the same treatment at type level with one refinement: `schema:PersonShape` conventionally validates instances of `schema:Person`, so the coder strips the `Shape` suffix and maps the type to the class IRI `https://schema.org/Person`. Two deliberate exclusions: the `a` / `rdf:type` predicate is never lifted (it states rdf-typing, not a property meaning), and a prefix the document never declared cannot be expanded, so it produces no mapping.

This is the coder's biggest dividend. `mapsTo` is the cross-standard backbone our whole transform stack reads: a ShEx schema over schema.org predicates aligns - automatically, by URI - with any schema in any other format that maps to the same URIs, and mapping guides can set `autoMatchByMapsTo` instead of matching fragile labels.

Encoding runs the same wire in reverse for IR that was *not* born in ShEx: no preserved predicate means the encoder tries to reverse a `mapsTo` URI back into a prefixed name (longest-namespace match wins), and only then falls back to synthesizing `cm:<label>`. Shape names synthesize as `cm:<PascalCaseLabel>Shape`. So a SQL-born column with a `mapsTo` of `https://schema.org/name` exports as the real `schema:name`, while its unmapped siblings export under `cm:`.

## The lossiness inventory

The decoder emits exactly two kinds of ledger records:

- **SemanticNarrowing** - a shape with multiple `EXTENDS` clauses. The IR holds single inheritance, so the first parent wins: *"ShEx multiple EXTENDS (2) narrowed to a single IR parent 'schema:PersonShape'."*
- **TypeApproximation** - the non-`xsd` node constraints described above, one record per occurrence, with the original token named.

Equally important is what is skipped *without* a ledger entry, because the coder does not model it at all: `BASE`, `IMPORT`, and `start` statements; the `CLOSED` and `EXTRA` shape modifiers; anything between a constraint's value expression and its terminating `;` - inline `//` annotations and facet-style qualifiers alike; comments; and stray literals between shapes. Parsing never hard-fails on these - they are consumed and dropped. If your governance depends on `CLOSED` semantics or annotation content, know that they do not enter the IR. And one honest failure mode: an empty or whitespace-only document is rejected outright with the error *"The ShExC document is empty."* at path `$`.

## What round-trips, exactly

Our test suite pins the invariant decode → encode → decode: the second IR is structurally identical to the first - same types, parents, element labels, value types, required flags, and cardinalities. Layered on top of that:

- **Exact re-emission for ShEx-born schemas.** Prefixes, shape names, predicates, and verbatim datatypes come back from the extras ledger character-for-character.
- **Cross-format meaning survives.** The suite also proves ShEx → IR → JSON Schema lands the predicate as `"x-maps-to": { "schema": "https://schema.org/name" }` on the corresponding property.
- **Numeric cardinalities round to symbols** on ShEx output, as described above.
- **Value-set terms are opaque labels.** Terms decode as term id/label pairs and re-encode as quoted string literals - `[ "male" "female" ]` is perfectly stable, but a value set of IRIs would come back quoted as literals rather than IRIs. Use datatype or shape constraints for IRI-valued properties you need to round-trip.

That is the whole machine: a one-to-one structural map, a dotted extras ledger for exact re-emission, an automatic mapsTo lift that turns RDF's global naming into cross-format alignment, and a lossiness inventory short enough to memorize. The same anatomy - map, extras, ledger - repeats for every format we support; the transform section of the CoreModels docs covers the rest of the lineup.