# Open Semantic Interchange — Deep dive: Every Key, Every Record: How the Ossie Coder Maps to the IR An Apache Ossie semantic model and a typed intermediate representation disagree about the world in one fundamental way: an Ossie field has no type. It is a name, an optional expression, an optional description, and optional dimension metadata. The IR that CoreModels (by ARAMAI) moves schemas through is typed, and so is every neighboring format - SQL, Avro, JSON Schema, LinkML. # Every Key, Every Record: How the Ossie Coder Maps to the IR An Apache Ossie semantic model and a typed intermediate representation disagree about the world in one fundamental way: an Ossie field has no type. It is a name, an optional expression, an optional description, and optional dimension metadata. The IR that CoreModels (by ARAMAI) moves schemas through is typed, and so is every neighboring format - SQL, Avro, JSON Schema, LinkML. Everything interesting about this coder follows from that mismatch, and from one rule: anything the target cannot hold goes into the ledger - never silently discarded, never smuggled into the wire format. Here is the complete map. ## The structural map | Ossie construct | IR construct | Notes | |---|---|---| | the document | one IR schema | id is the first model's sanitized name; label is the name as written | | first `semantic_model` entry | the schema node itself | its `description` and `ai_context` ride on the schema's annotations | | further `semantic_model` entries | flattened into the same schema | ids prefixed with the model name; recorded in the ledger | | `dataset` | Type | label = name as written; id = sanitized name | | `field` | Element | String - or DateTime when `dimension.is_time` | | `primary_key` membership | `Required` on the Element | plus a marker annotation so the key can be rebuilt on encode | | `relationship` | Relation in the `osiRelationship` group | one custom group, labeled *OSI Relationship*, Type → Type endpoints | | `metric` | Component | members are the datasets its expression names | | `custom_extensions` | schema annotations | read from the semantic model and, tolerantly, from the document root | Ids are sanitized because CoreModels node ids must be alphanumeric - the id index tokenizes on separators, so a raw `web_sales` could not be matched by an exact-id query. `web_sales` becomes `webSales`, `ws_item_sk` under it becomes `webSalesWsItemSk`, and the original names survive untouched as labels. Collisions get a numeric suffix: two datasets named `web_sales` and `web sales` decode to `webSales` and `webSales2`. Metric membership is a best-effort scan: each dataset's name or `source` is matched at word boundaries inside the metric's expression (preferring `ANSI_SQL`), so `SUM(web_sales.ws_sales_price)` attaches the metric to `webSales`. When nothing matches, the component is still created - with a ledger entry saying it references no types. ## The annotation contract Everything the IR has no first-class slot for rides in dotted `osi.*` annotations, which is what makes an exact re-emit possible: | Key | Carries | |---|---| | `osi.version` | the wire version string | | `osi.description` | description text (schema, dataset, field, metric) | | `osi.aiContext.instructions` / `.synonyms` / `.examples` | the `ai_context` block (arrays as JSON) | | `osi.source` | a dataset's `source` | | `osi.uniqueKeys` | `unique_keys`, verbatim as JSON | | `osi.primaryKey` | `"true"` when the field is in the primary key | | `osi.expression.` | one entry per dialect, expression text verbatim | | `osi.fromColumns` / `osi.toColumns` | a relationship's join columns | | `osi.name` | a relationship's original name | | `osi.metric` | `"true"` - marks a component as an Ossie metric | | `osi.ext.` | a custom extension's `data` | | `osi.raw.` | any unknown wire key, preserved verbatim as JSON | Decoding the two-dataset retail model we test against produces exactly this on the schema node: ```json { "osi.version": "0.2.0.dev0", "osi.description": "Retail sales semantic model.", "osi.aiContext.instructions": "Prefer web_sales for revenue questions.", "osi.aiContext.examples": "[\"What was total revenue last month?\"]", "osi.ext.ARAMAI": "{\"owner\":\"analytics-team\",\"refresh\":\"daily\"}" } ``` …and on a primary-key field with an expression: ```json { "osi.expression.ANSI_SQL": "ws_item_sk", "osi.description": "Item surrogate key.", "osi.primaryKey": "true" } ``` The relationship keeps its own: `osi.name`, `osi.fromColumns`, `osi.toColumns`. The metric keeps `osi.metric`, `osi.expression.ANSI_SQL`, `osi.description`. Note the version: `0.2.0.dev0` is preserved as written. We emit **0.1.1** when a model carries no version - pinned against the spec's JSON schema at the `osi-0.1.1-rc1` tag of the `apache/ossie` repository - but we never rewrite a version somebody chose. The one exception is the legacy `1.0`, a release that never existed; it normalizes to 0.1.1 with a `SemanticNarrowing` record, in both directions. ## Parsing posture The document is parsed once into a generic object graph and walked defensively - never bound to rigid classes, so an unfamiliar key cannot break a decode. JSON is a YAML subset, so the same parse reads the `osi-json` serialization; the two format keys differ only in what they *emit*. Consequences: - An unknown key on a dataset, field, relationship, or metric is preserved as `osi.raw.` and re-emitted verbatim. Give a dataset `refresh_cron: "0 3 * * *"` and it comes back out as `refresh_cron: "0 3 * * *"`. - An unknown key on the semantic model itself is reported (`Unrecognized semantic-model keys not modelled: owner_team.`) rather than carried. - Unknown keys inside `ai_context`, `expression`, and `dimension` are reported the same way. - `primary_key` accepts a scalar or a list. `expression` accepts the spec's `dialects` structure or a bare string, which is treated as `ANSI_SQL`. - `semantic_model` accepts a single mapping as well as a list. Three inputs fail outright: an empty document (`The OSI YAML document is empty.`), invalid YAML (`The document is not valid YAML: ...`), and a valid document with no model (`The document contains no semantic_model entry.`). ## The lossiness inventory **Decoding** can emit five records: | Kind | When | |---|---| | `TypeApproximation` at `$` | always, when the model has fields: *OSI carries no field type system; every field was decoded as String (or DateTime when dimension.is_time).* One summarized record for the whole schema, not one per field. | | `SemanticNarrowing` at `version` | the legacy `1.0` was normalized to 0.1.1 | | `SemanticNarrowing` at `semantic_model` | several models flattened into one schema with prefixed ids | | `StructuralDrop` at `...primary_key` | a key column matching no field: *they are dropped and a re-encode will emit the key without them* | | `StructuralDrop` at `...relationships[i]` | endpoints that resolve to no dataset: *the relation was dropped* | Plus `SemanticNarrowing` for a metric whose expression names no known dataset, for unrecognized keys in the places listed above, and for a custom extension with no `vendor_name`. **Encoding** reports what Ossie cannot express: | Kind | Path | Reason | |---|---|---| | `ConstraintRelaxation` | `Taxonomy[...]` | OSI has no controlled-list construct; the taxonomy (and any enum constraint) is not represented | | `SemanticNarrowing` | `Type[...]` | OSI has no inheritance; the parent link is not represented | | `SemanticNarrowing` | `Relation[...]` | only dataset-to-dataset OSI relationships are expressible; other relation groups are not | | `StructuralDrop` | `Component[...]` | only components marked `osi.metric` are exported | That last table is why a JSON Schema enum converts to Ossie with a `ConstraintRelaxation`: the values have nowhere to live. Two encode-side losses carry no record, because the wire format has no slot to lose them from. Field **types** are one: an Ossie field has none, and the decode-side approximation already states that rule for the format. **Requiredness that is not key membership** is the other: `primary_key` is rebuilt only from elements marked as key members, so a `required` flag arriving from JSON Schema or SQL does not invent a key. Both are worth remembering when you convert *into* Ossie. ## What round-trips Decode → encode → decode is stable. For the retail model, the structural signature - types, their element lists, each element's label, value type and requiredness, relations with their group and endpoints, components with their members - is identical before and after. So is the signature after a round trip through the JSON serialization, and the re-emitted YAML carries `ai_context`, `unique_keys`, the composite `primary_key`, per-dialect expressions, the time dimension, and the custom extension (abridged below to one dataset and two of its fields; the `item` dataset the relationship points at and the second metric are omitted for length): ```yaml version: "0.2.0.dev0" semantic_model: - name: "retail_analytics" description: "Retail sales semantic model." ai_context: instructions: "Prefer web_sales for revenue questions." examples: ["What was total revenue last month?"] datasets: - name: "web_sales" source: "tpcds.web_sales" description: "Web channel sales facts." primary_key: ["ws_item_sk","ws_order_number"] unique_keys: [["ws_item_sk","ws_order_number"]] ai_context: synonyms: ["online sales","web orders"] fields: - name: "ws_item_sk" description: "Item surrogate key." expression: dialects: - dialect: "ANSI_SQL" expression: "ws_item_sk" - name: "ws_sold_date" description: "Date of sale." expression: dialects: - dialect: "ANSI_SQL" expression: "ws_sold_date" dimension: is_time: true relationships: - name: "web_sales_item" from: "web_sales" to: "item" from_columns: ["ws_item_sk"] to_columns: ["i_item_sk"] metrics: - name: "total_revenue" description: "Total revenue across web sales." expression: dialects: - dialect: "ANSI_SQL" expression: "SUM(web_sales.ws_sales_price)" ai_context: synonyms: ["revenue"] custom_extensions: - vendor_name: "ARAMAI" data: "{\"owner\":\"analytics-team\",\"refresh\":\"daily\"}" ``` Four details in that document are decisions, not accidents: **Emission is hand-written.** Fixed key order, two-space indent, quoted scalars, JSON flow style for lists - output is byte-stable and exactly re-readable. The JSON serialization derives from the same emission, and scalar style disambiguates typing there: quoted scalars stay strings, plain ones re-type, so `is_time` becomes a real JSON `true`. **Extensions are emitted under the semantic model**, their home in the spec, even when the input placed them at the document root; `data` is written as a JSON string literal, because the spec types that field as a string. **Every field gets an expression.** The spec requires one, so a field that arrived without any dialect is re-emitted with its own name as the `ANSI_SQL` expression. Likewise a dataset with no `source` is emitted with its sanitized name - `web_sales` gets `source: "webSales"` - rather than producing a document that fails spec validation. **`primary_key` is rebuilt from the per-field markers.** That is why a key column matching no field is a `StructuralDrop` at decode time: it would quietly change the key's composition on the way back out. ## Fidelity beyond the coder Two boundaries beyond the coder change what survives - both visible in the ledger. Through the **mapping engine** (the `schema/map` route and the `transform_schema` tool), an inferred plan carries types, elements, and taxonomies. Relationships come back as `StructuralDrop` at `Relation[...]` - *source relation not carried by any relation-mapping operation* - until the plan carries a `RelationMapping` operation naming the `osiRelationship` group; metrics need a `ComponentMapping`. Add those two operations to the returned plan, replay it through `plan/execute`, and the `relationships` and `metrics` blocks come back. Through a **CoreModels project**, the graph stores types, elements, value types and requiredness - not the format-specific `osi.*` annotations, and not relation instances or components (both reported as `StructuralDrop` on import). An export from a project is therefore a clean 0.1.1 document: real dataset and field names, sources and expressions derived from those names, time dimensions intact, no `ai_context`. If you need an Ossie document to survive a conversion with its annotations whole, keep it in the stateless routes, where the document never leaves the request. The endpoint reference and the per-format quickstarts are in the CoreModels transform docs.