# MACH ODM — Automation: ODM at Scale: Batch Conversion, Deterministic Pipelines, and Replayable Plans

One entity document is a demo. A standards repository is dozens of them - identity, product, inventory, pricing - revised by pull request, consumed by teams who need formal schemas, not Markdown. The moment MACH Open Data Model documents become an input to your build, three engineering questions appear: can the conversion run unattended, will the same input always produce the same output, and what happens when one file in the batch is broken?

# ODM at Scale: Batch Conversion, Deterministic Pipelines, and Replayable Plans

One entity document is a demo. A standards repository is dozens of them - identity, product, inventory, pricing - revised by pull request, consumed by teams who need formal schemas, not Markdown. The moment MACH Open Data Model documents become an input to your build, three engineering questions appear: can the conversion run unattended, will the same input always produce the same output, and what happens when one file in the batch is broken?

The CoreModels transform surface was built with those questions as requirements. This article shows the pipeline patterns we use for ODM: batch conversion with per-file failure isolation, lossiness gates in CI, dry-run-then-write imports, and the mapping plan as a committed, replayable artifact.

## Determinism is the foundation

Every transform in CoreModels executes deterministically: the same source document produces the same output document, byte for byte. That single property is what makes the rest of this article possible. Converted schemas can live in git next to the entity documents they came from, a nightly job can regenerate them, and `git diff` becomes your change detector - an upstream edit to an entity document shows up as a reviewable schema diff, and an empty diff means nothing changed. No timestamps, no reordered keys, no noise.

## Batch conversion with failure isolation

The converter endpoint takes multiple entities in one call. The body is `entities: [{ name, markdown }, ...]`; the response is a `schemas` bundle keyed by kebab-cased entity title. Building the body from a directory of Markdown files is a few lines of shell:

```bash
#!/usr/bin/env bash
set -euo pipefail

BODY=$(jq -n '{ entities: [] }')
for f in entities/*.md; do
  BODY=$(jq --rawfile md "$f" --arg name "$(basename "$f" .md)" \
    '.entities += [{ name: $name, markdown: $md }]' <<< "$BODY")
done

curl -s -X POST \
  "https://coremodels.example.com/graph/transform/odm/convert/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data "$BODY" > bundle.json
```

The failure semantics are the point. Files that are not entity documents at all (a README with no `## YAML Schema Definition` section) are simply skipped. A file that *should* convert but cannot - say the section exists but contains no fenced YAML block - does not poison the batch. It is reported in the lossiness ledger as a `StructuralDrop` naming the file, and every other entity still converts:

```json
{
  "success": true,
  "lossiness": [
    {
      "kind": "StructuralDrop",
      "path": "pricing",
      "explanation": "Entity conversion failed and the file was skipped: The section contains no fenced yaml code blocks."
    }
  ],
  "errors": [],
  "schemas": {
    "product": { "...": "..." },
    "inventory": { "...": "..." }
  }
}
```

Only when *zero* entities convert does the call fail outright, with per-file errors. The error channel means "could not proceed"; per-file trouble rides the reported-and-continue channel. Splitting the bundle back into files is one `jq` loop:

```bash
for key in $(jq -r '.schemas | keys[]' bundle.json); do
  jq ".schemas[\"$key\"]" bundle.json > "schemas/$key.schema.json"
done
```

## Gate on the ledger, not just the status code

`success: true` means the run completed - the lossiness array is where fidelity lives. In a pipeline, that array is machine-readable policy input. A strict gate that fails the build if any entity was dropped:

```bash
jq -e '[.lossiness[] | select(.kind == "StructuralDrop")] | length == 0' \
  bundle.json > /dev/null || { echo "ODM batch had dropped entities"; exit 1; }
```

The four lossiness kinds (`StructuralDrop`, `TypeApproximation`, `ConstraintRelaxation`, `SemanticNarrowing`) support graduated policy: fail on drops, warn on narrowing (a dangling `$ref` in an entity document surfaces as `SemanticNarrowing` at its exact path), and log the rest. Because every record carries `kind`, `path`, and `explanation`, the failure message in your CI log already says which file, which construct, and why.

## Dry-run in the PR, write on merge

For pipelines that import ODM into a CoreModels project rather than just converting it, the import endpoint's dry-run contract maps cleanly onto CI stages. In the pull-request job, a **Viewer** token is enough, and nothing is written:

```bash
jq -n --rawfile md entities/product.md '{ markdown: $md, dryRun: true }' \
| curl -s -X POST \
  "https://coremodels.example.com/graph/transform/odm/import/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data @- | tee dryrun.json

jq -e '.success and (.lossiness | length == 0)' dryrun.json > /dev/null
```

The dry-run returns the would-be summary - `{ "types": n, "elements": n, "taxonomies": n }` - plus the full ledger, so reviewers see exactly what a merge will create before it creates anything. The merge job then repeats the call with `"dryRun": false` under an **Admin** token, writing the schema with its provenance stamped `odm`. Same body, two stages, least privilege at each.

## The plan is a pipeline artifact

Conversion is only half the automation story; the other half is mapping - aligning ODM entities onto a target shape your systems already speak. Here the engine's design pays off in a specific way: every mapping strategy produces a **plan**, every plan passes the same universal validation gate, and the executed plan comes back in the response as a replayable artifact.

That enables a workflow we recommend for any recurring ODM mapping:

1. **Once, interactively:** call `POST /graph/transform/schema/map/{projectId}` with `sourceFormat: "odm"`, your target, and a mapping kind. Review the returned `plan`, then commit it to the repository like any other build input. `schema/map` writes nothing - every call is inherently a dry run.
2. **Every run thereafter:** replay the committed plan with `POST /graph/transform/plan/execute/{projectId}`, passing the plan JSON as a string:

```bash
jq -n --rawfile md entities/product.md --rawfile plan plans/product-plan.json '{
  sourceFormat: "odm",
  sourceSchema: $md,
  plan: $plan,
  targetFormat: "jsonschema"
}' \
| curl -s -X POST \
  "https://coremodels.example.com/graph/transform/plan/execute/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data @-
```

A stored plan earns no shortcut - it re-enters the same validation gate on every execution - but once validated it executes deterministically: same plan plus same source equals same output, a property proven by test. Your pipeline's mapping behavior is now versioned, diffable, and reviewable in the same pull request as the entity documents themselves.

## Choosing a mapping kind for unattended runs

The three mapping kinds have distinct automation profiles when the source is ODM:

- **`inferred`** - label and type matching against a target hint, `caseInsensitive` by default. Zero configuration, ideal for the first pass and for targets whose field names track the ODM entity's. Deterministic, Viewer-role, safe on a schedule.
- **`explicit`** - an authored SIA mapping-guide JSON (`fieldMappings` with `sourceElementIds`/`targetElementIds`/`transformName`, `taxonomyDirectives`, `drops`, `autoMatchByMapsTo`). This is the production-pipeline kind: the guide is a committed file, and the parser rejects unknown keys with a path-carrying error - a typo in a guide fails the build instead of silently executing a plan you did not intend.
- **`ai`** - a server-side model proposes the plan, and the identical gate validates it with at most one repair attempt; a rejected repair is a rejection, and the proposal's self-reported confidence is advisory only - the gate never trusts it. It requires Editor or Admin project membership and a server-configured Anthropic key, and it sends schema content to the Anthropic API server-side. Our recommendation for pipelines: use `ai` interactively to *draft* a mapping, review the plan it produces, commit that plan - and let `plan/execute` do the unattended runs. The creativity happens once, under review; the automation replays a validated artifact.

## The shape of a good ODM pipeline

Pull the entity documents, batch-convert, gate on `StructuralDrop`, diff the regenerated schemas, dry-run the import in review, write on merge, and replay committed plans for any onward mapping. Every step is stateless or dry-run until the one that is not, every step returns the same envelope, and every fidelity decision the engine made is on the record. That is what it takes for authored documentation to become build input you can trust.

For ready-to-paste request bodies and the complete endpoint reference, see the CoreModels transform API guide in the product docs.