# Protocol Buffers — Automation: Deterministic proto3 Pipelines: The Plan Is the Artifact A conversion you run once is a favor. A conversion you run on every commit is infrastructure, and infrastructure needs two properties that one-off tooling never has to prove: the second run must produce the same bytes as the first, and when the input changes in a way the conversion cannot honor, the pipeline must stop rather than quietly produce something different. # Deterministic proto3 Pipelines: The Plan Is the Artifact A conversion you run once is a favor. A conversion you run on every commit is infrastructure, and infrastructure needs two properties that one-off tooling never has to prove: the second run must produce the same bytes as the first, and when the input changes in a way the conversion cannot honor, the pipeline must stop rather than quietly produce something different. The CoreModels transform surface is built on both. This article shows how to turn proto3 conversions into pipeline steps - deterministic outputs, plans as committed artifacts, drift that fails the build, and a batch pattern for a directory of `.proto` files. ## Property one: the same input produces the same bytes The proto3 encoder writes its output by hand in a fixed order - syntax, package, imports, file options, then messages and enums, with each message emitting its options, `reserved` statements, fields, nested enums, and nested messages in that sequence. Nothing depends on dictionary iteration order or on the wall clock. Decode a file and encode it twice and you get identical text; encode the result of decoding your own output and you get the same text again. It is a fixed point, not merely a repeatable function. That property is what makes generated artifacts safe to commit. Check the produced JSON Schema or `.proto` into the repository next to the source, regenerate in CI, and `git diff --exit-code` becomes a meaningful gate: a non-empty diff means a real change to the schema, never churn from the generator. One nuance for `.proto` sources specifically. A comment-free file that already follows the encoder's canonical layout survives a proto3 round trip byte-for-byte, including package, imports, options, `reserved` statements, field numbers, `optional` labels, `oneof` groupings, and enum numbers. Comments and bespoke formatting do not come back - the tokenizer consumes them and they are not part of the schema - so if your baseline file carries a license header or its own layout, compare the regenerated file against a regenerated baseline rather than against the hand-written original. ## Property two: the plan is a first-class artifact Every mapping call returns the plan it executed. For an identity conversion of a shipment event schema with seven fields, that is nine operations - one type mapping, one element mapping per field, one taxonomy mapping - each stamped with where it came from. Four of them: ```json { "operations": [ { "kind": "TypeMapping", "origin": "Inferred", "sourceTypeId": "ShipmentEvent", "targetTypeId": "ShipmentEvent", "targetLabel": "ShipmentEvent" }, { "kind": "ElementMapping", "origin": "Inferred", "sourceElementIds": ["ShipmentEventShipmentId"], "targetElementIds": ["ShipmentEventShipmentId"] }, { "kind": "ElementMapping", "origin": "Inferred", "sourceElementIds": ["ShipmentEventSignature"], "targetElementIds": ["ShipmentEventSignature"] }, { "kind": "TaxonomyMapping", "origin": "Inferred", "sourceTaxonomyId": "ShipmentStatus", "targetTaxonomyId": "ShipmentStatus", "targetTreatment": "InlineEnum" } ] } ``` `"origin": "Inferred"` means a heuristic matched it by label and compatible type. That is a fact worth having in code review: the mapping is not hidden inside a script, it is a document you can read, diff, and approve. Save it and replay it. `plan/execute` takes the same source, the plan **as a string**, and a target format: ```bash jq -n --rawfile p shipment_event.proto --rawfile q plans/shipment_event.plan.json '{ sourceFormat: "protobuf", sourceSchema: $p, plan: $q, targetFormat: "protobuf" }' > replay.json curl -sS -X POST \ "https://coremodels.example.com/graph/transform/plan/execute/$PROJECT_ID" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ --data-binary @replay.json > replayed.json ``` A stored plan earns no shortcut: it passes the identical validation gate as a freshly produced one before anything executes. Replay the same plan against the same source as many times as you like and the output is byte-identical every time. ## Editing the plan: a curated projection Because the target is exactly what the plan projects, editing the plan is how you author a projection - no script, no template, no second copy of the schema. Take the identity plan above and delete the one operation that maps `ShipmentEventSignature`. Replay it. The emitted `.proto` is the original minus that field: ```proto syntax = "proto3"; package acme.logistics; import "google/protobuf/timestamp.proto"; message ShipmentEvent { string shipment_id = 1; uint64 sequence = 2; ShipmentStatus status = 3; optional string carrier_note = 4; repeated string package_ids = 5; google.protobuf.Timestamp occurred_at = 6; } enum ShipmentStatus { SHIPMENT_STATUS_UNSPECIFIED = 0; PICKED_UP = 1; IN_TRANSIT = 2; DELIVERED = 3; } ``` The surviving fields keep their original numbers - 1 through 6 - because those numbers were preserved from the source, and a removed field's number is simply not re-used. And the omission is not silent. The ledger gains: ```json { "kind": "StructuralDrop", "path": "Type[ShipmentEvent].ShipmentEventSignature", "explanation": "Element is a member of the mapped type but no operation maps or drops it." } ``` That message is the engine's rule stated out loud: a field that belongs to a mapped type but has no operation is reported, never quietly omitted. If you meant to drop it, the record is your receipt. If you did not, it is your bug report. The same rule cuts the other way, and it is the one thing to know before authoring a plan by hand: **only what the plan maps reaches the target.** A plan with element mappings but no `TypeMapping` produces a valid `.proto` with a package and no messages, because no message was ever mapped. When you write operations yourself, map the type first. ## Drift that fails the build Here is the payoff that ad-hoc conversion scripts cannot give you. Store a plan. Six weeks later somebody renames a field in the source `.proto`. Your pipeline replays the stored plan, and the gate refuses it before the engine runs: ```jsonc { "success": false, "lossiness": [ /* the source-decode ledger still appears here */ ], "errors": [ { "path": "plan.Operations[1](ElementMapping).SourceElementIds", "message": "Source element 'ShipmentEventNoSuchField' does not resolve." } ], "schema": null } ``` The path names the operation index and the operation kind; the message names the construct that vanished. A script would have produced an artifact missing a field and exited zero. This exits with `success: false` and tells you which field and which operation, which is a five-minute fix instead of a downstream incident. The gate also rejects a plan that produces the same target id twice, a plan that both maps and drops the same source node, a named field transform that is not registered, and an element mapping whose arity or types do not line up. Every strategy passes through it - inferred, explicit, AI-proposed, and stored alike. ## The three mapping kinds, applied to proto3 - **`inferred`** (the default) matches source constructs to a target hint by label and compatible type. It suits proto3 unusually well, because a field's label *is* its wire name: `shipment_id` matches `shipment_id`. `caseInsensitive` defaults to `true`, which is what you want when snake_case proto fields meet a camelCase or upper-case target. Every operation it emits is stamped `Inferred`. - **`explicit`** takes a SIA mapping guide: `autoMatchByMapsTo`, `fieldMappings` (`{sourceElementIds, targetElementIds, transformName}`), `taxonomyDirectives`, and `drops`. It is the tool for the cases inference cannot express - joining two fields into one, splitting one into two, forcing a taxonomy treatment, deleting something deliberately. A multi-field `fieldMappings` entry must name a transform from the registry; a plain one-to-one entry may omit it. Its automatic backbone is `mapsTo`, the shared cross-standard URI two constructs declare; a bare `.proto` file carries none, so pair an explicit guide with a hint whose constructs are annotated, or author the type alignment as plan operations. - **`ai`** asks a server-side proposer for a plan and then puts it through the same gate, with at most one repair attempt. It requires Editor or Admin membership and a server-configured key, and it sends the schema content to an external model API - a deliberate choice for a proprietary `.proto`, and one the tooling advertises rather than hides. Whichever kind produced it, what you store and replay afterwards is the plan. That is the point of having one plan format. ## Batch conversion, end to end A directory of `.proto` files, one call each, artifacts and ledger collected: ```bash #!/usr/bin/env bash set -euo pipefail HOST="https://coremodels.example.com" mkdir -p build plans : > build/lossiness.tsv for f in schemas/*.proto; do name=$(basename "$f" .proto) jq -n --rawfile p "$f" '{ sourceFormat: "protobuf", sourceSchema: $p, targetFormat: "jsonschema", targetHintFormat: "protobuf", targetHintSchema: $p, mapping: { kind: "inferred" } }' > "build/$name.request.json" curl -sS -X POST "$HOST/graph/transform/schema/map/$PROJECT_ID" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ --data-binary @"build/$name.request.json" > "build/$name.result.json" jq -e '.success' "build/$name.result.json" > /dev/null jq '.schema' "build/$name.result.json" > "build/$name.schema.json" jq '.plan' "build/$name.result.json" > "plans/$name.plan.json" jq -r --arg n "$name" \ '.lossiness[] | "\($n)\t\(.kind)\t\(.path)\t\(.explanation)"' \ "build/$name.result.json" >> build/lossiness.tsv done ``` Then gate on the ledger rather than on exit codes alone. Different teams draw the line in different places; a strict one fails on anything structural and lets type approximations through with a warning: ```bash if cut -f2 build/lossiness.tsv | grep -qx StructuralDrop; then echo "A structural drop appeared in a conversion - review build/lossiness.tsv" >&2 exit 1 fi cut -f2 build/lossiness.tsv | sort | uniq -c ``` For a directory of proto3 files, that count is usually dominated by two entries you should expect and can whitelist: unresolved `import` clauses, and `TypeApproximation` records for `uint32`, `uint64`, `sint32`, `sint64`, the fixed-width encodings, and `bytes`. Those are the wire facts no other format carries. What you are watching for is the record that appears for the first time - a new `oneof` flattened into optional fields, a `map` approximated as a string, a `service` block dropped, or a `StructuralDrop` naming a field your plan forgot. Two closing notes on pipeline hygiene. Plans are source-specific: element ids derive from message and field names, so a plan belongs to the file it was produced from - store it beside that file, not in a shared directory. And a plan's replay is only as stable as the source it names, which is precisely why the gate checks before the engine runs. The transform section of the CoreModels documentation has the endpoint list, the role each route needs, and the full mapping-guide vocabulary.