# Protocol Buffers — Deep dive: Inside the proto3 Coder: The IR Mapping, the Extras, and Every Lossiness Record

Proto3 has fifteen scalar types. A neutral intermediate representation that also hosts JSON Schema, SQL, Avro, LinkML, OWL, and half a dozen other formats cannot afford fifteen distinct numeric primitives, and any tool that claims otherwise is hiding something. The arithmetic settles the question up front: some proto3 facts map structurally, some ride alongside as preserved detail, and some are declared as loss. This is the complete account, in that order.

# Inside the proto3 Coder: The IR Mapping, the Extras, and Every Lossiness Record

Proto3 has fifteen scalar types. A neutral intermediate representation that also hosts JSON Schema,
SQL, Avro, LinkML, OWL, and half a dozen other formats cannot afford fifteen distinct numeric
primitives, and any tool that claims otherwise is hiding something. The arithmetic settles the
question up front: some proto3 facts map structurally, some ride alongside as preserved detail, and
some are declared as loss. This is the complete account, in that order.

Import is decode (proto3 text to IR), export is encode (IR to proto3 text), and the scope is a
**single `.proto` file** - we do not resolve an include graph. The format key is `protobuf`, alias
`proto`, valid in both directions.

## What maps structurally

| proto3 construct | IR construct | Notes |
|---|---|---|
| `message` | Type | id from the full name, label = the simple name |
| field | Element | id = type id + capitalized field name; label = the wire name |
| nested `message` | its own Type | enclosing type recorded, re-nested on encode |
| `enum` | Taxonomy | values become terms in order, with their numbers |
| enum value | term | label = the wire spelling |
| `repeated` | collection cardinality | item type becomes the element's value type |
| explicit `optional` | non-required element | the label is remembered and re-emitted |
| plain scalar / enum field | **required** element | proto3 implicit presence: it always carries a value |
| message-typed field | non-required element | message fields track presence |
| `google.protobuf.Timestamp` | DateTime primitive | the one well-known type treated specially |
| message/enum reference | type / taxonomy reference | resolved by proto3 scoping rules |
| `oneof` | optional elements + a group marker | exclusivity declared as loss, group reassembled on encode |
| `map<K,V>` | String element + key/value kept | declared as loss |
| `package` | schema id and label | `acme.orders` becomes id `acmeOrders`, label `acme.orders` |
| `service`, `extend`, `extensions`, `group` | - | skipped, declared as loss |

Reference resolution follows the specification rather than approximating it. Every message and enum
full name is registered before any field type is resolved, so forward references and nested
references bind correctly. A leading dot means fully qualified; otherwise the enclosing scopes are
walked innermost-out. A name that resolves to nothing in this file - an imported or unknown type -
becomes a String with its verbatim token preserved and a record filed.

## The scalars

| proto3 | IR primitive | Recorded as an approximation? |
|---|---|---|
| `int32`, `int64` | Integer | no |
| `uint32`, `uint64`, `sint32`, `sint64` | Integer | yes - unsignedness and zigzag encoding are not modeled |
| `fixed32`, `fixed64`, `sfixed32`, `sfixed64` | Integer | yes - fixed-width encoding is not modeled |
| `float`, `double` | Double | no |
| `bool` | Boolean | no |
| `string` | String | no |
| `bytes` | String | yes - proto3 `bytes` has no IR equivalent |

Encoding IR that never saw a `.proto`, the defaults invert: Integer becomes `int64`, Double
`double`, Boolean `bool`, and everything else - String, DateTime, rich text - becomes `string`. The
last two are recorded: "proto3 has no date-time scalar; DateTime degrades to string" and "proto3 has
no rich-text scalar; RichText degrades to string." A definition that *did* arrive from proto3 skips
that degradation entirely, because its original token was preserved. That is the whole purpose of
the next section.

## The extras channel

Every IR node carries an annotation bag. The coder writes each proto3 fact the IR has no slot for
into it under a dotted `protobuf.*` key, and encode reads them back:

| Key | On | Carries |
|---|---|---|
| `protobuf.package` | schema | the package statement |
| `protobuf.imports` | schema | the import clauses verbatim, as a JSON array |
| `protobuf.option.<name>` | schema, type, taxonomy | the option constant, verbatim |
| `protobuf.parent` | type, taxonomy | the enclosing message's type id |
| `protobuf.reserved` | type, taxonomy | `reserved` statement bodies, as a JSON array |
| `protobuf.type` | element | the verbatim wire type token |
| `protobuf.fieldNumber` | element | the field number |
| `protobuf.optional` | element | `"true"` when the `optional` label was explicit |
| `protobuf.oneof` | element | the enclosing `oneof`'s name |
| `protobuf.mapKey` / `protobuf.mapValue` | element | the `map<K,V>` key and value types, verbatim |
| `protobuf.fieldOptions` | element | bracketed field options, verbatim |
| `protobuf.enumNumber` | term | the enum value's number |
| `protobuf.valueOptions` | term | bracketed enum-value options, verbatim |

Decoding an order schema, `sequence` becomes an Integer element carrying
`protobuf.fieldNumber = "6"` and `protobuf.type = "uint64"`; `total` carries
`protobuf.fieldOptions = json_name = "orderTotal"`; the `Order` type carries
`protobuf.option.deprecated = true` and `protobuf.reserved = ["100 to 199","\"legacy_id\""]`. Encode
re-emits every one of them exactly.

This is also the boundary worth knowing: the extras channel lives in the transform, and a project's
schema store does not persist it. Round trips through the stateless routes keep the wire facts;
round trips through a governed project keep the meaning and mint fresh field numbers.

## Ids, labels, and names

CoreModels node ids must be alphanumeric, so ids are sanitized while the original spelling is kept on
the label. The rules, and their visible consequences:

- A package `acme.orders` yields schema id `acmeOrders`, label `acme.orders`.
- A nested message `Order.LineItem` yields type id `OrderLineItem`, label `LineItem`, plus
  `protobuf.parent = "Order"`.
- A field id is its type id plus the capitalized, sanitized field name: `order_id` in `Order`
  becomes `OrderOrderId`, label `order_id`.
- Ids are unique across types, elements, and taxonomies, because relation endpoints resolve by bare
  id. A collision gets a numeric suffix - in the order schema the nested `enum Status` claims
  `OrderStatus` first, so the `status` *field* becomes `OrderStatus2`. The label is still `status`.
- Enum term ids drop non-alphanumerics: `SHIPMENT_STATUS_UNSPECIFIED` becomes id
  `SHIPMENTSTATUSUNSPECIFIED` with the label holding the exact spelling. Targets that emit ids - JSON
  Schema `enum` arrays, for instance - show the stripped form; the proto3 encoder emits labels, so a
  proto-to-proto trip restores the underscores. If exact symbols matter downstream, check this line.

On the encode side, names come from labels, disambiguated per nesting scope, and forced into valid
proto3 identifiers. A label that is not a legal identifier is sanitized and declared: `body ref`
becomes `body_ref`. A label that would be mis-read in type position - a statement keyword, a field
label, a range word, or one of the fifteen scalar names - gains a trailing underscore: a type
labeled `message` emits as `message_`, and a reference to it emits `message_ body_ref = 1;`, which
re-decodes to the same binding.

One consequence to internalize when authoring plans: the emitted field name comes from the element's
**label**, not its id. Renaming a target element id in a plan does not rename the wire field.

## Field numbers

Preserved numbers re-emit verbatim. Elements without one are minted the next free number in
declaration order, skipping numbers already taken and the implementation-reserved 19000–19999 range.
So a message whose fields are 1 and 4, plus one freshly added element, emits:

```proto
syntax = "proto3";

message Sparse {
  string a = 1;
  string b = 4;
  optional int64 c = 2;
}
```

`c` takes 2 - the lowest free number - rather than 5, and it carries `optional` because the element
was not required. Growth is wire-safe by construction, not by remembering to be careful.

## The complete lossiness inventory

**Decode side.** Every one of these is a record, not a failure:

| Situation | Kind | Path |
|---|---|---|
| an `import` clause | SemanticNarrowing | `imports` |
| a `oneof` group | SemanticNarrowing | `<Message>.<oneofName>` |
| an option inside a `oneof` | StructuralDrop | `<Message>.<oneofName>` |
| `map<K,V>` | SemanticNarrowing | `<Message>.<field>` |
| `uint*`, `sint*`, `fixed*`, `sfixed*` | TypeApproximation | `<Message>.<field>` |
| `bytes` | TypeApproximation | `<Message>.<field>` |
| a type this file does not define | TypeApproximation | `<Message>.<field>` |
| `service` / `extend` at file level | StructuralDrop | `service <Name>` |
| `extensions` / `extend` / `group` in a message | StructuralDrop | the message's full name |
| no `syntax` statement | SemanticNarrowing | `$` |

The last one is worth quoting in full, because it states an assumption most tools make silently:
"No syntax statement: the file defaults to proto2 per the spec; it was decoded with proto3
semantics."

**Encode side**, for IR constructs proto3 cannot express:

| Situation | Kind |
|---|---|
| an IR component (a curated view) | StructuralDrop |
| an IR relation instance | SemanticNarrowing |
| type inheritance | SemanticNarrowing - no inheritance; inherited fields are not repeated |
| a term hierarchy | SemanticNarrowing - enum values are flat |
| an element belonging to no type | StructuralDrop - proto3 has no top-level fields |
| a nullable element | ConstraintRelaxation - proto3 does not distinguish null from absent |
| collection min/max bounds | ConstraintRelaxation - `repeated` carries no item bounds |
| DateTime or rich text without a preserved token | TypeApproximation |
| a required message-typed field | ConstraintRelaxation - message fields always track presence |
| a name that had to be escaped or sanitized | SemanticNarrowing |
| a taxonomy with no terms | SemanticNarrowing - a `UNSPECIFIED = 0;` placeholder is minted |

That last case exists because proto3 enums require at least one value; the placeholder keeps the
output compilable and the record keeps it honest.

## Round-trip fidelity

The coder's own test suite decodes a schema exercising nested messages, an enum with a gap in its
numbering, a `oneof`, a `map`, a bracketed field option, message options, two `reserved` forms, and a
`service` block - then encodes it, decodes the result again, and asserts that the structural
signature of the two IR schemas is identical and that both validate. The emitted text carries
`reserved 100 to 199;`, `optional string note = 2;`, `repeated Order.LineItem items = 3;`,
`google.protobuf.Timestamp placed_at = 8;`, `map<string, string> labels = 9;`, the reassembled
`oneof payment {`, `double total = 12 [json_name = "orderTotal"];`, and `CANCELLED = 5;` - the gap in
the enum numbering preserved rather than renumbered.

Three details of emitted layout, all deliberate:

- References to nested definitions are emitted as dotted full names - `Order.LineItem`,
  `Order.Status` - so a re-decode binds to the same definition regardless of scope.
- Inside a message the order is options, `reserved`, fields (with `oneof` groups reassembled in
  place), nested enums, then nested messages.
- Encoding is a fixed point: encode, decode, encode again, and the text is unchanged. For a
  comment-free file already in the encoder's canonical layout, the first encode is byte-identical
  to the input. Comments are consumed by the tokenizer and never come back.

## Edge cases worth knowing

**Where it refuses.** Only genuinely malformed or non-proto3 input fails: `Only proto3 is supported;
the file declares syntax "proto2".`, `'required' fields are proto2; only proto3 is supported.`, `The
protobuf schema is empty.`, `The text is not a protobuf schema (no syntax, package, message, or enum
statement was found).`, and parse errors that name the token, such as `Expected ';' but found '}'.`
Everything else decodes as far as it can.

**Nesting cycles.** Encode rebuilds nesting from `protobuf.parent`. If a chain of parents forms a
cycle - possible only in hand-assembled IR - the type is emitted at the top level instead of
recursing forever.

**oneof round trips.** Members are flattened into optional elements carrying `protobuf.oneof`, and
encode regroups them under the original name at the position of the first member. The exclusivity
constraint is not modeled anywhere, which is exactly what the SemanticNarrowing record says.

**Enum value options and negative numbers.** Both survive: value options ride in
`protobuf.valueOptions`, and the parser accepts a leading minus on a value number.

**Duplicate term names.** If two enum values sanitize to the same id, the second gets a numeric
suffix; the labels stay distinct, so the emitted file is unchanged.

The transform section of the CoreModels documentation covers the routes, roles, and mapping
vocabulary that surround this coder.