# AWS Glue — Deep dive: What Happens to a Glue Table on Its Way into the Graph

Take one entry out of an `aws glue get-tables` response - a table called `events` in database `lake`, a few columns, one partition key, a `classification` parameter, an S3 location - and follow it into CoreModels. By the end it is a governed Type with Elements, a vendor identity, a metadata mixin, an entry in an estate snapshot, and a set of audit rules watching it. This is that journey, plus the honest edges: what is approximated, what is deliberately not invented, and which behaviors will surprise you.

# What Happens to a Glue Table on Its Way into the Graph

Take one entry out of an `aws glue get-tables` response - a table called `events` in database
`lake`, a few columns, one partition key, a `classification` parameter, an S3 location - and
follow it into CoreModels. By the end it is a governed Type with Elements, a vendor identity, a
metadata mixin, an entry in an estate snapshot, and a set of audit rules watching it. This is
that journey, plus the honest edges: what is approximated, what is deliberately not invented, and
which behaviors will surprise you.

## One neutral estate model

Every CoreModels connector parses its vendor's artifacts into the same neutral shape: datasets
carrying fields and normalized checks, plus lineage edges between datasets. Vendor-specific
detail that has no neutral slot rides in `Meta` bags rather than becoming new top-level concepts.
The Glue connector itself is pure - it parses, maps native types, contributes audit rules and
generates artifacts, and never touches the graph. All graph I/O lives in vendor-neutral services,
which is why the Glue mapping behaves like every other connector's once parsing is done.

## Parsing: what the connector reads

The `tables` artifact may arrive as a single `{"TableList": […]}` response, an array of such
responses, or a bare array of table objects; the collector walks all three. Here is the input,
trimmed to the keys the connector actually reads:

```json
{
  "Name": "events",
  "DatabaseName": "lake",
  "Description": "Raw product events landed from Kinesis.",
  "TableType": "EXTERNAL_TABLE",
  "Parameters": { "classification": "parquet" },
  "StorageDescriptor": {
    "Location": "s3://example-lake/events/",
    "Columns": [
      { "Name": "event_id", "Type": "string", "Comment": "Event key." },
      { "Name": "user_id", "Type": "bigint" },
      { "Name": "payload", "Type": "struct<action:string,value:double>", "Comment": "Raw payload." }
    ]
  },
  "PartitionKeys": [
    { "Name": "event_date", "Type": "date", "Comment": "Partition." }
  ]
}
```

A table object needs a `Name` to be considered at all. From each one:

- **Identity** is `DatabaseName` and `Name` joined with a dot - `lake.events`. Empty segments are
  skipped, so a table object without `DatabaseName` gets the bare name as its identity.
- **Kind** comes from `TableType` (default `EXTERNAL_TABLE`): a value containing `VIEW` becomes a
  view with materialization `view`, anything else a table with materialization `external_table`.
- **Physical name** is `StorageDescriptor.Location`, falling back to the identity when absent.
- **Classification** is the `classification` entry from `Parameters`, lifted into the dataset's
  meta bag as `glue.classification`.
- **Fields** are `Columns` followed by `PartitionKeys`, in that order, de-duplicated
  case-insensitively (first wins). Each keeps its `Comment` and its `Type` **verbatim**;
  partition keys additionally carry `glue.partitionKey` in their own meta bag.

The whole artifact is hashed into a 16-character fingerprint at parse time; that value travels
into audit reports, history records and the stored snapshot, and is what proves two runs saw
byte-identical content. Two refusals are explicit rather than silent: an artifact that is not
valid JSON, and one that parses but yields no named tables.

## Two write paths, one import

Schema shape - types, fields, enums, references - travels through the intermediate representation
the whole CoreModels transform stack shares, written by the same hardened graph writer every
format import uses. Estate facts it cannot hold - lineage, checks, provenance, last-import state
- are written natively by the integration enricher. For Glue the first path does little beyond
structure, because Glue declares little:

| Estate fact | Becomes | Notes for Glue |
|---|---|---|
| table | governed Type | label = table name |
| column / partition key | Element on that Type | in catalog order |
| accepted values | Taxonomy + controlled list | never - Glue declares none |
| foreign key / reference | expected-type relation | never - Glue declares none |
| not-null check | required flag | never - Glue declares none |
| vendor identity | `mapsTo { standard: "glue", uri: "lake.events" }` | queryable, survives exports |
| native type, description, tags, meta, materialization, physical name | vendor-metadata mixin | see below |
| lineage edge | `Depends On` relation | never - `get-tables` declares no dependencies |

That table is the connector's design rule in one view: **preserve verbatim, never invent.** A
governed model built from Glue has no primary keys and no nullability, because the catalog
declares none - a guessed constraint would poison every audit that followed.

## Node ids, labels and identity

CoreModels node ids must be camelCase alphanumerics, so identities are sanitized: `lake.events`
becomes the Type id `lakeEvents`, and its `event_id` column the Element id `lakeEventsEventId`
(dataset id plus the PascalCased field name - elements are per-dataset). The native names survive
on the node's label, its `mapsTo` value and the mixin, and audit, reconciliation and re-import
all join on the `mapsTo` identity - never on the id or the label.

Sanitization folds separators away, so two distinct catalog identities could in principle
collapse onto one id. Rather than silently merging two tables, the import refuses and names both
sources: *"Distinct vendor identities collapse onto the same graph id after sanitization …
Rename one of them in the vendor estate."*

## The type map

Alongside the verbatim Hive type, each Element gets a governed primitive so the graph can reason
across vendors:

| Hive type | Governed primitive | Approximated? |
|---|---|---|
| `int`, `integer`, `bigint`, `smallint`, `tinyint` | Integer | no |
| `double`, `float` | Double | no |
| `decimal(p,s)` | Double | **yes** - precision and scale are not carried |
| `boolean` | Boolean | no |
| `timestamp`, `date` | DateTime | no |
| `string` | String | no |
| `char(n)`, `varchar(n)` | String | **yes**, when a length is declared |
| `binary`, `array<…>`, `map<…>`, `struct<…>`, `uniontype<…>` | String | **yes** |

Every approximation is reported as a `TypeApproximation` lossiness record on the import response,
naming the column: *"Native type 'struct<action:string,value:double>' was approximated as String;
the exact native type is preserved in the vendor metadata mixin."* That is also why generated DDL
re-emits `decimal(19,4)` and `struct<…>` exactly: the primitive is the cross-vendor working type,
the recorded Hive string remains the authority.

## What rides the vendor-metadata mixin

Every imported node gets one mixin value on a mixin type called **AWS Glue Data Catalog
Metadata**; property ids are the vendor key plus the property name. On a Type:
`glueUniqueId` (the catalog identity), `glueResourceKind`, `glueMaterialization`,
`glueRelationName` (the S3 location), `glueContract`, `glueAccess`, `glueDescription`,
`glueTags`, `glueMeta` (the meta bag as JSON - this is where `glue.classification` lives), and
`glueChecks`. On an Element: `glueDataType` (the verbatim Hive type), `glueDescription`,
`glueTags`, `glueMeta` (where `glue.partitionKey` lives) and `glueChecks`.

Note `glueDescription` in both lists: CoreModels has no schema-level description slot, so the
catalog's `Description` and each column's `Comment` ride the mixin and are read back from there
by generation and audit.

These values are the one thing an import *refreshes* rather than merely adds: they describe the
estate, not the meaning. Governed nodes are never mutated by an import. That asymmetry is the
posture in one line - import is additive, drift is the audit's job, and applying a meaning change
is a human act.

Two more nodes appear beside the model. A **state node**, labeled *AWS Glue Data Catalog
Integration - lake*, records the last import: vendor, project name, timestamp, fingerprint,
counts and parser facts. An **estate snapshot node** stores the parsed catalog itself,
gzip-compressed and base64-encoded, so re-audits can run without fresh artifacts. It is
size-capped (roughly 1.5 MB encoded); over the cap, import reports `snapshotStored: false` with a
lossiness record rather than failing.

## The rules that watch it afterwards

Audits run in three sections against the live governed graph - no stored fingerprints to go
stale, the graph is the baseline.

**Coverage** - `dataset-unmapped` (Warning: a catalog table nothing governs) and `field-unmapped`
(Info: a column added since the last import).

**Drift** - `dataset-removed` (Error), `field-removed` (Error), `field-type-drift` (Error),
`enum-narrowed` (Error), `enum-widened` (Warning), `enum-constraint-removed` (Warning) and
`contract-drift`. A drift finding names both sides:

```json
{ "section": "Drift", "severity": "Error", "code": "field-type-drift",
  "subject": "lake.events.user_id",
  "message": "Field type changed since the last import.",
  "detail": "governed: bigint, estate: string" }
```

**Conformance** - the three rules the Glue connector contributes:

| Code | Severity | Fires when |
|---|---|---|
| `table-no-description` | Info | the catalog table has no description |
| `classification-missing` | Info | a table (views are exempt) has no `classification` parameter, so crawlers and Athena treat the format as unknown |
| `semi-structured-column` | Warning | the table has `struct` / `map` / `array` / `uniontype` columns - one aggregate finding per table, column names in the detail |

Aggregating that last one is deliberate: a table with a 40-field struct is one governance
decision, not 40 rows of noise.

## Generation, in reverse

Generation emits `coremodels_glue_tables.sql` - Athena/Hive `CREATE EXTERNAL TABLE IF NOT EXISTS`
statements. Each column prefers its recorded native type and falls back to the primitive mapping
(Integer → `bigint`, Double → `double`, Boolean → `boolean`, DateTime → `timestamp`, otherwise
`string`) for governed elements that never came from Glue; reference-typed elements are emitted
as `bigint`. Descriptions and taxonomy allowed-values compose into column `COMMENT`s,
partition-flagged elements return to `PARTITIONED BY`, tables are `STORED AS PARQUET`, and the
`LOCATION` is a placeholder - the model knows your schema, not your bucket layout. Views and
element-less types are skipped as declared `StructuralDrop` lossiness.

## The edges worth knowing

- **Governing a taxonomy on a Glue column produces a standing warning.** Glue never declares
  accepted values, so once an element carries a governed taxonomy, every audit reports
  `enum-constraint-removed`. That is literally true, and for a Glue-only estate it is permanent.
  Expect it, and gate on codes rather than raw warning counts.
- **`contract-drift` is inert here.** Glue has no contract-enforcement flag, so both sides of the
  comparison are always `false`.
- **`dataset-removed` is namespace-scoped.** It only fires for governed datasets whose database
  prefix appears in the export being audited, so auditing one database never claims another
  database's tables disappeared.
- **Merging databases is safe; the report name is first-come.** Identities are `database.table`,
  so merged exports cannot collide - but the recorded vendor-side project name is taken from the
  first table seen.
- **Cross-estate reconciliation matches on the physical relation.** For Glue that is the S3
  location, not `database.schema.table`, so pairing a Glue table with a warehouse table only
  works where those strings genuinely agree.
- **Imports are not transactional.** If a write fails midway the nodes already written stay, and
  the error says so - re-running is additive and safe by design.

Every one of these is reported in-band: as a lossiness record, a finding, an explicit flag, or an
error that names the fix. The governed graph holds exactly what your catalog declared, annotated
with exactly what it could not hold. For the operational side - extraction recipe, routes,
payloads, the CI gate - see the AWS Glue quickstart in the CoreModels documentation.