# AWS Glue — Quickstart: Ten Minutes to a Governed Glue Catalog Open a terminal. Everything in this tutorial is one AWS CLI command and two HTTP calls, and at the end of it your AWS Glue Data Catalog has a governed twin in CoreModels plus a first drift audit you can read line by line. Nothing gets installed in your AWS account, no IAM role is granted to us, and no credential of yours travels anywhere: you run the CLI, you upload the JSON it printed. # Ten Minutes to a Governed Glue Catalog Open a terminal. Everything in this tutorial is one AWS CLI command and two HTTP calls, and at the end of it your AWS Glue Data Catalog has a governed twin in CoreModels plus a first drift audit you can read line by line. Nothing gets installed in your AWS account, no IAM role is granted to us, and no credential of yours travels anywhere: you run the CLI, you upload the JSON it printed. We use `https://coremodels.example.com` as the API base URL, `$TOKEN` as your CoreModels login token, and `$PROJECT_ID` as the 32-character hex id of the CoreModels project that will govern the estate. You need `curl` and `jq` locally, plus the AWS CLI authenticated against the account that owns the catalog. Two roles matter. **Import** writes to the graph and requires the **Admin** role on the project. **Audit** never writes and requires only **Viewer** - which is why the automation you build later can run with a deliberately weak key. ## The artifact contract The connector is registered under the vendor key `glue`, display name *AWS Glue Data Catalog*, and it asks for exactly one artifact: > `tables` - required - `aws glue get-tables` JSON (one response, an array of responses, or a > bare TableList) That permissiveness is deliberate. The parser walks whatever you send it and collects table objects from three envelopes: a single `{"TableList": [...]}` response, an array of such responses, or a bare array of table objects. If the payload is valid JSON but contains no named tables, you get an explicit error rather than a successful import of nothing. ## Step 1 - Export the catalog ```bash aws glue get-tables --database-name lake > tables.json ``` If your estate spans several Glue databases, run the command once per database and merge the responses into a single array: ```bash aws glue get-tables --database-name lake > db-lake.json aws glue get-tables --database-name reference_data > db-reference.json jq -s '.' db-lake.json db-reference.json > tables.json ``` Merging is safe because a table's governed identity is `database.table`, so tables from different databases can never collide. One consequence worth knowing up front: the vendor-side project name recorded for the import is taken from the first table's `DatabaseName`, so a merged export will be labeled with whichever database came first. ## Step 2 - Import Build the body with `jq` so the file content is embedded as a correctly escaped JSON string - never paste a multi-megabyte export onto a command line. ```bash jq -n --rawfile tables tables.json '{artifacts: {tables: $tables}}' > import-request.json curl -sS -X POST \ "https://coremodels.example.com/graph/integrations/glue/import/$PROJECT_ID" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ --data-binary @import-request.json ``` ## Step 3 - Read the counts For a small `lake` database holding two tables - an `events` table with four columns and one partition key, and a `user_profiles` table with three columns - a first import answers: ```json { "success": true, "vendor": "glue", "projectName": "lake", "datasetsAdded": 2, "datasetsSkippedExisting": 0, "fieldsAdded": 0, "lineageEdgesAdded": 0, "lineageEdgesSkipped": 0, "nodesEnriched": 10, "snapshotStored": true, "lossiness": [ { "kind": "TypeApproximation", "path": "lake.events.payload", "explanation": "Native type 'struct' was approximated as String; the exact native type is preserved in the vendor metadata mixin." }, { "kind": "TypeApproximation", "path": "lake.user_profiles.email", "explanation": "Native type 'varchar(255)' was approximated as String; the exact native type is preserved in the vendor metadata mixin." } ], "errors": [] } ``` Four of those numbers routinely surprise people, so here is what each one actually counts. **`datasetsAdded: 2`** - two tables became governed Types. Their columns and partition keys came in with them as Elements, which is why **`fieldsAdded` reads `0`**: that counter is specifically "columns added to datasets that were *already* governed". On a first import it is always zero; on later re-imports it is the number that tells you the lake grew a column. **`nodesEnriched: 10`** - every Type and every Element received its vendor-metadata value: two tables plus eight columns (four `events` columns, one `events` partition key, three `user_profiles` columns). **`lineageEdgesAdded: 0`** - expected, permanently. `get-tables` output declares no dependencies between tables, and we do not infer lineage from naming conventions. The same restraint applies to keys and nullability: Glue declares neither, so the governed model contains neither. **`snapshotStored: true`** - the parsed catalog was persisted next to the project. That is what makes a one-call re-audit possible later without re-exporting anything. And `lossiness` is not an error list. It is the honesty channel: the import succeeded *and* it tells you exactly which two columns are governed by an approximation. The verbatim Hive type (`struct`, `varchar(255)`) is preserved on the node's vendor metadata regardless - the approximation only concerns the cross-vendor primitive the graph reasons with. ## Step 4 - The first audit The audit compares fresh artifacts against the live governed model. It is read-only and runs at Viewer role. Run it now with the same file and you have your baseline: ```bash jq -n --rawfile tables tables.json \ '{artifacts: {tables: $tables}, recordHistory: true}' > audit-request.json curl -sS -X POST \ "https://coremodels.example.com/graph/integrations/glue/audit/$PROJECT_ID" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ --data-binary @audit-request.json ``` `recordHistory` defaults to `false` - the audit verb stays strictly read-only unless asked. Set it to `true` when you want the run appended to the project's rolling trail, which is also what feeds the status badge. ## Step 5 - Read the result ```json { "success": true, "vendor": "glue", "projectName": "lake", "errorCount": 0, "warningCount": 1, "infoCount": 2, "codes": { "semi-structured-column": 1, "table-no-description": 1, "classification-missing": 1 }, "driftedObjects": [], "fingerprint": "3f9c1d5a7b2e4086", "metrics": { "Datasets (estate)": "2", "Datasets governed": "2 / 2", "Fields governed": "8 / 8", "Governed nodes with canonical mappings": "0 / 10 (0%)", "Last import": "2026-08-04T09:14:22.1043117+00:00" }, "findings": [ { "section": "Conformance", "severity": "Warning", "code": "semi-structured-column", "subject": "lake.events", "message": "1 struct/map/array column(s) carry ungoverned inner schemas.", "detail": "payload" }, { "section": "Conformance", "severity": "Info", "code": "table-no-description", "subject": "lake.user_profiles", "message": "Catalog table has no description - undocumented lake assets resist governance and agent grounding.", "detail": null }, { "section": "Conformance", "severity": "Info", "code": "classification-missing", "subject": "lake.user_profiles", "message": "Table has no classification parameter - crawlers/Athena treat the format as unknown.", "detail": null } ], "markdown": "# glue Schema Audit - lake\n\n🟡 **0 errors · 1 warnings · 2 info**\n…", "historyRecorded": true, "lossiness": [] } ``` Read it in this order. **`errorCount` first.** Zero means nothing in the catalog violates governed meaning. This single integer is the whole contract that automation keys off later. **Then `metrics`.** `Datasets governed: 2 / 2` and `Fields governed: 8 / 8` confirm the import landed completely. `Governed nodes with canonical mappings: 0 / 10 (0%)` counts how many of your governed nodes also carry a mapping to a *non-vendor* standard - right after an import that is naturally zero, and it is the number that moves as you ground your model in shared vocabularies. **Then `findings`.** Every finding carries a `section` (`Coverage`, `Drift`, or `Conformance`), a `severity` (`Info`, `Warning`, `Error`), a stable kebab-case `code`, the `subject` in vendor identity terms, a human `message`, and an optional `detail`. Coverage is what the catalog has and the model does not (`dataset-unmapped`, `field-unmapped`) - clean here, because we just imported. Drift is where the two disagree; `driftedObjects` collects those subjects separately for exactly this reason. Conformance is where the Glue connector's own rules live, and all three fired on this tiny estate: - `semi-structured-column` (Warning) - `lake.events` has a `struct` column whose inner schema is not governed. One aggregate finding per table, with the column names in `detail`, because a 40-field struct is one decision to make, not 40 rows of noise. - `table-no-description` (Info) - `lake.user_profiles` is undocumented. - `classification-missing` (Info) - `lake.user_profiles` has no `classification` parameter, so crawlers and Athena treat its format as unknown. None of those fail a build. They tell you where the catalog is thinner than your governance deserves - and they are a backlog you generated for the price of one API call. **Finally `markdown`.** The same report rendered for humans: a verdict line, the metrics table, and each section folded into a collapsible block. Paste it into a review, an incident channel, or a job summary as-is. ## Where to go from here You now have a governed baseline, a stored catalog snapshot, and one recorded run. Three obvious next moves, each a single call: `GET .../glue/status/$PROJECT_ID` to see the last-import bookkeeping; `POST .../glue/reaudit/$PROJECT_ID` with an empty JSON body (`{}`) to re-check the stored snapshot after someone edits governed meaning; and `POST .../glue/generate/$PROJECT_ID` to get Athena/Hive `CREATE EXTERNAL TABLE` DDL back out of the governed model. The AWS Glue quickstart in the CoreModels documentation collects every route, payload, and caveat on one page.