# Databricks — Automation: Fail the Build, Not the Dashboard: A CI Drift Gate for Databricks Unity Catalog Somewhere in your lakehouse, a notebook is about to rebuild a table with a widened column type, and nothing will fail until a dashboard does - days later, in front of the wrong audience. The cheapest place to catch that is a pull request. This article turns the CoreModels schema audit into a merge-blocking CI gate for a Unity Catalog estate, then builds out the rest of the operational loop most drift setups forget: the status badge, the rolling history, the one-call re-audit for when the governed model itself changes, and the scheduled heartbeat that keeps watch between releases. # Fail the Build, Not the Dashboard: A CI Drift Gate for Databricks Unity Catalog Somewhere in your lakehouse, a notebook is about to rebuild a table with a widened column type, and nothing will fail until a dashboard does - days later, in front of the wrong audience. The cheapest place to catch that is a pull request. This article turns the CoreModels schema audit into a merge-blocking CI gate for a Unity Catalog estate, then builds out the rest of the operational loop most drift setups forget: the status badge, the rolling history, the one-call re-audit for when the governed model itself changes, and the scheduled heartbeat that keeps watch between releases. ## The endpoint CI calls CI should never hold an interactive login, so the gate targets the machine-to-machine surface, which accepts CoreModels user API keys: ```text POST https://coremodels.example.com/v1/$PROJECT_ID/integrations/databricks/audit ``` The audit runs at Viewer role and never writes, so the key in your CI secrets needs no write access to anything. The response comes wrapped in the standard `ApiResponse` envelope: the report lives under `data`, and the gate keys on one number, `data.errorCount`. The semantics decide when builds go red, so they are worth being precise about. **Errors** mean the extract contradicts governed meaning - a governed field removed (`field-removed`), a column retyped against the model (`field-type-drift`), an enum narrowed (`enum-narrowed`). **Warnings** are the connector's best-practice rules; for Databricks that is `key-column-undeclared`, a key-shaped column with no declared informational constraint. **Info** findings, such as `table-no-comment`, are documentation nudges. Failing on `errorCount > 0` and letting warnings through keeps the signal crisp: red means meaning was violated, not that someone forgot a comment. Gate harder only once your estate is clean enough that warnings are actionable. ## Getting the extract into CI The audit needs one artifact: `information_schema`, the JSON rows of the documented Unity Catalog query. Two patterns work. - **Commit the extract with schema-change PRs.** The PR then carries both the intended change and the evidence, and the audit runs on exactly what is proposed. - **Re-extract in a prior CI step** against a staging catalog - for instance with `databricks-sql-cli` - so the pipeline always audits current reality. Either way, the audit step itself needs nothing but `curl` and `jq`. ## The gate, in full ```yaml name: coremodels-schema-audit on: pull_request: paths: ["schemas/**"] jobs: audit: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # A prior step places information_schema.json in the workspace - # committed with the PR, or re-extracted against a staging catalog. - name: CoreModels Schema Audit env: COREMODELS_API_URL: ${{ vars.COREMODELS_API_URL }} COREMODELS_API_KEY: ${{ secrets.COREMODELS_API_KEY }} COREMODELS_PROJECT_ID: ${{ vars.COREMODELS_PROJECT_ID }} run: | set -euo pipefail jq -n --rawfile info information_schema.json \ '{artifacts: {information_schema: $info}, recordHistory: true}' > audit-request.json curl -sS -o audit-response.json -X POST \ "$COREMODELS_API_URL/v1/$COREMODELS_PROJECT_ID/integrations/databricks/audit" \ -H "Authorization: Bearer $COREMODELS_API_KEY" -H "Content-Type: application/json" \ --data-binary @audit-request.json test "$(jq -r '.success' audit-response.json)" = "true" jq -r '.data.markdown' audit-response.json >> "$GITHUB_STEP_SUMMARY" test "$(jq -r '.data.errorCount' audit-response.json)" -eq 0 ``` Reading it line by line: `jq -n --rawfile` wraps the extract as a string value inside the request body, escaping included. The first `test` distinguishes "the audit could not run" from "the audit ran and found problems" - a transport failure or unusable payload should fail the build loudly, not masquerade as a clean report. The `markdown` field goes to the job summary, so reviewers see the human-readable report - findings grouped by section, coded and subject-lined - without leaving the PR. The final `test` is the gate itself: any error finding fails the step. `recordHistory: true` matters more than it looks. Every gated run is appended to the project's rolling audit trail with trigger `ci`, which is what makes the next two pieces work. One note if you already use our composite GitHub Action elsewhere: its artifact inputs are manifest-shaped, built for build-artifact vendors. For a Unity Catalog extract the curl-and-jq step above is the recipe - same endpoint, same envelope, same gate semantics. ## The badge The same machine-to-machine surface serves an SVG badge rendered from the latest recorded run: ```bash curl -sS -H "Authorization: Bearer $COREMODELS_API_KEY" \ "https://coremodels.example.com/v1/$PROJECT_ID/integrations/databricks/badge" \ -o databricks-audit.svg ``` Green means the last recorded run was clean, yellow means warnings only, red means errors, gray means nothing recorded yet. The route authenticates with the API key, so for a public README the practical pattern is the one above: fetch the badge in a scheduled job and publish the SVG wherever your docs live. Because the badge reflects *recorded* runs, a gate that records on every PR keeps it truthful for free. ## The rolling history The badge answers "now"; the history answers "lately": ```bash curl -sS -H "Authorization: Bearer $TOKEN" \ "https://coremodels.example.com/graph/integrations/databricks/history/$PROJECT_ID" ``` ```jsonc { "success": true, "vendor": "databricks", "projects": [ { "projectName": "main", "runs": [ { "at": "2026-08-03T09:12:44.6183920+00:00", "trigger": "ci", "errorCount": 0, "warningCount": 2, "infoCount": 3, "codes": { "key-column-undeclared": 2, "table-no-comment": 3 }, "fingerprint": "9f2c4a1b0e77d3aa" }, { "at": "2026-08-01T16:40:02.1170455+00:00", "trigger": "reaudit", "errorCount": 1, "warningCount": 2, "infoCount": 3, "codes": { "field-type-drift": 1, "key-column-undeclared": 2, "table-no-comment": 3 }, "fingerprint": "9f2c4a1b0e77d3aa" } ] } ] } ``` Runs come back newest first, and `trigger` is one of four lowercase values: `audit`, `ci`, `reaudit`, `scheduled`. The trail is a rolling window - the most recent 50 runs per estate - so treat it as a drift signal, not an archive: if you need audit records kept forever, store the full audit response your gate already downloaded as a build artifact. Two further details make this trail genuinely useful. The `fingerprint` is a content hash of the audited artifacts: identical fingerprints with different finding counts mean the *governed model* moved, not the estate - exactly the situation in the sample above, where a `field-type-drift` error appeared and then cleared between two runs of the same extract. And `codes` gives you drift trends per rule without parsing findings. Note the surface boundary: history (and re-audit, next) live on the interactive surface with a login token; the API-key surface carries audit and badge. ## Re-audit: the gate for the other direction CI catches the estate drifting from the model. But models change too - someone tightens a type or removes a governed field in CoreModels. Re-audit checks the current governed model against the estate snapshot stored at import time, no fresh extract required: ```bash curl -sS -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ "https://coremodels.example.com/graph/integrations/databricks/reaudit/$PROJECT_ID" \ -d '{}' ``` The run is always recorded in the history (trigger `reaudit`), so governed-side changes leave the same audit trail estate-side changes do. The dependency to know about: re-audit needs the snapshot stored at import. Very large estates can exceed the snapshot storage cap (roughly 1.5 MB encoded), in which case import reports `snapshotStored: false` and re-audit has nothing to run against - fresh-artifact audits still work as usual. ## The heartbeat PR gates only fire when someone opens a PR. For the quiet weeks, CoreModels deployments have a scheduled server-side re-audit - a background worker, off by default and enabled by the operator through deployment configuration, per opted-in project - that periodically re-audits those projects against their stored snapshots and records the runs into the same history under trigger `scheduled`. No credentials, no vendor API calls, no CI minutes: it replays the last-known estate against the current model on a schedule, and a project with no stored snapshot is simply skipped. Combined with the PR gate you get both directions covered continuously: estate-side drift is caught at merge time, governed-side drift is caught by the heartbeat, and both write to one trail behind one badge. ## The loop, assembled One secret (a Viewer-scoped API key), one committed or re-extracted JSON file, and a five-line audit step buy you: a merge gate on governed meaning, a PR-visible report, a truthful badge, a queryable drift history with content fingerprints, and a scheduled heartbeat for the weeks nobody touches the schema. None of it holds a Databricks credential. The extraction queries and the interactive-surface equivalents of these calls are in the Databricks quickstart in the CoreModels documentation.