# Apache Airflow — Quickstart: Zero to First Audit: Bringing an Apache Airflow Deployment under CoreModels Governance

Your Airflow deployment can already describe itself. The stable REST API v1 will tell you every DAG, every task, and - if you use data-aware scheduling - every asset your pipelines produce and consume. What it will not tell you is whether any of that still matches what your organization *thinks* it runs: who owns each pipeline, which assets are consumed but produced by nothing, which consumers are quietly running on stale data because every producer is paused.

# Zero to First Audit: Bringing an Apache Airflow Deployment under CoreModels Governance

Your Airflow deployment can already describe itself. The stable REST API v1 will tell you every DAG, every task, and - if you use data-aware scheduling - every asset your pipelines produce and consume. What it will not tell you is whether any of that still matches what your organization *thinks* it runs: who owns each pipeline, which assets are consumed but produced by nothing, which consumers are quietly running on stale data because every producer is paused.

That gap is what a CoreModels audit closes. In this tutorial we go from three `curl` calls against your own Airflow API to a governed model of your orchestration estate and a first drift audit - no plugins installed in Airflow, no credentials shared with us, nothing running inside your deployment.

A note on capabilities before we start: the Airflow connector (vendor key `airflow`) supports **Import** and **Audit**. It deliberately does not support Generate - orchestration code is not derivable from schema, and the connector refuses honestly rather than emitting something half-true.

## Step 1: Extract the artifacts

CoreModels never connects to your Airflow instance. You call your own stable REST API v1 and upload the JSON responses as *artifacts*. Three artifacts exist; only the first is required.

| Artifact | Source | Required? |
|---|---|---|
| `dags` | `GET /api/v1/dags` - the full response or just its `dags` array | **yes** |
| `tasks` | `GET /api/v1/dags/{dag_id}/tasks`, aggregated as `{"<dag_id>": <response>}` | no |
| `datasets` | `GET /api/v1/datasets` - data-aware scheduling assets ("Assets" in Airflow 3) | no |

The exact recipe:

```bash
AIRFLOW=https://your-airflow.example.com
AUTH='-u your-user:your-password'   # or -H "Authorization: Bearer <token>"

curl -s $AUTH "$AIRFLOW/api/v1/dags?limit=200" > dags.json

for d in $(jq -r '.dags[].dag_id' dags.json); do
  curl -s $AUTH "$AIRFLOW/api/v1/dags/$d/tasks" | jq --arg d "$d" '{($d): .}'
done | jq -s 'add' > tasks.json

curl -s $AUTH "$AIRFLOW/api/v1/datasets" > datasets.json
```

Skip `tasks.json` and you still get governed DAGs and lineage, just without per-task detail. Skip `datasets.json` and you lose cross-DAG lineage - the most valuable part if you use data-aware scheduling - so include it if you can.

## Step 2: Import into a CoreModels project

You need a CoreModels project (its 32-character hex id is `$PROJECT_ID` below) and an **Admin** role on it, because import writes to the graph. It writes *additively*: existing governed nodes are never mutated, so re-importing later is safe.

Artifact values are the raw file contents passed as JSON strings, which is exactly what `jq --rawfile` produces:

```bash
jq -n --rawfile dags dags.json --rawfile tasks tasks.json --rawfile ds datasets.json \
  '{artifacts: {dags: $dags, tasks: $tasks, datasets: $ds}}' > import-request.json

curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/airflow/import/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @import-request.json
```

A successful import answers with counts and two honesty channels:

```json
{ "success": true, "vendor": "airflow", "projectName": "airflow",
  "datasetsAdded": 12, "datasetsSkippedExisting": 0, "fieldsAdded": 0,
  "lineageEdgesAdded": 9, "lineageEdgesSkipped": 0, "nodesEnriched": 60,
  "snapshotStored": true,
  "lossiness": [
    { "kind": "TypeApproximation", "path": "daily_revenue.build_report",
      "explanation": "Native type 'PythonOperator' was approximated as String; the exact native type is preserved in the vendor metadata mixin." }
  ],
  "errors": [] }
```

Read this as: each DAG and each data-aware-scheduling asset became a governed Type (`datasetsAdded`); each task became an Element on its DAG, with the operator class recorded as its native type - new DAGs carry their tasks in within `datasetsAdded`, while `fieldsAdded` counts only tasks added to already-governed DAGs on a later re-import, so it is 0 on a first import; each producing/consuming relationship became a lineage edge in the same `Depends On` graph that warehouse and dbt estates use (`lineageEdgesAdded`). The `lossiness` array is where the import confesses anything it approximated or dropped - for Airflow it carries one type-approximation record per task, because an operator class is not a data type - and `errors` is empty unless it could not proceed at all. `snapshotStored: true` means the parsed deployment snapshot was persisted, which enables one-call re-audits later without fresh artifacts.

## Step 3: Run your first audit

The audit compares fresh artifacts against the governed model. It is strictly read-only and needs only the **Viewer** role; setting `recordHistory: true` opts the run into the rolling drift trail:

```bash
jq -n --rawfile dags dags.json --rawfile ds datasets.json \
  '{artifacts: {dags: $dags, datasets: $ds}, recordHistory: true}' > audit-request.json

curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/airflow/audit/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @audit-request.json > audit-response.json
```

Run immediately after an import, the drift sections should be clean - you just imported this exact estate. What you will almost certainly see on a real deployment is conformance findings, because the connector ships four Airflow-specific hygiene rules.

## Step 4: Read the result

Start with the counts:

```bash
jq '{errorCount, warningCount, infoCount, codes}' audit-response.json
```

```json
{
  "errorCount": 0,
  "warningCount": 3,
  "infoCount": 2,
  "codes": { "dag-no-owner": 2, "dag-no-description": 2, "paused-producer": 1 }
}
```

`errorCount` is the machine signal - greater than zero means the estate violates governed meaning, and it is what a CI gate keys on. Warnings and infos are hygiene. Each finding in the `findings` array carries a section (`Coverage`, `Drift`, or `Conformance`), a severity, a stable code, a subject, and a human message. Here is a real one:

```json
{ "section": "Conformance", "severity": "Warning", "code": "dag-no-owner",
  "subject": "daily_revenue",
  "message": "DAG has no real owner (default 'airflow') - nobody is accountable for this pipeline." }
```

The four Airflow-specific rules are worth knowing on day one:

| Code | Severity | What it catches |
|---|---|---|
| `dag-no-description` | Info | a DAG without stated intent |
| `dag-no-owner` | Warning | owner missing or the default `airflow` - nobody accountable |
| `asset-unproduced` | Warning | an asset DAGs consume but nothing in this deployment produces |
| `paused-producer` | Warning | every producer of an asset paused while active DAGs still consume it - silent staleness |

On top of those, the shared audit engine reports coverage (`dataset-unmapped`, `field-unmapped`) and drift (`dataset-removed`, `field-removed`, `field-type-drift`, `contract-drift`, and enum-change codes) once your estate and governed model start evolving apart.

The response also includes `markdown` - a complete, PR-comment-ready report - so a human never has to read the JSON:

```bash
jq -r '.markdown' audit-response.json
```

And because you passed `recordHistory: true`, this run is now the first entry in the project's audit trail. Two quick follow-ups you can try right away, both Viewer-role `GET`s:

```bash
curl -sS -H "Authorization: Bearer $TOKEN" \
  "https://coremodels.example.com/graph/integrations/airflow/status/$PROJECT_ID"

curl -sS -H "Authorization: Bearer $TOKEN" \
  "https://coremodels.example.com/graph/integrations/airflow/badge/$PROJECT_ID" > airflow-audit.svg
```

`status` returns the last-import state (`imported`, `state`, `governedDatasets`); `badge` returns a shields-style SVG rendered from the latest recorded run - green for clean, yellow for warnings, red for errors, gray when nothing is recorded yet.

## Where this goes next

You now have a governed model of your orchestration estate and a repeatable audit. From here the same three files plug into a CI drift gate on your DAG repository (the machine-to-machine `v1` audit route accepts user API keys), the stored snapshot enables one-call re-audits whenever the governed model changes, and every governed fact - ownership, schedules, the full cross-DAG dependency graph - is queryable by AI agents over MCP. Each of those is a tutorial of its own.

One honest limitation to keep in mind: on very large deployments, when the encoded snapshot exceeds the storage cap (~1.5 MB), import reports `snapshotStored: false` with a lossiness record. Fresh-artifact audits keep working; only the stored-snapshot re-audit is unavailable.

For the condensed version of everything above, see the Apache Airflow quickstart in the CoreModels integration docs.