Scripts as pipelines, and a catalog that stays out of the way

TL;DR

We shipped the data catalog and the first real transform story this week. The interesting part is what we didn't ship: there is no pipeline entity, no orchestration service, no DAG editor. A pipeline in ProForma is just a Script that calls context.step() — and it turns out that carries its weight.

Why this matters

Every data warehouse product eventually grows a pipeline object. Airflow has DAGs. dbt has models with ref graphs. Dagster has assets. Each one adds a second mental model on top of "the code that does the thing" — a configuration surface, a runtime, a UI, a way for the AI to talk about it. The second model always wants to be the first.

ProForma's thesis is that a warehouse agent should lean on strong abstractions, not many of them. We already had Scripts (arbitrary TypeScript the agent and users can run) and Models (saved SQL definitions). The question was whether "pipeline" needed to join them, or whether Script + Model could carry the job.

We spent a plan on the longer path, then rewrote it. The shipped design is the shorter one.

What changed

• ScriptContext gained three methods: context.step(name, fn) for structured logging, context.scripts.run(name, input?) for composition, and context.models.refresh(key) for materialization.
• The Catalog surfaced: Sources (raw tables, auto-discovered from the engine) and Models (saved SQL, materialized as VIEW or TABLE) both get titles, descriptions, tags, and column-level notes — all searchable and editable in the new Catalog page.
• The AI agent picked up four new tools (search_catalog, create_model, refresh_model, sync_catalog) and its system prompt now includes the catalog contents so it knows what data exists without calling tools.

Full plan: plans/ARCHIVE/pipelines-catalog-ontology.md.

How it works

A pipeline in ProForma looks like this — it's just a Script:

const orders = await context.step("sync-orders", async () => {
  const res = await context.http.fetch("https://api.stripe.com/v1/charges", {
    headers: { Authorization: `Bearer ${context.secrets.require("STRIPE_KEY")}` },
  });
  await context.warehouse.query(
    `CREATE OR REPLACE TABLE raw_orders AS
     SELECT * FROM read_json_auto('${JSON.stringify(res.body)}')`,
  );
  return { count: res.body.data.length };
});

await context.step("refresh-models", async () => {
  await context.models.refresh("clean_orders");
  await context.models.refresh("customer_360");
});

Each step writes a step:start / step:end JSON entry into the execution log with timing and status. The Scripts page parses those entries and renders them as a structured progress list, so the pipeline UI is the script UI. The worker that already runs scheduled scripts runs scheduled pipelines too — zero new infrastructure.

The catalog follows the same "ontology is descriptions" rule. Instead of a relationships table or a semantic-type enum, every source, model, and column carries plain-text descriptions. The AI system prompt gets stuffed with them and the AI can edit them. When we need something stronger, we'll know; today this covers the 80% case at roughly 5% of the code.

What's next

The structured step logs open a feedback loop we want the agent to close: "why did last night's daily refresh fail?" becomes a question the AI can answer without new tools, because list_executions already surfaces the step-level failure. Next up: tighter AI usage of that signal, and lineage derived from parsing model SQL for FROM references — still without a new entity, still at the Source/Model layer.

See plans/ for what's in flight.