Your AI Pipeline's Provenance Should Never Record Which Model Spoke

TLDR

Logging LLM model versions and prompt completions as primary data lineage creates fragile audit trails because generative model outputs are non-deterministic. Recording the deterministic tools dispatched by the orchestrator produces auditable, reproducible provenance chains.

Provenance Strategy	Primary Log Target	Deterministic Reproducibility	Durability Across Model Upgrades
Model-Centric Logging	LLM prompt & completion strings	Low (Non-deterministic outputs)	Fails on model deprecation
Tool-Centric Lineage	Deterministic Tool Functions	100% Verifiable & Reproducible	Durable Across Model Swaps

---

Logging non-deterministic model completions creates fragile, non-reproducible audit trails

Logging raw LLM prompts and model versions is popular for conversational AI applications. However, for data processing pipelines that require auditable outputs, logging model strings is unhelpful.

Because LLMs are non-deterministic, recording "GPT-4 generated this cell" provides no guarantee that re-running the prompt will yield identical results. Replacing or upgrading the underlying LLM invalidates historical logs.

---

Recording deterministic tool executions produces durable, verifiable data lineage chains

We structured provenance logging around deterministic tools (extractors, validators, math engines) executed by the orchestrator:

   MODEL-CENTRIC LINEAGE (Fragile):
   [Lineage Log] ──> "GPT-4 generated this value at 2:00 PM"
         │
         ▼
   (Auditor attempts to verify by re-running prompt) ──> Output changes! (Audit fails)
TOOL-CENTRIC LINEAGE (Durable):    [Lineage Log] ──> "ExtractionTool_v2 executed with regex_pattern_A"          │          ▼    (Auditor attempts to verify by re-running Tool_v2) ──> Identical Output (Audit passes!)

Here is the implementation of a deterministic tool lineage log:

// Record deterministic tool execution lineage instead of raw model completions
export function logToolLineage(previousRecord, toolName, inputParams, outputValue) {
  return {
    value: outputValue,
    lineage: [
      ...(previousRecord?.lineage || []),
      {
        tool: toolName,           // e.g., 'table-parser-v2'
        params: inputParams,      // Deterministic parameters used
        timestamp: Date.now()
      }
    ]
  };
}

Auditors verify results by re-running deterministic tool routines with saved parameters, ensuring lineage logs remain valid across model upgrades.

Rule of thumb: Record deterministic tool executions rather than model prompts to build auditable data provenance chains.