Epic: plan for agent provenance on yamls

[cmungall]

1. Have a mechanism such the CLI curators run https://ai4curation.io/ai-blame/ to inject provenance into yaml (or should we do sidecars?)
2. Have a mechanism such that github actions also does this
3. for github actions, preserve the claude code trace, at least as an artefact (see the action here https://github.com/ai4curation/go-ontology-agent-eval)
4. explore more granular provenance, not just file level

cc @matentzn

[github-actions]

Issue Summary: Agent Provenance Tracking for YAML Files

This epic outlines a comprehensive plan to track AI agent provenance for disorder YAML files in the dismech knowledge base. The goal is to maintain transparent audit trails showing which AI models and agents created or modified content.

Background Context

ai-blame is a provenance tracking tool (similar to git blame) that extracts structured audit trails from AI agent execution traces. It shows line-level attribution (which AI model wrote specific content) and supports multiple agents including Claude Code and GitHub Copilot.

Key features:

• Parses execution traces to create structured metadata (timestamps, model names, versions, changes)
• Desktop UI for visual exploration
• Configurable output policies for embedding provenance

The Four Work Items

1. CLI Curator Integration with ai-blame

Decision needed: Direct injection vs. sidecar pattern

Direct injection approach:

# In kb/disorders/Asthma.yaml
edit_history:
  - timestamp: "2026-01-08T01:30:00Z"
    model: "claude-sonnet-4-5"
    agent: "claude-code"
    changes: "Added MAXO treatment terms"

Sidecar pattern approach:

# Creates kb/disorders/Asthma.history.yaml
# Original kb/disorders/Asthma.yaml remains untouched

The sidecar pattern uses a configurable naming scheme (default: {stem}.history.yaml) and keeps provenance separate from primary data files.

Trade-offs:

• Direct injection: Single source of truth, embedded metadata, but modifies original files
• Sidecar: Preserves original files, cleaner separation, but requires managing paired files

Recommendation for dismech: Consider sidecar pattern to avoid polluting disorder YAML files with metadata and to maintain schema compliance without adding provenance fields to the LinkML schema.

2. GitHub Actions Integration

Currently, the repository has multiple Claude Code workflows:

• .github/workflows/claude.yml - Main Claude Code action (runs on @claude mentions)
• .github/workflows/dragon-ai.yml - Dragon AI agent (runs on @dragon-ai-agent mentions)
• Other triage/review workflows

Implementation approach:

• Add ai-blame as a post-processing step after AI agent modifications
• Generate provenance metadata for all YAML changes made by agents
• Store metadata using chosen pattern (direct or sidecar)

3. Preserve Claude Code Traces as Artifacts

Reference implementation: The go-ontology-agent-eval repository demonstrates this pattern for the Gene Ontology project.

Implementation:

# In GitHub Actions workflow
- name: Run Claude Code
  id: claude
  uses: anthropics/claude-code-action@beta
  # ... existing config ...

- name: Upload Claude trace
  uses: actions/upload-artifact@v4
  with:
    name: claude-trace-${{ github.run_id }}
    path: .claude/traces/
    retention-days: 90

Benefits:

• Debugging failed agent runs
• Understanding agent decision-making
• Compliance and audit requirements
• Training data for improving agent prompts

4. Explore Granular Provenance (Beyond File-Level)

Current ai-blame provides line-level attribution, but for structured YAML, we could track:

Field-level provenance:

phenotypes:
  - name: Airway inflammation
    phenotype_term:
      preferred_term: airway inflammation
      term:
        id: HP:0031387
        label: airway inflammation
    _provenance:
      added_by: "claude-sonnet-4-5"
      timestamp: "2026-01-08T01:30:00Z"
      agent_run: "gh-actions-123"

Evidence-level provenance:

evidence:
  - reference: PMID:12345678
    snippet: "..."
    _provenance:
      validated_by: "linkml-reference-validator"
      validation_timestamp: "2026-01-08T02:00:00Z"
      validator_version: "0.3.0"

Benefits:

• Track which AI agent added specific ontology terms
• Audit evidence validation history
• Identify high-confidence vs. low-confidence additions
• Support selective rollback of AI-generated content

Challenge: Requires extending the LinkML schema to support provenance metadata fields, or using a separate provenance layer.

Related Issues

No directly related issues found in the repository. This is a new infrastructure initiative.

Relevant Resources

• ai-blame Documentation
• Sidecar Pattern Reference
• go-ontology-agent-eval Repository

Recommended Next Steps

1. Prototype sidecar pattern with ai-blame on 2-3 disorder files
2. Add ai-blame to one GitHub Action workflow (suggest dragon-ai.yml first)
3. Implement trace artifact preservation in claude.yml workflow
4. Design granular provenance schema - create a proposal for field-level metadata structure
5. Document provenance standards - add to CLAUDE.md and developer docs

Questions for Discussion

1. Preference between direct injection vs. sidecar pattern?
2. Should provenance be tracked for human curators too (not just AI)?
3. What retention period for Claude trace artifacts?
4. Should granular provenance be in-file or in a separate database/index?

[matentzn]

This is great - we should probably try to align a little here on what I am currently trying to do with the evidence paket system - I am very interested in integrating the ai-blame traces as evidence-level provenance for example.