Editorial: Frequency qualifiers often lack direct evidence support

[claude]

Origin

From review comment on PR #83 by @cmungall:

good support for the phenotype but not the frequency (this is noted in many reviews of other disorders, we should find a good common solution)

This was noted in the context of the Scurvy disorder file where evidence supports the phenotype itself but not its frequency qualifier (FREQUENT, VERY_FREQUENT, etc.).

Scope

This is a cross-cutting pattern affecting multiple disorder files across the knowledge base. Evidence items typically quote text that confirms a phenotype exists, but the quoted text often doesn't include quantitative frequency data that would justify the assigned frequency qualifier.

Current State

• Evidence snippets confirm phenotypes are present
• Frequency qualifiers (VERY_FREQUENT, FREQUENT, OCCASIONAL, RARE) are assigned
• The connection between evidence and frequency is often implicit or missing

Examples

In Scurvy (PR #83):

- category: Dermatological
  name: Ecchymoses
  frequency: FREQUENT
  evidence:
  - reference: PMID:37366866
    snippet: "Common manifestations include... ecchymoses."

The word "common" provides some frequency information, but it's unclear if "common" maps to FREQUENT vs VERY_FREQUENT.

Proposed Solutions (for discussion)

1. Require frequency-specific evidence: Each frequency qualifier should be supported by evidence containing quantitative data (percentages, cohort studies, etc.)

2. Separate frequency evidence: Allow frequency_evidence as a separate field from phenotype_evidence

3. Relaxed validation: Accept frequency qualifiers based on clinical judgment even without explicit evidence

4. Standardized mapping guidelines: Define how terms like "common", "rare", "occasional" in literature map to our frequency enums

5. Optional frequency field: Make frequency optional when supporting evidence is unavailable

Next Steps

• Gather input from curators on preferred approach
• Survey existing disorder files to assess scope
• Update schema and validation rules if needed
• Provide guidance in curation documentation

[github-actions]

Analysis Summary: Frequency Qualifier Evidence Gap

I've conducted a comprehensive survey of all disorder files in the knowledge base to assess the scope and patterns of frequency qualifier usage. Here are the findings:

Scope Assessment

Statistics:

• Total disorder files analyzed: 140
• Files with frequency qualifiers: 140 (100%)
• Total frequency occurrences: 1,129 (across phenotypes and pathophysiology)
• Files with clear quantitative evidence: ~49 (35%)

This confirms that frequency-evidence alignment is a pervasive cross-cutting issue affecting the entire knowledge base.

Identified Patterns

Pattern 1: Qualitative-Only Evidence

The most common pattern is evidence containing qualitative frequency language without quantitative support:

Example (Scurvy, as noted in issue):

- name: Ecchymoses
  frequency: FREQUENT
  evidence:
    - reference: PMID:37366866
      snippet: "Common manifestations include... ecchymoses."

The term "common" provides some frequency signal but doesn't clearly map to FREQUENT (vs VERY_FREQUENT).

Pattern 2: Orphaned Frequency Values

Multiple phenotypes have frequency qualifiers but empty evidence lists:

Example (Monkeypox):

- name: Lymphadenopathy
  frequency: FREQUENT
  evidence: []  # No evidence provided

Pattern 3: Evidence-Frequency Misalignment

Cases where quantitative evidence contradicts the assigned frequency:

Example (Jeavons Syndrome):

- name: Generalized Tonic-Clonic Seizures
  frequency: OCCASIONAL
  evidence:
    - reference: PMID:30082241
      snippet: "70% experienced generalized tonic-clonic seizures"

70% should map to FREQUENT or VERY_FREQUENT, not OCCASIONAL.

Pattern 4: Best Practice (Rare)

Strong quantitative support with clear alignment:

Example (Down Syndrome):

- name: Congenital Heart Defects
  frequency: FREQUENT
  evidence:
    - reference: PMID:26504441
      snippet: "The prevalence of congenital heart disease in infants with Down syndrome is 40%..."

HPO Frequency Terms

The schema uses HPO frequency terms with the following mappings:

Enum Value	HPO Term	HPO ID	Implied Range
OBLIGATE	Obligate	HP:0040280	100%
VERY_FREQUENT	Very frequent	HP:0040281	80-99%
FREQUENT	Frequent	HP:0040282	30-79%
OCCASIONAL	Occasional	HP:0040283	5-29%
VERY_RARE	Very rare	HP:0040284	1-4%

(Note: HPO defines these ranges; we should verify current HPO guidelines)

Research Context

Phenotype frequency annotation is a known challenge in clinical genetics and rare disease curation:

• ClinGen/Monarch best practices emphasize evidence-based frequency assignment when possible
• Many rare disease resources struggle with frequency data due to limited cohort studies and publication bias (severe presentations over-represented)
• Qualitative terms like "common" or "rare" in literature often reflect author impression rather than systematic data

Recommendations

Based on this analysis, I recommend Solution 4: Standardized Mapping Guidelines as the most pragmatic approach:

Proposed Guidelines

1. Primary: Use quantitative evidence when available

   • Extract percentages/prevalence from snippets
   • Map to HPO ranges: 80-99% → VERY_FREQUENT, 30-79% → FREQUENT, etc.

2. Secondary: Map qualitative terms with documentation

   • Define standard mappings: "very common/typical" → VERY_FREQUENT, "common" → FREQUENT, "occasional/uncommon" → OCCASIONAL, "rare" → VERY_RARE
   • Document the mapping rationale in evidence explanation field

3. Tertiary: Clinical judgment with transparency

   • When no frequency evidence exists but frequency is clinically well-established, make frequency optional rather than removing it
   • Add a validation warning (not error) for phenotypes with frequency but no supporting evidence
   • Curators can add note: "Frequency based on clinical experience; quantitative evidence not available"

4. Quality Control

   • Add compliance check for "frequency without evidence"
   • Flag misalignments (quantitative evidence contradicting assigned frequency)
   • Prioritize adding frequency evidence for diagnostic phenotypes

Implementation

This could be supported by:

• Updated curation documentation in CLAUDE.md
• Validation tooling enhancement (warnings for unsupported frequencies)
• Compliance dashboard metric for "frequency evidence coverage"

Files Needing Attention

High Priority (clear misalignments):

• kb/disorders/Jeavons_Syndrome.yaml - 70% marked as OCCASIONAL
• kb/disorders/Monkeypox.yaml - Multiple empty evidence lists

Medium Priority (qualitative-only evidence):

• Most disorder files (~65%) use qualitative terms without clear quantitative support

Would you like me to proceed with any specific action, such as:

1. Creating detailed curation guidelines for frequency annotation?
2. Adding compliance checks for frequency-evidence gaps?
3. Prototyping a batch correction for clear misalignments?

[dragon-ai-agent]

Frequency Evidence: Schema Design Proposals

Problem Statement

Issue #112 identifies a fundamental problem: the current schema conflates evidence for two distinct claims:

1. Disease-to-Phenotype (D2P) Association: "This phenotype occurs in this disease"
2. Frequency Claim: "This phenotype occurs in X% of patients with this disease"

Currently, a Phenotype has:

• frequency: a simple enum value (FREQUENT, OCCASIONAL, etc.)
• evidence: a list of EvidenceItem objects

The evidence list is intended to support the D2P association, but curators often include frequency-related evidence in the same list, or worse, assign frequency qualifiers without any quantitative evidence at all.

Example of the Problem

phenotypes:
- name: Memory Impairment
  frequency: VERY_FREQUENT   # Claims 80-99% prevalence
  evidence:
  - reference: PMID:24459411
    snippet: "Loss of memory is among the first symptoms reported..."
    # This snippet supports D2P but says nothing about frequency

The snippet confirms memory impairment occurs in Alzheimer's but provides no data supporting "VERY_FREQUENT" (80-99%).

---

Option 1: Add frequency_evidence Slot (Minimal Change)

Schema Change

Add a new slot frequency_evidence to the Phenotype class:

slots:
  frequency_evidence:
    description: Evidence specifically supporting the frequency qualifier claim
    multivalued: true
    range: EvidenceItem
    inlined_as_list: true

classes:
  Phenotype:
    slots:
      - category
      - name
      - phenotype_term
      - frequency
      - frequency_evidence   # NEW
      - description
      - diagnostic
      - sequelae
      - evidence            # Scoped to D2P association
      - context
      - ...

Example Usage

phenotypes:
- name: Memory Impairment
  frequency: VERY_FREQUENT
  evidence:
    - reference: PMID:24459411
      snippet: "Loss of memory is among the first symptoms reported..."
      explanation: Confirms memory impairment is characteristic of AD.
  frequency_evidence:
    - reference: PMID:12345678
      snippet: "In our cohort of 500 AD patients, 94% exhibited memory impairment at presentation."
      explanation: Quantitative data supporting VERY_FREQUENT classification.

Pros

1. Minimal disruption: Existing data remains valid; migration is optional
2. Clear separation: Unambiguous which evidence supports which claim
3. Backward compatible: Files without frequency_evidence still validate
4. Simple implementation: No new classes, just one new slot
5. Low cognitive load: Easy for curators to understand

Cons

1. Flat structure: Frequency remains a simple enum; no room for additional metadata (e.g., population, source study, confidence)
2. Duplication potential: Same PMID might appear in both evidence and frequency_evidence
3. Incomplete model: Does not capture frequency as a first-class entity with its own semantics
4. Validation gaps: Cannot easily validate that frequency_evidence contains quantitative data

---

Option 2: Promote Frequency to Full Object (Comprehensive Refactor)

Design Philosophy

Following the pattern established by other descriptor classes (PhenotypeDescriptor, TreatmentDescriptor, etc.), frequency becomes a structured object that can hold its own evidence, source context, and metadata.

Schema Changes

classes:
  FrequencyDescriptor:
    description: A structured descriptor for phenotype frequency with supporting evidence
    slots:
      - frequency_category    # The enum value (FREQUENT, OCCASIONAL, etc.)
      - percentage            # Optional: exact percentage if known (e.g., "70%", "30-50%")
      - population            # Optional: which population this applies to
      - source_cohort_size    # Optional: sample size from source study
      - evidence              # Evidence supporting this frequency claim
      - notes                 # Additional context
    slot_usage:
      evidence:
        description: Evidence specifically supporting this frequency claim

slots:
  frequency_category:
    range: FrequencyEnum
    description: The HPO frequency category (FREQUENT, OCCASIONAL, etc.)

  frequency_descriptor:
    description: Structured frequency information with evidence
    range: FrequencyDescriptor
    inlined: true

classes:
  Phenotype:
    slots:
      - category
      - name
      - phenotype_term
      - frequency_descriptor   # REPLACES: frequency
      - description
      - diagnostic
      - sequelae
      - evidence              # Scoped to D2P association
      - context
      - ...

Example Usage

phenotypes:
- name: Memory Impairment
  phenotype_term:
    preferred_term: Memory impairment
    term:
      id: HP:0002354
      label: Memory impairment
  evidence:
    - reference: PMID:24459411
      snippet: "Loss of memory is among the first symptoms reported..."
      explanation: Confirms memory impairment is characteristic of AD.
  frequency_descriptor:
    frequency_category: VERY_FREQUENT
    percentage: "94%"
    population: "Sporadic late-onset AD"
    source_cohort_size: 500
    evidence:
      - reference: PMID:12345678
        snippet: "In our cohort of 500 AD patients, 94% exhibited memory impairment at presentation."
        explanation: Large cohort study providing quantitative frequency data.
    notes: Rate may vary in early-onset familial forms.

Pros

1. Rich semantics: Captures percentage, population context, cohort size alongside the category
2. Pattern consistency: Follows established Descriptor pattern used throughout schema
3. Future-proof: Easy to add additional metadata fields (e.g., confidence intervals, method)
4. Self-documenting: Structure makes it obvious what frequency evidence should contain
5. Population-specific frequencies: Can model different frequencies in different populations
6. Validation potential: Can validate that frequency_category aligns with percentage

Cons

1. Breaking change: Requires migration of all existing files
2. Increased complexity: Deeper nesting may be harder for curators
3. Migration effort: Need scripts to transform frequency: ENUM to frequency_descriptor: {...}
4. Tooling updates: HTML renderer, compliance checker, etc. all need updates
5. Optional bloat: Many files may only use frequency_category with empty optional fields

---

Migration Considerations

Option 1 Migration

Essentially zero migration required. Existing files remain valid. Curators can optionally add frequency_evidence over time.

Option 2 Migration

Requires a migration script. Basic transformation:

# Before
phenotype = {
    "name": "Memory Impairment",
    "frequency": "VERY_FREQUENT",
    "evidence": [...]
}

# After (automated migration - conservative)
phenotype = {
    "name": "Memory Impairment",
    "frequency_descriptor": {
        "frequency_category": "VERY_FREQUENT"
        # evidence intentionally left empty - needs curation
    },
    "evidence": [...]
}

The migration script would:

1. Convert frequency: ENUM to frequency_descriptor: {frequency_category: ENUM}
2. Leave evidence in place (scoped to D2P)
3. Flag files for curator review to add frequency evidence

---

Recommendation Summary

Criterion	Option 1	Option 2
Implementation effort	Low	High
Migration required	No	Yes
Semantic clarity	Good	Excellent
Captures population context	No	Yes
Captures exact percentages	No	Yes
Pattern consistency	Moderate	High
Curator learning curve	Low	Moderate
Backward compatibility	Full	None

When to Choose Option 1

• If the primary goal is distinguishing D2P evidence from frequency evidence
• If rapid deployment is needed
• If curator bandwidth for migration is limited

When to Choose Option 2

• If long-term semantic richness is prioritized
• If population-specific frequencies are important (e.g., "90% in Europeans, 70% in East Asians")
• If capturing exact percentages alongside categories is valuable
• If consistency with other Descriptor patterns is desired

---

Questions for Discussion

1. How important is capturing population-specific frequencies?
2. Do we want to record exact percentages, or are HPO categories sufficient?
3. What is the appetite for a breaking change requiring migration?
4. Should we consider a hybrid: Option 1 now, Option 2 later?