documentation

Veritas — leakage & robustness auditing

Veritas is a model-agnostic, post-hoc auditor for sequence-based biological predictors. It answers one question honestly: how much of a model's reported performance survives once train/test leakage is removed? It never runs your model — it works on predictions you already have, so it applies equally to a protein language model, a DNA CNN, a docking score, or a black-box API.

Overview

ML benchmarks leak: test examples are often homologous to training examples, so a model can score well by recognizing relatives rather than generalizing. Veritas detects that homology between an evaluation set and a reference set the model could have memorized, re-scores the metric on the de-leaked set, and reports the gap — the reported → honest collapse — with full provenance and a verifiable signature.

How it works

Predictions in → audit out. You give Veritas a benchmark (sequences + labels + a declared split, or a model + its reference set) and the model's per-example predictions. It then runs a three-step pipeline:

1. detect    cross-set homology  →  contamination graph
2. re-score  metric on the de-leaked set  +  bootstrap CIs
3. stratify  performance by difficulty  +  sign with audit_hash

Detectors

Contamination is detected three ways, combined into one graph:

• sequence — identity search with MMseqs2 (protein or nucleotide), e.g. ≥30% identity over ≥50% coverage.
• family — shared profile-HMM family via Pfam / pyhmmer (protein-only).
• structural — fold similarity with foldseek TMalign (protein-only). This is fold-level, a more permissive signal than interface-level redundancy (e.g. iDist) — reported as its own quantity, not directly comparable.

The audit report

Every report is a single JSON document. The headline is three traced metrics — reported, honest, and their delta (the leakage), each with a bootstrap confidence interval. Alongside: a leakage summary (how many eval items were contaminated, by which detector), an optional stratification (performance by difficulty, with silent-failure flags), the provenance record, and the disclosed limitations. Every number is a traced value carrying where it came from.

Report kinds

Not every audit produces a reported-vs-honest metric, so a report declares its report_kind:

• metric_audit — the reported→honest collapse (a model was scored).
• detection — a leakage splits-matrix (split × detector) when no model was scored.
• stratification — a performance-by-difficulty curve.

A validator forbids a detection or stratification report from carrying a (fabricated) metric — the no-fabrication guarantee is structural.

Tamper-evidence

Every report carries an audit_hash: a SHA-256 over its canonical content (everything except wall-clock stamps). The report viewer recomputes that hash in your browser and shows a seal — verified or mismatch. Change one number in a report and the seal breaks. The viewer is as tamper-evident as the tool that produced the report.

Results

Five locked audits, each a real run on pinned data, guarded by a test — open any in the viewer:

Guarantees & limitations

Veritas states both. Guarantees: provenance on every number; byte-identical reports on the pinned platform (determinism); honest uncertainty — bootstrap CIs and the applicable limitations travel inside the report and are hashed into it.

Limitations (measured, not assumed): confidence intervals use the percentile bootstrap, which under-covers at small n (~0.927 coverage at n=25; BCa not implemented). The MinHash prefilter is recall-oriented (~56% at the nominal Jaccard threshold) — disable it for an exhaustive comparison. Family/structural detectors are protein-only. Structural detection is fold-level, not interface-level. Results corroborate prior work qualitatively, not numerically.

Using Veritas

Two install paths. Docker bakes in the version-pinned detector binaries (MMseqs2, Diamond, Foldseek, HMMER) plus the CLI, so it runs on any OS with no conda setup — the binaries match the versions stamped into the report's provenance:

docker pull ghcr.io/shreyjain11/veritas-leakage:latest
docker run --rm -v "$PWD:/work" ghcr.io/shreyjain11/veritas-leakage audit \
  --sequences /work/eval.fasta --table /work/table.csv \
  --reference /work/reference.fasta --config /work/config.json \
  --metric accuracy --out /work/report.json
# …or build it yourself: docker build -t veritas-leakage .

Or install from PyPI with the cli extra and bring your own detector binaries (pinned in environment.yml):

pip install "veritas-leakage[cli]"
veritas audit --sequences eval.fasta --table table.csv \
  --reference reference.fasta --config config.json --metric accuracy --out report.json

Full usage, the dataset manifests, and the reproducible demos live on GitHub. This is an independent research project; every number on this site comes from a real run on pinned data, locked by a test — nothing is fabricated.