veritas
leakage & robustness auditor

ML models report inflated performance because their benchmarks leak.

Veritas measures how much survives once train/test homology is removed — model-agnostic, provenance-stamped, reproducible. It never runs your model.

Open the report viewer →See how it works ↓
OverfitNN · random splitmetric audit
reported
0.165
honest
0.018

leakage Δ +0.147 · 89% of the reported metric

open this audit →
four more audits · across model types

Each is a real run on pinned data, locked by a test — spanning genomic, variant-effect, and protein-interaction models, and all three detector kinds. Nothing is fabricated.

OverfitNN · chr8+chr9

metric audit
reported
0.074
honest
0.030

leakage Δ +0.044 · 59% of the reported metric

Even holding out whole chromosomes leaves residual leakage: 0.074 reported, 0.030 honest.

Open audit →

hashFrag · reverse-complement

detection
80.8%reverse-complement

On a naive genomic split, 80.8% of test sequences are exact reverse-complements of training sequences.

Open audit →

ProteinGym · MSA depth

stratification
0.298
Low
0.384
Medium
0.531
High

Variant-effect performance rises with alignment depth: 0.298 (low) → 0.384 → 0.531 (high).

Open audit →

PPI · interface leakage

detection
demonstration
Pr/PI control
AB/AG hold-out
TCR/pMHC hold-out

A 30%-identity sequence split leaves family + fold leakage the sequence detector can't see; a published control stays near zero.

Open audit →
how it works

Predictions in → audit out. Veritas works on predictions you already have, so it applies to a protein language model, a DNA CNN, a docking score, or a black-box API.

  1. 01

    Detect leakage

    sequence (mmseqs) · family (Pfam / HMMER) · structural (foldseek, fold-level)

  2. 02

    Re-score honestly

    the metric recomputed on the de-leaked set, with bootstrap confidence intervals

  3. 03

    Stratify & sign

    performance by difficulty, plus an audit_hash over every number in the report

provenance

every number carries where it came from

deterministic

byte-identical reports on the pinned platform

honest CIs

uncertainty + disclosed limitations travel inside the report

tamper-evident

re-verify the audit_hash yourself, in the browser

An independent research project. Every number on this site is from a real run on pinned data, locked by a test — nothing is fabricated. Structural detection is fold-level (foldseek), a more permissive signal than interface-level redundancy; results corroborate prior work qualitatively, not numerically.

ValidationGitHubDocs