gosai_k562_overfitnn_random_split
metric auditreported → honestspearman
reported0.165[0.137, 0.196]
honest0.018[-0.016, 0.050]
Δ leakage+0.147[0.107, 0.186]
reported
honest
89% of the reported metric is leakage — it disappears once train/test homology is removed. The honest metric's CI includes zero: a statistical null.
24.6%contaminated by train/test homology
mmseqs flagged 1230/5000 · identity ≥ 0.80
| identity to nearest reference | spearman | 95% CI | n | note |
|---|---|---|---|---|
| [0.0, 0.25) | 0.018 | — | 3770 | silent failure |
| [0.75, 1.0) | 0.352 | — | 1230 | — |
input hashes
regulatory_dna_gosai:evaluator_data.zip
identity_threshold
0.8
n_test
5000
n_train
20000
split
random_split
tool versions
toolpinnedruntime
mmseqs18.8cc5c18.8cc5c
Bootstrap confidence intervals under-cover at small n
Confidence intervals use the percentile bootstrap, which mildly under-covers at small sample sizes: empirically about 0.927 coverage at n=25 for a nominal 95% interval, rising toward nominal as n grows. Treat intervals on small buckets or sets as optimistic.