documentation · schema v2

Auditing benchmark integrity

Overview

Veritas checks whether a model's reported benchmark performance transfers beyond the exact evaluation protocol. It records prompts and responses, runs reproducible transformations and controls, computes paired uncertainty, and packages the evidence with provenance and a deterministic audit hash.

Workflow

audit specification
  → validate benchmark + privacy choices
  → inspect request, token, and cost limits
  → evaluate canonical items
  → generate and validate transformed items
  → run available detectors and controls
  → compute paired statistics
  → write JSON + Markdown + HTML evidence package

Model adapters support replay, mock responses, OpenAI-compatible HTTP endpoints, vLLM-compatible servers, local Hugging Face Transformers, and custom subprocesses. Content-addressed caching makes interrupted runs resumable and includes model and generation settings in every cache key.

Evidence, not a verdict

Findings distinguish exact or near-reference exposure, semantic exposure, template and protocol dependence, choice or position sensitivity, evaluator exploitation, distribution shift, fresh-set transfer, legitimate capability, and inconclusive or unavailable evidence. Every detector declares its requirements and emits not_runwhen those requirements are absent.

Statistics

Score variants retain parent-item lineage. Comparisons use paired parent-item bootstrap intervals, permutation tests, and McNemar analyses where applicable, with clustered uncertainty and effect sizes. The configurable Benchmark Robustness Index is explicitly uncalibrated and must not be read as a probability of contamination.

Install

The package is not published on PyPI yet. Install from a local checkout; the second form adds direct local Transformers inference, the named benchmark catalog, and may install PyTorch.

python -m pip install -e "/absolute/path/to/veritas[cli]"
python -m pip install -e "/absolute/path/to/veritas[cli,local-model,benchmarks]"

# or, from the repository root
uv sync --extra cli --extra local-model --extra benchmarks

veritas benchmarks
veritas benchmarks mmlu

Run an audit

veritas inspect --config audit.yaml
veritas audit --config audit.yaml --out audit-run/

# uv-managed checkout
uv run veritas inspect --config examples/benchmark_integrity/open_weight_audit.json
uv run veritas audit \
  --config examples/benchmark_integrity/open_weight_audit.json \
  --out audit-run/

A catalog audit uses benchmark: {adapter: catalog, name: mmlu, limit: 100}. Presets currently cover MMLU, MMLU-Pro, ARC-Challenge, OpenBookQA, HellaSwag, CommonsenseQA, PIQA, WinoGrande, TruthfulQA, BoolQ, GPQA, HumanEval, and SWE-bench Verified. inspect validates the specification and estimates requests, tokens, cost, and private-data transfer before inference. The audit directory contains machine-readable data and human-readable reports. Open report.json in the report viewer to inspect prompts, responses, transformations, score comparisons, evidence, unavailable tests, and provenance.

Private benchmarks

External adapters require an explicit per-run acknowledgement when the benchmark is marked private. Transfer choices are redacted in provenance, and secrets are excluded from reports and cache keys. Use a local adapter when benchmark terms prohibit external disclosure.

Interpretation

“Benchmark maxxing” is useful shorthand, not a scientific outcome class. A large drop under a valid transformation can show protocol fragility; it does not by itself show training-set contamination. Strong claims require calibrated detectors, matched controls, alternatives, and uncertainty that support that specific claim.

Legacy biology

The sequence-based leakage auditor remains supported as a distinct legacy domain. Its existing report fixtures open in the legacy v1 viewer. Command migration and the v2 schema are documented in the repository's reference docs.