Evaluation, Benchmarking & Experimentation Datasheet — SOTA++ 2026¶

A model does not simply “have performance.” It produces a distribution of outcomes under a specific experimental configuration. Evaluation is the engineering discipline required to separate real gains from noise, harness effects, prompt effects, contamination, and wishful thinking.

The central conceptual equation is:

\[ \boxed{ ObservedScore = f( Model, Prompt, Inference, Harness, Dataset, Metric, Randomness, Environment ) } \]

Therefore:

\[ Score(Model)\quad\text{is usually a dangerous shorthand.} \]

1. What exactly is being evaluated?¶

Before computing any metric, define the System Under Test — SUT.

It may be:

1.1 Raw checkpoint¶

\[ SUT=Weights+Tokenizer \]

1.2 Model + prompt/template¶

\[ SUT=Model+PromptPolicy \]

1.3 Model + inference configuration¶

\[ SUT=Model+Prompt+Sampler \]

1.4 Pipeline¶

\[ SUT=Retriever+Reranker+Model \]

1.5 Agent¶

\[ SUT=Model+Scaffold+Tools+Environment \]

1.6 Complete product¶

\[ SUT=EverythingTheUserExperiences \]

If the SUT is not explicit, score comparisons may be invalid.

2. Experimental error decomposition¶

Think in terms of:

\[ Y_{ijk} = \mu +M_i +H_j +R_k +\epsilon_{ijk} \]

where:

\(M_i\): model/configuration effect;
\(H_j\): harness/prompt/scaffold effect;
\(R_k\): seed/randomness effect;
\(\epsilon\): residual.

We want to estimate \(M_i\), but we often measure all of them at once.

3. Benchmark ≠ metric ≠ harness¶

Benchmark¶

Defines the set of tasks/examples/protocol.

Metric¶

Maps an outcome to a scalar/vector:

\[ Metric(y,\hat y) \]

Harness¶

Executes:

prompt construction;
model call;
decoding;
parsing;
scoring;
aggregation.

Changing the harness can change the score even when the weights are identical.

4. The minimum experimental unit¶

A record should contain at least:

experiment:
  model_id: ...
  model_hash: ...
  tokenizer: ...
  quantization: ...
  backend: ...
  backend_version: ...
  prompt_template: ...
  system_prompt_hash: ...
  context_limit: ...
  generation:
    temperature: ...
    top_p: ...
    min_p: ...
    max_tokens: ...
    seed: ...
  dataset:
    name: ...
    version: ...
    split: ...
    hash: ...
  metric: ...
  hardware: ...
  timestamp: ...

Without versioning, a result becomes historical anecdote.

5. Deterministic metric versus judged metric¶

Deterministic¶

\[ score=g(reference,prediction) \]

Examples:

exact match;
test pass/fail;
numerical tolerance;
WER;
retrieval recall.

Judged¶

\[ score=Judge(prompt,response,rubric) \]

Another model or a human infers quality.

This adds another source of error:

\[ Var_{judge} \]

See 12_REWARD_VERIFIER_JUDGE_DATASHEET.

6. Accuracy¶

\[ Accuracy= \frac{\sum_i\mathbf1(\hat y_i=y_i)}{N} \]

Simple, but it requires an unambiguous definition of “correct.”

For imbalanced classes, accuracy can be misleading.

7. Precision, Recall, and F1¶

\[ Precision= \frac{TP}{TP+FP} \]

\[ Recall= \frac{TP}{TP+FN} \]

\[ F1= 2\frac{Precision\cdot Recall}{Precision+Recall} \]

Macro and micro averaging answer different questions. These ratios require nonzero denominators; declare how undefined cases are scored or excluded before averaging.

8. Exact Match¶

\[ EM= \frac1N\sum_i\mathbf1(normalize(\hat y_i)=normalize(y_i)) \]

The normalize() function is part of the metric.

Change it and you change the benchmark.

9. Pass@k¶

In code generation, with \(n\) samples and \(c\) correct ones:

\[ pass@k = 1- \frac{\binom{n-c}{k}}{\binom nk} \]

when the estimator's assumptions hold.

Pass@1 measures a different operational reality from pass@100.

\[ SearchBudget\uparrow \Rightarrow pass@k\uparrow \]

without the model itself changing.

10. Retrieval metrics¶

Recall@K¶

\[ Recall@K = \frac{RelevantRetrieved@K}{TotalRelevant} \]

Precision@K¶

\[ Precision@K = \frac{RelevantRetrieved@K}{K} \]

MRR¶

\[ MRR = \frac1N \sum_i \frac1{rank_i} \]

Here \(rank_i\) is the rank of the first relevant result for query \(i\); its reciprocal-rank contribution is zero if no relevant result is retrieved.

DCG¶

\[ DCG@K = \sum_{i=1}^{K} \frac{2^{rel_i}-1}{\log_2(i+1)} \]

NDCG¶

\[ NDCG@K = \frac{DCG@K}{IDCG@K} \]

Declare the policy for queries with no relevant documents or zero ideal DCG, and for fewer than \(K\) returned results; these choices affect averages and comparability.

See 11_EMBEDDING_RETRIEVAL_RERANKER_DATASHEET.

11. Perplexity¶

For a sequence:

\[ PPL= \exp\left( -\frac1T \sum_t \log p(x_t|x_{\lt t}) \right) \]

Useful for probabilistic language modeling, but:

\[ LowerPPL \not\Rightarrow BetterInstructionFollowing \]

necessarily.

Comparing perplexity across different tokenizers requires extreme care.

12. Calibration¶

If a model reports probability \(p\), ideally:

\[ P(correct|confidence=p)\approx p \]

Brier score¶

For binary outcomes \(y_i\in\{0,1\}\) and predicted probabilities \(p_i\) of outcome 1:

\[ BS= \frac1N\sum_i(p_i-y_i)^2 \]

ECE¶

Bins predictions by confidence and measures the gap between confidence and accuracy.

Calibration is distinct from accuracy.

13. ASR¶

Word Error Rate¶

\[ WER= \frac{S+D+I}{N} \]

where:

\(S\): substitutions;
\(D\): deletions;
\(I\): insertions.

See 06_AUDIO_MUSIC_TTS_GENERATION_DATASHEET.

14. Time-series metrics¶

MAE¶

\[ MAE= \frac1N\sum_i|y_i-\hat y_i| \]

RMSE¶

\[ RMSE= \sqrt{ \frac1N\sum_i(y_i-\hat y_i)^2 } \]

MAPE caveat¶

\[ MAPE= \frac{100}{N} \sum_i \left| \frac{y_i-\hat y_i}{y_i} \right| \]

It explodes or becomes unstable when \(y_i\approx0\).

Probabilistic forecasts require distribution- or quantile-aware metrics.

See 14_TIME_SERIES_TABULAR_FOUNDATION_MODELS.

15. Image/video/audio generation¶

There is no single sufficient “quality metric.”

Distinct dimensions include:

fidelity;
prompt adherence;
identity consistency;
temporal consistency;
perceptual quality;
diversity;
human preference.

Therefore:

\[ Quality \in \mathbb R^d \]

not necessarily:

\[ Quality\in\mathbb R \]

A single scalar ranking hides trade-offs.

16. 3D evaluation¶

May include:

Chamfer distance;
normal consistency;
IoU;
watertightness;
non-manifold edges;
geometric validity;
render-space perceptual metrics.

For manufacturing:

\[ Printable\neq VisuallyPlausible \]

See 07_3D_GENERATION_REPRESENTATION_DATASHEET.

17. Agent evaluation¶

Agents require metrics beyond the final text:

\[ SuccessRate \]

\[ StepsToSuccess \]

\[ ToolCalls \]

\[ Cost \]

\[ UnsafeActions \]

\[ RecoveryRate \]

\[ StateValidity \]

A polished answer can still accompany a terrible operational trajectory.

18. World/VLA evaluation¶

Separate:

prediction accuracy;
rollout fidelity;
action success;
collision/safety rate;
horizon degradation;
sim-to-real gap.

World-model error can accumulate:

\[ \epsilon_{t+H} =f(\epsilon_t,H,dynamics) \]

See 13_WORLD_MODEL_VLA_EMBODIED_AI_DATASHEET.

19. Statistical uncertainty is not optional¶

An observed score is a sample:

\[ \hat\mu \]

We want to estimate:

\[ \mu \]

with uncertainty.

20. Standard error¶

For \(N\) independent observations with a common mean and common finite variance, the estimated standard error of their mean is:

\[ SE= \frac{s}{\sqrt N} \]

Under these assumptions, increasing the number of independent observations reduces uncertainty approximately as:

\[ 1/\sqrt N \]

not linearly with \(N\).

21. Confidence interval¶

A simple asymptotic form:

\[ CI \approx \hat\mu \pm z_{\alpha/2}SE \]

Here \(z_{\alpha/2}=\Phi^{-1}(1-\alpha/2)\) is the positive upper-tail critical value of the standard normal distribution, for \(0<\alpha<1\); \(\Phi\) is its CDF.

For many benchmarks, bootstrap is more flexible.

22. Bootstrap¶

First define the statistic, target population and sampling unit. For independent examples:

sample examples with replacement;
recompute the metric;
repeat \(B\) times;
obtain an empirical distribution.

\[ \{\hat\theta_1^*,\ldots,\hat\theta_B^*\} \]

Useful for metrics without a simple analytical variance formula. For clustered or dependent observations, choose a cluster/block resampling scheme or a statistical model consistent with the sampling design and estimand; do not treat every item/seed row as independent. Preserve A/B pairs when resampling paired comparisons. A hierarchical bootstrap is appropriate only when its hierarchy matches that design.

23. Paired design¶

If models A and B answer the same items, use paired differences:

\[ d_i=s_i(A)-s_i(B) \]

Then study:

\[ \bar d \]

This removes some variation caused by item difficulty.

It is generally better than treating the two means as independent samples.

24. McNemar for paired correct/incorrect outcomes¶

For binary classification, count discordant outcomes:

A correct / B wrong: \(b\)
A wrong / B correct: \(c\)

Evidence for a difference comes from \(b\) versus \(c\), not from isolated total accuracies.

25. Permutation test¶

Under the null hypothesis that labels/configurations are exchangeable, permute the paired assignments and compute the distribution of \(\Delta\).

Useful when we do not want to assume normality.

26. Effect size¶

Statistical significance does not imply practical significance.

Ask:

\[ |\Delta|\quad\text{is it large enough to justify the cost/risk?} \]

Example:

+0.2 benchmark points may not justify +70% VRAM.

27. Multiple comparisons¶

If you test hundreds of configurations, some “winner” will emerge by chance.

\[ P(false\ positive)\uparrow \]

with the number of hypotheses.

Use:

a held-out final set;
corrections when appropriate;
preregistered hypotheses;
do not select seeds/configurations on the test set.

28. Seed variance¶

For stochastic generation:

\[ Y_{m,s} \]

where \(s\) is the seed.

Estimate:

\[ Var_s(Y|m) \]

before interpreting differences between models.

If:

\[ |\Delta_{models}| \lt \sigma_{seed} \]

a single run is weak evidence.

29. Hierarchical variance¶

In a benchmark with tasks, examples, and seeds:

\[ Y_{task,item,seed} \]

variance can be decomposed across multiple levels.

This reveals whether the result is dominated by:

a few categories;
a few items;
randomness.

A macro-average can hide all of this. Use the sampling design and target population to choose the independent unit for the uncertainty estimates in sections 20–22; repeated seeds on the same item do not create new independent items.

30. Stochastic benchmark protocol¶

For open-ended generation:

for model/config:
    for item:
        for seed in S:
            generate
            score

Then report:

mean;
median;
std/IQR;
CI;
failure rate;
tails.

31. Tail metrics¶

Users do not experience only “the mean.”

For latency:

\[ p50,p95,p99 \]

For safety/reliability, worst-case and failure-tail behavior can dominate risk.

For agents:

\[ P(catastrophic\ failure) \]

may matter more than mean success rate.

32. Benchmark contamination¶

Types:

Exact contamination¶

The identical item appears in training.

Near-duplicate¶

A paraphrase/transformation appears.

Solution contamination¶

The answer/solution appears in the data.

Benchmark-aware post-training¶

The model was explicitly optimized for the benchmark.

Even without literal leakage, the score stops being a neutral estimate of generalization.

33. Benchmark saturation¶

If nearly all relevant models are close to the ceiling:

\[ Var_{models}(score)\rightarrow0 \]

the benchmark loses discriminative power.

A 2026 study analyzing 60 benchmarks found saturation across a large and growing share as benchmarks aged; expert-curated benchmarks were, on average, more resistant than crowdsourced ones.

The lesson:

\[ BenchmarkHalfLife\lt\infty \]

34. Dynamic benchmarks¶

Benchmarks updated periodically try to reduce contamination and saturation.

LiveBench exemplifies:

refreshed questions;
recent sources;
objective ground truth where possible;
multiple categories.

But updating also means:

\[ Score_{2025}\not\equiv Score_{2026} \]

unless the dataset version is tracked.

35. Public versus private test sets¶

Private test data reduces direct exposure, but does not eliminate:

conceptual leakage;
benchmark-targeted optimization;
saturation;
similar training examples.

Therefore:

\[ Private\neq ContaminationProof \]

36. LLM-as-a-Judge¶

Judge:

\[ J(prompt,response,rubric) \rightarrow score/verdict \]

Common problems:

position bias;
verbosity bias;
style bias;
self-preference;
reference anchoring;
domain mismatch.

See 12_REWARD_VERIFIER_JUDGE_DATASHEET.

37. Swap test¶

For a pairwise judge:

judge A,B;
reverse the order to B,A;
compare.

If:

\[ J(A,B)\neq reverse(J(B,A)) \]

there is inconsistency/position sensitivity.

38. Rubric decomposition¶

Instead of a monolithic score:

\[ Quality = (w_1q_1,\ldots,w_kq_k) \]

Possible dimensions:

correctness;
completeness;
relevance;
style;
safety;
evidence.

This improves auditability.

39. Human evaluation¶

Human evaluation is not magical ground truth.

It has:

inter-rater disagreement;
fatigue;
rubric ambiguity;
expertise variation;
context effects.

Agreement metrics help quantify consistency.

40. Model versus scaffold¶

A coding-agent benchmark often measures:

\[ Model + AgentLoop + Tools + Prompt + Search + RetryBudget \]

not only the model checkpoint.

SWE-bench Verified, for example, explicitly compares systems ranging from simple loops to RAG/multi-rollout/review systems.

Therefore an agent leaderboard is not a pure LLM leaderboard.

41. Harness sensitivity¶

Small changes in:

few-shot examples;
answer parsing;
stop sequences;
whitespace;
chat template;
reasoning budget;

can change results.

A benchmark without a versioned harness is not fully reproducible.

42. Quality versus test-time compute¶

If configuration A uses:

\[ N_A=1 \]

sample while B uses:

\[ N_B=64 \]

plus a verifier,

we are not comparing only models.

We are comparing:

\[ Model\times SearchBudget \]

Reporting:

\[ Quality(Cost) \]

is more honest.

43. Best-of-N evaluation¶

Generate:

\[ y_1,\ldots,y_N \]

Select using a fixed tie-breaking rule:

\[ i^*=\arg\max_{i\in\{1,\ldots,N\}} V(y_i),\qquad y^*=y_{i^*} \]

Score depends on:

generator diversity;
N;
verifier quality.

For a fixed verifier and candidate sets enlarged by inclusion, the maximum proxy score does not decrease. True task quality can decrease through verifier error or overoptimization; greater diversity also does not guarantee a gain.

Attributing the entire gain to the generator is a causal error.

44. Runtime evaluation¶

TTFT¶

Time To First Token.

If a request starts at \(t_0\) and the first token arrives at \(t_1\):

\[ TTFT=t_1-t_0 \]

Inter-token latency / TPOT¶

Average time per token after the first token.

Generation TPS¶

\[ TPS= \frac{N_{generated}}{T_{decode}} \]

Count only tokens produced during the stated \(T_{decode}\) window. If timing from the first token's arrival \(t_1\) to the last token's arrival \(t_N\), the rate is \((N-1)/(t_N-t_1)\) for \(N>1\), with the first token excluded from the count. Report the window separately from end-to-end request time.

Throughput¶

\[ Throughput= \frac{TotalTokens}{WallClock} \]

Single-user TPS and server throughput optimize different things.

45. Prefill versus decode benchmark¶

Never conflate:

\[ PromptTPS \]

with:

\[ DecodeTPS \]

See 09_LOCAL_AI_INFERENCE_RUNTIME_DATASHEET.

Prefill is much more parallelizable; decode is often memory-bandwidth bound.

46. Memory metric¶

Record:

model-load VRAM;
idle VRAM;
peak prefill;
peak decode;
host RAM;
KV allocation;
temporary workspace.

A single “VRAM used” number is ambiguous without an execution phase.

47. Energy / efficiency¶

When available:

\[ EnergyPerToken = \frac{\int P(t)dt}{N_{tokens}} \]

or:

\[ J/request \]

For local inference, performance per watt can affect hardware decisions as much as TPS.

48. Pareto analysis¶

Instead of searching for the “best model,” construct a vector:

\[ z= (Quality,Latency,Memory,Cost,Context) \]

A model is Pareto-dominated if another model is at least as good in every dimension and strictly better in at least one.

For an RTX 2060 12 GB:

\[ VRAM\le12GB \]

is a real constraint, not a decorative leaderboard column.

49. Quality per physical budget¶

Useful metrics include:

\[ Quality/GB \]

\[ Quality/Joule \]

\[ Quality/Second \]

\[ Quality/TokenBudget \]

especially when comparing quantizations and model sizes. These ratios require a declared quality metric with a meaningful zero and a positive budget denominator. For scores with an arbitrary origin, use Pareto analysis or explicitly defined gains over a baseline; shifting the score origin can reverse ratio rankings.

50. Quantization experiment¶

When comparing Q4 versus Q5:

Control:

model source;
context;
sampler;
backend;
prompt set.

Observe:

task quality;
logit divergence where available;
VRAM;
TPS;
TTFT.

The questions are:

\[ \frac{\Delta Quality}{\Delta Memory} \]

and:

\[ \frac{\Delta Quality}{\Delta Throughput} \]

51. A/B sampler experiment¶

To compare Min-P:

same model;
same prompt set;
same seed set where the backend permits;
all other samplers fixed;
multiple outputs.

Do not use “I thought this text looked nicer” as the only statistic.

Capture:

repetition rate;
lexical diversity;
judge rubric;
human pairwise preference;
failure counts.

52. Evaluation dataset design¶

The sample should represent the target distribution:

\[ D_{eval}\sim D_{deployment} \]

as closely as practical.

If the benchmark is academic while deployment is Portuguese-language B2B customer service:

\[ ExternalValidity\downarrow \]

even with an excellent score.

53. Slice analysis¶

Never report only an aggregate score.

Partition by:

language;
difficulty;
length;
domain;
modality;
risk level;
tool type.

It is possible that:

\[ Mean_A>Mean_B \]

while:

\[ A\lt B \]

on the slice that actually matters in production.

54. Regression suite¶

After selecting a system, maintain a fixed regression set:

must-pass
known-hard
known-failure
security
format
latency

Run it again after every update.

This turns benchmarking into process control.

55. Golden cases¶

Small, semantically important cases inspected manually.

They do not replace a broad benchmark.

They are useful for detecting obvious regressions with high interpretability.

56. Canary tests¶

Cases designed to detect one specific failure.

Examples:

template corruption;
context truncation;
tool hallucination;
Unicode/tokenizer bugs.

A good canary has clear causality.

57. Adversarial evaluation¶

Normal evaluation measures the nominal distribution.

Adversarial evaluation measures a hostile neighborhood:

\[ x' = x+\delta \]

or inputs deliberately constructed to trigger failure.

See 18_SECURITY_ROBUSTNESS_FAILURE_MODES_DATASHEET.

58. Evaluation leakage through the development loop¶

Even without training model weights:

if you repeatedly inspect the test set and adjust prompts/configuration:

\[ TestSet\rightarrow DevelopmentSignal \]

then it is no longer a clean test set.

Maintain:

a development set;
a final holdout.

59. Organizational benchmark overfitting¶

A team can optimize the KPI until it loses external validity.

\[ Leaderboard\uparrow \not\Rightarrow RealWorldUtility\uparrow \]

This is Goodhart applied to evaluation.

60. Benchmark retirement¶

Reasons to retire or downgrade a benchmark include:

saturation;
known contamination;
poor external validity;
metric no longer relevant;
harness ambiguity;
obsolete task.

There is no virtue in preserving historical comparability at the expense of current signal.

61. SOTA snapshot — 2026¶

Points particularly relevant in October 2026:

Benchmark contamination is no longer a peripheral caveat; it is a central design variable.
2026 studies show broad saturation across older benchmarks and reinforce the need for rotating and expert-curated suites.
Live/dynamic benchmarks are a useful response, but require temporal dataset versioning.
LLM-as-a-Judge has scaled open-ended evaluation, but position, verbosity, style, and calibration bias must be measured rather than assumed absent.
Coding-agent evaluation has made the model vs scaffold/system distinction explicit.
HELM remains a conceptual reference for holistic/reproducible evaluation, although the general project entered maintenance mode in June 2026; this reinforces the need to avoid dependence on a single harness.
Modern evaluation is multi-objective: capability, reliability, safety, latency, memory, and cost should be reported together.

62. Experiment card — recommended template¶

## Hypothesis

## System under test

## Independent variable

## Controlled variables

## Dataset + version + hash

## Metric definitions

## Seed/repetition policy

## Hardware/runtime

## Results
- mean:
- median:
- std/IQR:
- confidence interval:
- failure count:

## Paired differences

## Slice analysis

## Physical cost
- VRAM:
- RAM:
- TTFT:
- TPS:
- energy/cost:

## Decision rule

## Caveats

63. Decision rule before the experiment¶

Define in advance:

\[ Accept\ A \quad\text{if}\quad \Delta Q>\delta_{min} \]

subject to:

\[ Memory\lt M_{max} \]

\[ Latency\lt L_{max} \]

This reduces post-hoc rationalization.

64. Minimal local-model test battery¶

For a local stack, I would keep at least:

task accuracy / instruction adherence;
Portuguese naturalness;
code correctness where applicable;
long-context retrieval;
repetition/degeneration;
structured-output validity;
TTFT;
decode TPS;
peak VRAM/RAM;
context scaling;
your own known failure cases;
safety/tool tests for agentic systems.

For local decisions, this battery should be more valuable than an isolated position on a public leaderboard.

65. The final equation¶

\[ \boxed{ Evidence = Measurement + Uncertainty + Protocol + Reproducibility } \]

Without uncertainty, we have a number.

Without protocol, we have benchmark folklore.

Without reproducibility, we have an anecdote.

Checklist for any benchmark that appears tomorrow¶

What SUT is being measured?
Raw model or complete system?
Which dataset version?
Public or private?
Is there contamination evidence?
Is it saturated?
What metric?
Is the metric objective or judged?
Which judge?
Was the judge calibrated?
Was order-swapping tested?
How many samples/seeds?
What inference budget?
Which prompt/template?
Which harness/version?
What context limit?
What quantization?
What hardware/backend?
Are CI/variance reported?
Is the comparison paired?
Is there slice analysis?
Is physical cost reported?
What is the external validity?
Does the score improve something that actually matters?
Would the decision change if the benchmark moved within its CI?

Snapshot references¶

LiveBench — https://arxiv.org/abs/2406.19314
LiveBench ICLR 2025 — https://proceedings.iclr.cc/paper_files/paper/2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html
When AI Benchmarks Plateau — https://proceedings.mlr.press/v306/akhtar26a.html
HELM — https://github.com/stanford-crfm/helm
SWE-bench Verified — https://www.swebench.com/verified.html
Judging the Judges: Position Bias — https://arxiv.org/abs/2406.07791
Mitigating the Bias of Large Language Model Evaluation — https://arxiv.org/abs/2409.16788
12_REWARD_VERIFIER_JUDGE_DATASHEET for 2026 judge/reward references