Benchmarks
Promptivo’s scoring is calibrated against an LLM-judge standard — a five-run Gemini 2.5 Flash consensus over 4,541 prompts — and then frozen into a deterministic engine. This page shows the full evidence: how prompts score on two academic datasets, how closely the frozen engine tracks the judge it was calibrated to, and where the two still disagree.
IFEval Dataset 541 prompts
Google Research — Instruction Following Evaluation
IFEval contains carefully constructed prompts designed to test whether models can follow specific instructions. These are research-grade prompts written by domain experts.
Per-Dimension Scores
Key finding: Even research-quality prompts leave headroom — the weakest areas were Concise Chain-of-Thought and Structural Compliance, confirming that even careful writers rarely spell out reasoning structure or explicit output requirements.
LLM-as-a-Judge Comparison Gemini 2.5 Flash · 5 runs · 539 prompts
Promptivo Deterministic vs. Gemini 2.5 Flash — IFEval Dataset, 5 independent runs
We ran the same 541 IFEval prompts through Google’s Gemini 2.5 Flash model five independent times, asking it to score each prompt on the same 7 dimensions using a 1–5 scale. Running multiple trials lets us measure both how Gemini compares to Promptivo and how consistent Gemini’s own judgments are across repeated evaluations. Promptivo’s calibrated scoring heads were trained on the consensus of these runs; the correlation figures below are reported on a 20% hold-out of MePO pairs that the training never saw.
| Dimension | Promptivo | Gemini (avg) | Diff | Gemini σ |
|---|---|---|---|---|
| Overall Score | 3.09 | 3.45 | +0.36 | 0.007 |
| Clarity | 4.22 | 3.85 | −0.37 | 0.004 |
| Precision | 3.50 | 3.07 | −0.43 | 0.001 |
| Chain-of-Thought | 2.27 | 2.68 | +0.41 | 0.009 |
| Info Completeness | 3.54 | 3.10 | −0.44 | 0.006 |
| Constraint Verifiability | 2.69 | 3.79 | +1.10 | 0.003 |
| Structural Compliance | 2.46 | 3.38 | +0.92 | 0.007 |
| Info Integrity | 2.97 | 3.38 | +0.41 | 0.004 |
Result Discrepancy Analysis
The five Gemini runs were independent — no shared state, same temperature (0). Comparing their outputs reveals how stable LLM-as-a-judge scoring is in practice.
| Stability bucket | Prompts | Share | Interpretation |
|---|---|---|---|
| σ < 0.2 — very stable | 475 | 88.1% | Gemini agrees with itself across runs |
| σ 0.2–0.5 — minor jitter | 64 | 11.9% | Typically a 0.5-point swing in one run |
| σ ≥ 0.5 — high variance | 0 | 0% | No prompts showed large score swings |
Dimensional pattern: The remaining gaps are concentrated and systematic. Gemini is more generous on Constraint Verifiability (+1.10) and Structural Compliance (+0.92): it credits constraints a reader would infer, while the deterministic engine only scores what is written explicitly. In the other direction no dimension diverges by more than 0.44. The offsets are stable across runs (per-dimension Gemini σ ≤ 0.009) — known trade-offs of an explicit-markers engine, not noise.
Divergent prompts: 43% of prompts land within half a point of the judge consensus (mean gap 0.65); 36 of 541 (6.7%) still diverge by more than 1.5 points. The stubborn cases follow one pattern: short prompts whose constraint is semantically obvious to a reader (“answer in lowercase only”) but carries none of the structural markers a deterministic engine can score. Closing those would require semantic comprehension at scoring time — exactly the runtime cost Promptivo is designed to avoid. We trade a few edge cases for determinism, privacy, and zero marginal cost.
Key finding: The frozen engine lands close to the judge on average (3.09 vs 3.45) and, on the wide-range MePO set, ranks prompts at ρ = 0.55 against the consensus — 0.53 on held-out pairs the training never saw. IFEval’s deliberately narrow quality band keeps rank correlation modest on this dataset; the dimensional gaps above are where the honest differences live. Gemini itself is remarkably self-consistent at temperature 0 (88% of prompts identical across 5 runs, mean σ = 0.04) — a good standard-setter, but one that charges tokens, latency, and your prompt’s privacy on every call. Promptivo froze that standard into an engine that costs nothing at scoring time.
A Second Judge Claude Sonnet 4.6 · 541 prompts
Claude Sonnet 4.6 vs. Gemini 2.5 Flash vs. Promptivo — IFEval Dataset
To test whether the calibration generalizes beyond its training judge, we had Anthropic’s Claude Sonnet 4.6 score the same 541 prompts with the same 7-dimension rubric. Sonnet contributed zero training labels — it is a fully independent second opinion, on both the engine and on Gemini itself.
| Dimension | Promptivo | Gemini (avg) | Sonnet |
|---|---|---|---|
| Overall Score | 3.09 | 3.45 | 2.81 |
| Clarity | 4.22 | 3.85 | 3.65 |
| Precision | 3.50 | 3.07 | 3.10 |
| Chain-of-Thought | 2.27 | 2.68 | 1.81 |
| Info Completeness | 3.54 | 3.10 | 2.99 |
| Constraint Verifiability | 2.69 | 3.79 | 3.85 |
| Structural Compliance | 2.46 | 3.38 | 3.07 |
| Info Integrity | 2.97 | 3.38 | 2.43 |
Key finding: Two frontier judges, same prompts, same rubric — and they disagree by 0.64 points on average, with 15% of prompts more than a full point apart and only ρ = 0.61 rank agreement. That number is the practical ceiling for any judge-aligned scorer: you cannot agree with “the judge” more closely than judges agree with each other. Against that ceiling, the frozen engine tracks the judge it never saw at ρ = 0.30 on this deliberately narrow dataset — nearly three times its agreement with its own training judge here — evidence the calibration learned transferable structure, not one model’s quirks. On five of seven dimensions the engine lands within 0.45 of at least one judge; where it sits below both (Constraint Verifiability, Structural Compliance) it is by design: judges credit constraints a reader would infer, while the engine scores only what is written. And the judges’ own 0.64-point offset is the “no fixed standard” problem, measured: change the judge, and every score in your history moves.