Omni-MATH

Overview

Omni-MATH was built after GSM8K and MATH had stopped separating the strongest reasoning systems. It contains 4,428 text-only competition problems, all at Olympiad level rather than mixing mathematics with multimodal perception or general knowledge. The source pool spans introductory competitions, national and international Olympiads, Putnam, and the International Mathematics Competition for University Students.

The construction counts are unusually transparent. Official contest pages contribute 3,242 retained problems after rule filtering and human review, AoPS Wiki contributes 298, and AoPS Forum contributes 888. Forum items must provide multiple candidate solutions with consistent final answers before they reach annotators. The finished problems are organized into more than 33 subdomains and fine-grained difficulty ratings from 0 to 10.

Free-form Olympiad answers defeat simple string matching: equivalent expressions may look different, and a correct final answer may require interpreting the surrounding reasoning. The main experiment therefore uses GPT-4o to compare each complete model solution with the reference. Omni-Judge is an open, lower-cost verifier trained on those judgments, while Omni-MATH-Rule is a separate deterministically checkable subset. The TokenWave table uses only full-set, GPT-4o-evaluated Overall Accuracy from the paper's main Table 3.

Why it matters

One overall score is useful only if the underlying test remains discriminative. Omni-MATH's upper difficulty tiers and Olympiad-only scope leave substantial headroom even for reasoning-focused systems, while its domain tree shows whether gains come from algebra and calculus or extend to scarcer areas such as discrete mathematics.

Evaluator choice is part of the benchmark contract. GPT-4o achieves 98% agreement with human labels on the paper's meta-evaluation, and Omni-Judge exceeds 90% consistency with GPT-4o. Keeping the full-set judge-based scores separate from Omni-MATH-Rule prevents an easier-to-grade subset from being mistaken for the benchmark's main 4,428-problem result.

Contributions

• Releases 4,428 English, text-only Olympiad-level mathematics problems with solutions, final answers, source metadata, domain labels, and difficulty labels.
• Builds a hierarchical taxonomy with more than 33 mathematical subdomains and difficulty ratings from introductory competitions through international Olympiad and university contests.
• Validates free-form answer evaluation with GPT-4o and releases Omni-Judge, trained on GPT-4o labels, as a lower-cost open verifier with more than 90% consistency against GPT-4o.
• Evaluates 15 vanilla and test-time-scaled model settings by overall, domain, and four difficulty-tier accuracies, with contamination and evaluator-reliability analyses.

Method & evaluation

• The collection starts with 3,750 items from official contest pages, 8,575 from AoPS Wiki, and 17,502 from AoPS Forum. Rule filtering and human review retain 3,242, 298, and 888 items respectively, totaling 4,428.
• Official PDFs are converted to LaTeX with MathPix. Forum problems require at least three candidate answers and consistent extracted finals; four annotators each review 550 of the 1,100 filtered items so every item has two labels, yielding 92.7% cross-validation accuracy and 97.3% accuracy in a 150-item audit.
• Difficulty is assigned on a 0–10 contest scale, including fractional levels, using AoPS ratings where available and GPT-4o in-context classification otherwise. GPT-4o also maps each problem into a guidebook-derived hierarchical domain tree with more than 33 leaves.
• Fifteen model settings answer the full set with repository-specific prompts. GPT-4o receives the original problem, the reference answer, and the complete model solution, then judges whether the model's conclusion is consistent with the reference.
• Omni-Judge is trained from roughly 17,618 GPT-4o-labeled examples, with about 2,200 validation and 2,200 test examples drawn from non-overlapping questions. It generates an extracted answer, judgment, and explanation, but the leaderboard here retains the paper's GPT-4o-evaluated main-table scores.
• The table ranks Overall Acc from the full 4,428 problems. Omni-MATH-Rule is a separate filtered subset for deterministic checking and is not used in any displayed row.

Evaluation metrics

Overall Acc (%) — Higher is better. Percentage of all 4,428 model solutions judged correct by GPT-4o; the primary metric for the displayed leaderboard. Score range: [0, 100].

Domain Accuracy — Higher is better. Accuracy reported for first-level domains: algebra, precalculus, calculus, geometry, discrete mathematics, number theory, and applied mathematics. Score range: [0, 100].

Difficulty-tier Accuracy — Higher is better. Accuracy by four bins derived from the fine-grained difficulty tag: T1 = 1–3, T2 = 3–5, T3 = 5–7, and T4 = 7–10. Score range: [0, 100].

Leaderboard

Top six rows by Overall Acc in Table 3. Every score covers the complete 4,428-problem Omni-MATH set and uses GPT-4o as evaluator. Vanilla and reward-model-selected inference settings remain explicitly labeled; no Omni-MATH-Rule result is included.

#	Model	Overall Acc (%)
1	OpenAI o1-mini	60.54
2	OpenAI o1-preview	52.55
3	Qwen2.5-MATH-72B-Instruct RM@8	36.34
4	Qwen2.5-MATH-72B-Instruct	36.2
5	Qwen2.5-MATH-72B-Instruct RM@256	35.95
6	Qwen2.5-MATH-7B-Instruct RM@256	35.79

Reading the results

OpenAI o1-mini leads at 60.54% Overall Accuracy, with o1-preview at 52.55%. The 7.99-point gap between the two is large, but even o1-mini misses nearly two out of five problems. The best vanilla open model in the source table, Qwen2.5-MATH-72B-Instruct, scores 36.20%, 24.34 points behind the leader.

Reward-model selection does not produce a clean scaling curve. Qwen2.5-MATH-72B RM@8 reaches 36.34%, just 0.14 points above the vanilla model, while RM@256 falls to 35.95%. The 7B RM@256 variant scores 35.79%. Those tightly grouped values are why the paper concludes that conventional Best-of-N scaling is limited here: increasing candidates and applying a reward model does not reliably locate better Olympiad solutions.

The domain and tier columns reveal where the aggregate comes from. o1-mini scores 67.82 in algebra but 37.68 in discrete mathematics; its T1 accuracy is 82.23 and its T4 accuracy is 42.69. The benchmark is not merely lowering every model by the same amount—harder tiers and discrete problems expose specific weaknesses that easier arithmetic suites average away.

Figures & tables

← Back to Mathematician