Professional roles we evaluate.
Each role groups papers and benchmarks around concrete tasks in programming, mathematics, machine learning, and audio/video production.
Programmer
Repository-level implementation, code editing, review, testing, and debugging.
A programmer lives inside the repository and the toolchain: requirements arrive through issues and tickets, context flows through Slack threads and design docs, code moves through GitHub — branches, pull requests, reviews — and nothing ships until CI is green. The job demands reading a requirement and turning it into working code, navigating and editing a living codebase without breaking it, writing the tests that prove the change, and debugging until everything passes.
Requirement understanding · Coding & code review · Testing & debugging
NL2Repo-Bench
Complete Python libraries generated from one requirements document, with correctness decided by the original hidden test suite.
CodeSimpleQA
Bilingual factual programming questions, scored separately from executable code generation.
CodeEditorBench
Execution-checked edits across debugging, translation, optimization, and changing requirements.
OpenCoder
OpenCoder is a code-language-model and reproducibility paper, not a benchmark.
OpenCodeInterpreter
OpenCodeInterpreter is a family of open code-generation systems trained to revise programs after compiler output and user feedback, rather than a new benchmark.
Mathematician
Problem solving, proof construction, formalization, and proof checking.
A mathematician works between the literature and the blackboard — reading papers, writing LaTeX, and increasingly working inside proof assistants like Lean, where every step must satisfy the checker. The job demands stating a problem precisely, constructing a proof argument step by step, verifying each claim rather than trusting intuition, and reviewing others' work critically — from competition problems to formalized theorems.
Problem formalization · Proof construction · Critical review
FormalMATH
A 5,560-statement Lean 4 test across Olympiad and undergraduate mathematics.
Omni-MATH
A 4,428-problem Olympiad set with domain, difficulty, and judge-reliability analysis.
CriticLean
CriticLean is a critic-guided training framework for checking whether a Lean 4 formalization preserves the meaning of a natural-language mathematics problem.
Machine Learning Engineer
Data preparation, model training, validation, and pipeline debugging.
A machine learning engineer moves between notebooks, data warehouses, and GPU clusters: pulling and cleaning data, tracking experiments, scheduling training runs, and watching validation curves — with the pipeline versioned in Git like any other code. The job demands understanding the problem behind the data, engineering features, training and validating models, debugging pipelines when metrics collapse, and iterating over long horizons until the submission actually scores.
Data understanding · Training & validation · Long-horizon iteration
Audio & Video Editor
Video and audio editing, synchronization, and delivery quality control.
An editor works inside professional production suites — non-linear editors, digital audio workstations, asset libraries, and review tools — turning raw footage and audio into a finished cut. The job demands reading a creative brief, cutting to structure and pacing, locking sound and image together, driving long multi-step tool workflows without losing the thread, and quality-checking the deliverable before it ships.
Brief understanding · Cutting & sound-image sync · Delivery QA
AutoMV
AutoMV is a training-free multi-agent system for turning a complete song into a shot-based music video; the paper is principally a generation-system study, with a 30-song evaluation suite used to test it.
Workflow-GYM
Specialized desktop work scored at the final artifact or GUI state after 30 to 110 dependent actions.