Evaluating local models has a recurring dilemma: you want a number, but no number is very trustworthy. With ten questions the noise is bigger than the difference; with a thousand questions, where the bank came from, who grades the subjective items and whether the judge has its own preferences are each a trap.

LMExamBench is my answer to that dilemma. Its shape is not “run a benchmark” but “organize an exam”: there are stages, subjects, weights and grading rules, and an explicit “not enough samples this time”.

Why an exam

Because the exam format has already solved some of the problems we are re-inventing:

  • Layering: primary school, middle school, high school, university and master's, plus the software engineer exam, big-tech interviews and LeetCode — eight stages. Question difficulty and grading differ by stage by design; averaging them together is meaningless.
  • Weights: a job profile (profiles.json) decides which stages a “Java backend” candidate sits, how much each is worth, and how capability dimensions are weighted. Same question bank, different profile, different exam.
  • Subjects: data structures, algorithms, operating systems, networks, databases — a score has to break down by subject before you can see which part is weak.

The question bank itself is not in the repo (copyright); what is in the repo is a skill for generating an original bank with AI (skills/generate-bank) plus a schema. The workstation currently holds 617 questions, between 57 and 106 per stage across the eight stages.

Paper mode

The most easily overlooked feature, and possibly the most important: --paper N. It assembles a paper by drawing N questions from each “stage × subject/profile” section, rather than drawing N questions at random.

The difference is large. Random sampling produces papers like “25 primary-school questions + 1 master's question”, and then you use it to compare two models — what you measured is who is better at primary school. Per-section sampling guarantees every stage is represented, and it also tells you honestly: a section with fewer than 2 questions is flagged ⚠ insufficient sample, do not conclude.

Three grading tracks

Question typeGradingDetail
ObjectiveRule matchingNumeric tolerance; multi-select scored partially as (hits − wrong picks) / correct count; fill-in accepts multiple answers
CodeSubprocess runs the casesPass rate per sample, pass@k for multiple samples; hidden cases supported
SubjectiveLLM judge0–1 score per rubric point; with debias, graded in forward and reverse order and averaged

Code questions have to actually run. This is the rule I hold to hardest: have a model write code and then eyeball whether it “looks right”, and evaluation becomes aesthetics. Extract the code block, wrap it in a harness, run the unit tests — the answer is pass or fail, blunt but not arguable.

The judge must be swappable

The judge for subjective questions is an LLM, which introduces a new problem: the judge has preferences. I did two things.

First, the model under test and the judge are fully decoupled: two independent base_url / api_key / model. A local 7B can judge a local 27B, or the other way round. Whether a judge runs at all is one line of config (judge.enabled); turn it off and only the objective questions run.

Second, debias: grade the same answer once in forward order and once in reverse, then average. It cannot remove preference, but it weakens position bias — the tendency to score the first candidate higher, the most common and most reproducible failure mode for an LLM judge.

One small but critical detail: run records scrub keys automatically. The thing an evaluation tool leaks most easily is not the model, it is the config.

67 and 84

Two runs on the workstation, same model (deepseek-v4.1-flash, local 8080), 1 question per stage for 8 total, judge off (objective and code questions only):

RunConfig differenceWeighted scoreGradeCoverageElapsed
Firstmax_tokens 204867.39D · pass75%215 s
Secondmax_tokens 4096 · reasoning_effort=low83.7B · good87.5%201 s

Same model, a 16-point spread. Looking only at the scores, the conclusion would be “the second configuration is better”. The real reason is different: both runs drew only 8 questions, covering 75% and 87.5% — some stages got no question at all, or only one.

The framework got one thing right here: it puts coverage and ungraded_n next to the total, and flags sections with fewer than 2 questions as an insufficient sample. An 8-question exam can produce both 67 and 84, which means the score describes luck, not ability.

The first duty of an evaluation is not to produce a score, but to state the uncertainty of that score.

In hindsight those two runs were not wasted: they exercised the whole chain (select → call → grade → aggregate → report), and they confirmed that the insufficient-sample warning actually fires. They just should not be treated as conclusions.

Re-gradable

Evaluation tools tend to become one-off scripts: run, export, never open again. So two features are treated as first-class:

  • regrade: re-grade stored answers under the current grading rules. Change a rubric, add a tolerance, fix a bug — no need to call the model again. That matters especially for local models, where a full run can take tens of minutes.
  • audit: question-bank audit, for near-duplicates and coverage. A bank keeps growing as the generation skill runs, and duplicate questions quietly make one subject heavier.

There is also compare: multi-run comparison with discrimination and difficulty statistics. A question that every model gets right carries no information for this evaluation — which is not the same as “the question is too easy”; the former is a statistical property.

What it cannot do

It does not decide whether a model is good, it answers “how many points it scored on this paper”. Change the bank, the weights or the judge, and the answer changes.

It also does not solve the legality of the question bank: the repo deliberately ships no bank, because the questions are copyrighted. The bank-generation skill is for people willing to write their own questions, not for people who want a standard answer key.

Finally, 8 questions is 8 questions. Paper mode makes one measurement more representative, but it cannot turn 8 questions into 80. When the sample is too small, the most professional output is “the sample is too small”.

Method / Code and docs at NorwinYu/LMExamBench; the two run records come from the local runs/ directory (2026-10-29, 8 questions each, judge off). The repo ships no question bank; the 617 questions here were generated locally and are not distributed with the repo. Scores are the framework's weighted percentage, not a leaderboard rank.