This is not an evaluation about making a model score better. It answers a plainer question: when an agent runs an engineering task, where does the money go and where do the points go.

The short version: capping the per-task budget at a third of the original cost nothing in score; rewriting the project documents into a form the model can actually read raised the partial score by a fifth; and every change that asked the model to check its own work did nothing at all.

85tasks
7dead ends
1/3the cost
¥0.49per task

What the competition tests

Physical-Engineering's Last Exam (PLE) comes from Google Cloud AI Research with Mercor: 85 expert-reviewed long-horizon tasks across structural, civil, electrical and mechanical engineering. Twenty-nine are a public development set; 56 are hidden.

Each task is a real project folder inside an isolated container, up to fifty files: multi-sheet PDF drawings, IFC4 and DXF models, load and wire schedules, geotechnical boring logs, code excerpts, and local MCP services for mail, calendar, documents and CAD. Six hours per task.

What you submit is not a predictions file but a repository: submission.yaml declaring the model and the start command, serve.py as a WebSocket server, and artifacts/ from the 29 public tasks. Stage one is a writeup under 2,000 words scored out of 100 — insight and failure analysis 25, engineering reasoning 25, cross-model generalisability 15, efficiency and scaling 20, clarity 15. The top hundred go to stage two on the hidden set.

Four hundred thousand dollars across three prizes. Forty-eight teams when I wrote this.

The shape of it

The harness runs in its own container; the task runs in a second one, offline except for the model endpoint. The protocol is a stream of JSON frames: exec a command, read_file, write_file, http into a local MCP server, and done to finish.

So everything the model does is a request the harness sends on its behalf for the evaluator to execute. No back door, no touching the container directly, no quiet network call.

Grading comes in three kinds: checks is a deterministic Python grader reading the deliverable and running assertions; rubric is an LLM judge against criteria; checks-judge is both.

The hard part is the score structure. Take the structural task I measured most:

Score structure

reward = 0.4 × partial + 0.6 × [all hard gates pass] × objective. Twenty-one hard gates. Any one of them failing erases sixty per cent of the score.

My first run scored 0.223: partial 0.559, which looks reasonable, but all twenty-one gates failed. Not "slightly worse" — all at once or nothing.

Measure the bill before the score

The official custom-openai sample is a complete working harness. I changed none of its structure, ran one task, and wrote down three numbers:

MeasureValueNote
Wall clock2,016 s33.6 minutes, no budget cap
Input tokens22.75 M99.5% cache hits
Output tokens218 k—
Cost¥1.55DeepSeek Flash, off-peak
Score0.257partial 0.559, no gate passed

Twenty-two million input tokens for a deliverable of five small files. Where does it go? Reading drawings, re-reading, verifying, re-reading again. The model does not remember; every step resends the whole history.

The cheapest knob is the budget

One line changed: at start-up the harness clamps the session deadline to HARNESS_BUDGET_SEC. When time is up it forces a close, and the working tree is graded as it stands.

BudgetWall clockInput tokensCostScore
Uncapped2,016 s22.75 M¥1.550.257
20 min398 s2.79 M¥0.560.259
10 min146 s0.42 M¥0.140.227
25 min708 s6.0 M¥0.780.284

A twentyfold token difference, and the score moves by ±0.03 — while the single-task standard deviation is already ±0.05. Those extra tokens bought almost nothing.

One lesson

The budget is not "time to think harder". It is "no time to repeat". Forcing a close makes the deliverable more complete, because there is no budget left for a third pass over the same drawing.

The score lives in the input

With the budget down, the remaining failures showed what was really wrong. Not arithmetic — wrong numbers. The design scheduled seven wall piers where the as-built log recorded five; the cost came out at ¥429k against a ¥395k cap; the seismic weight was 3,449 kip against a reference 2,171.

The model could calculate. It simply had not read those numbers, and had assumed a typical value instead. The fix is not a better prompt; it is translating the project documents into something the model can use directly.

So: digest.py. At task start it scans the working directory and writes one structured summary.

  • Tables: small schedules in full. An erection log has fifteen rows; showing six is the same as showing none.
  • Drawings: DXF entity census, layers, blocks, and every label grouped by layer — panel marks, opening marks and sizes.
  • PDFs: the first fourteen pages, plus a "rule line" index of sentences containing shall, equals, inequalities or units.
  • Point clouds: bounding box, z percentiles, field layout.
  • Conflict scan: every line containing superseded, differ, accepted by RFI, erected position, record set — quoted verbatim. This is the "which document governs" question.
  • Opening inference: classify by mark prefix (D door, P penetration, L window, EY equipment) and state that a window run is still a pier, as a spandrel.

Each layer moved the partial score and the gate count:

VersionPartialFailing gatesTotal
No digest0.559110.223
+ input digest0.618110.247
+ conflict scan0.70660.282
+ opening inference0.693—0.277
+ clause lookup0.70650.284

The largest single jump was the conflict scan. The erection log said, in a surveyor's note: "erected positions differ from PC-101 Rev 2, accepted by RFI-041". The drawing is not where the wall is; the record set is. No model will compare seven files looking for the sentence that says "I am out of date".

Turn the code into searchable evidence

The second tool is lookup.py: paragraph-level BM25 over every PDF, DOCX and Markdown file in the working directory, cached locally. One line to use:

python3 _harness/lookup.py "parapet tributary height anchorage"

It returns the passage, the file and the page. Add --examples and it prefers worked calculations. No model calls, no cost, no blowing up the context.

Three real hits, each the answer to a failing gate:

  • SDC-02 §12.11.2.2.1: the parapet's cantilever lever arm for anchorage force.
  • FEMA P-2192 Eq 12.10-1: the Fpx formula with its minimum and maximum.
  • DAP-2 stage A: the wall segment flexibility fuu = h³/(3EI) + h/(GAv), with cracked inertia at 0.5 Ig.

Failing gates went from six to five. What remains is "agree with the reference calculation item by item" — beyond what retrieval can do.

Seven dead ends

This is the section I actually wanted to write. Each of the seven changes below looked, before I built it, like it should help.

AttemptMechanismResult
Generic self-checkAudit the deliverable against the task before finishing0.224 (−0.03)
Numeric cross-checkRecompute every graded number down a second pathFlat; +0.016 on another task at three times the cost
Contract checkDeterministic comparison of template JSON keys, sheets, layers0.210
Fresh-context criticA clean context, without the reasoning, lists gapsInside the noise
Milestone enforcerHalf the budget gone and nothing written: force a crude packageNever fired — the file existed, the content was wrong
Output compressionClip tool results from 30k to 8k characters to save contextTokens doubled: the model re-read the files
Forced method extractionPull the numbered method out of the protocol, then follow it0.197 (−0.09)

Read together, there is one cause: a model cannot fix an error it does not understand.

Verification after the fact only works when the model already knows the right answer. When it does not, asking it to check its work replays the same faulty reasoning and doubles the bill. The numeric audit is the clearest case: it wrote a genuine second derivation, but the premises were still wrong, so both numbers were wrong in the same way and agreed with each other.

Compression is the counter-intuitive one. Withholding context was meant to save money, and it doubled the token count, because a model that cannot see the full output goes back and reads the same file again.

Under coupled constraints, it does not converge

The most useful finding came from the hardest task: rebuild a 27 × 21 m data hall from seven PNG sheets, then route three overhead services — cable tray, busway, chilled water — with zero hard clashes, each service a single connected network reaching its entry, every rack row covered to 80% by all three, chilled water within 2.5 m of every CRAH, plus headroom, sprinkler and PDU-clearance rules.

I wrote a geometry checker: world-coordinate bounding boxes through ifcopenshell, 3D overlap beyond 1 mm as a hard clash, connected components per service, per-row coverage, nearest chilled-water distance per CRAH. Then the harness ran it automatically whenever the model tried to finish, and handed back the report.

FeedbackClashesWorst coverageClash gateConnectivity gate
None10.55failfail
Model checks itself ×300.55passfail
Automatic check, one round71.00failfail (network fully connected)
Multiple rounds20.55failfail

It oscillates. Show it the clashes and it drives them to zero, and coverage falls back to 0.55. Show it the coverage and it lays all three services across every row with the networks fully joined, and seven clashes appear.

The mechanism

One metric can be optimised; several coupled ones do not converge. This is why the benchmark's mean score is so low: the task is not "a bit short of right", it has to be right all at once.

After two rounds

A single run is not evidence here — across seven runs of the same task, six scored zero and one scored 0.56. So the last measurement was two full rounds over five tasks:

TaskGradingRound ARound BMeanCost
Landscape sheet revisionrubric0.8960.5710.734¥0.31
Fire test memorubric0.7020.6300.666¥0.28
Seismic diaphragm shearchecks0.2920.2820.287¥0.50
EOR review correctionschecks+judge0.1760.2410.209¥0.75
MEP routingchecks0.0000.0000.000¥0.62
Set—0.4130.3450.379¥0.49

At that rate the 29 public tasks cost about ¥14 and three to four hours serially. Which means the budget has stopped being the constraint; the constraint is now the score itself.

Three caveats. First, against the official sample the score barely moved: 0.257 to 0.287 on the structural task, 0.271 to 0.209 on the review task, both inside the noise. What did move reliably is cost, down about threefold. Second, only two of the five tasks were ever run against a true baseline, which is the largest methodological hole in this work. Third, the score is almost entirely determined by the kind of deliverable: rubric tasks that ask for a document sit at 0.67–0.73, checks tasks that demand exact numbers at 0.00–0.29.

Conclusion

Three months of changes reduce to three sentences.

Measure the bill first. A hard deadline that forces a close is the only knob that is both cheap and free of score cost. It does not buy more thinking; it buys an end to repetition.

Fix the input. Lay out schedules, layers, labels, conflict lines and rule lines in the units the model can use. That is worth more than any prompt. When the score stalls, suspect what the model is looking at before suspecting how smart it is.

Do not rely on after-the-fact verification. Self-checks, recomputation, critics and contract checks all fail once the model genuinely does not understand the task. Measure the gates, not the prose.

Method / Numbers come from 7 of the 29 public development tasks across two full rounds; per-task cost uses DeepSeek V4.1 Flash off-peak pricing ($0.003/M cached input, $0.15/M uncached, $0.60/M output) at 7.1 RMB to the dollar. Four tasks could not be run locally because their images need network access to build or the dataset download was incomplete. The submitted artifacts/ were generated with DeepSeek while the declared model must be a google-cloud/* one; that is disclosed in the writeup and remains the largest open risk in the entry. No rank is claimed.