Can you tell which model wrote a paragraph? Not “is this AI-written” — that question is already hard to win. I'm asking something narrower and stranger: given a piece of text, can you name its model family and its exact variant? This started as a benchmark exercise. It turned into a lesson about data hygiene, uncertainty, and how easily one number convinces us.
The question
Most “AI text detection” only cares whether a passage was machine-generated. Attribution asks the harder question: which machine? The framing is useful because it forces you to separate a model's own voice from everything else in the pipeline — the prompt, the register, formatting habits, and whatever post-processing the vendor does before the text reaches you.
If a classifier can learn that separation, it becomes a small, testable provenance tool. If it can't, the failure is usually more interesting than the accuracy. You find out what your labels actually contain.
Setup
The work uses an open text pool covering 18 model variants. Each training round uses 45,000 texts — 2,500 per label — and is scored on 2,699 real competition texts whose prompts never appeared in training. The classifier is deliberately small: a Qwen3-1.7B base with LoRA. Two random seeds per condition. The goal is not to win a leaderboard but to find out which interventions actually change the result.
“Small” is the feature. A 1.7B model is cheap enough to rerun, and its mistakes are legible. When a large model fails, the reason may be buried in capacity; when a small one fails, you can usually point back at the data.
Clean the room first
The first intervention wasn't architecture, it was housekeeping: remove obvious junk and off-topic text from the training pool while keeping labels balanced and the eval set fixed. The raw pool scores 0.466 balanced accuracy; after cleaning it scores 0.493. Same model, same data volume, same seeds, a 0.026 gain.
That is not a dramatic number, but it is useful. It says the classifier was spending a good part of its capacity on “this text came from this dataset” versus “this text looks like task text”, not on telling model voices apart. Cleaning didn't add information. It removed a competing signal.
Better data didn't make the model smarter. It made the question easier to hear.
If your attribution model learned the corpus instead of the model, more parameters only make it wrong with more confidence. Fix the corpus first.
The second seed tells the same story: 0.464 raw, 0.492 cleaned. The numbers move, the direction doesn't. That consistency matters more than the headline gain, because a single seed can make any intervention look like progress.
Error bars are results too
The experiment scores three ways: all prompts, half the prompts held out of fitting, and two answers per prompt to get closer to the test distribution. The intervals are not decoration. Several ideas that looked promising on the point estimate collapsed once the range was drawn. One feature improved on the first seed but didn't reproduce on the second, so it never made the final file.
This is the least exciting and most trustworthy part of applied research. A result that survives prompt-level resampling and a result that only holds on one random seed are two different things. You can build on the first. The second is a rumor with a decimal point.
The 3.7 problem
The most instructive failure came from two variants that should be different: Gemini 3.7 and 3.8, from the same source. A classifier trained only to separate them scores 0.498 AUC — essentially random. There is no stable signal in the text. On real mixed-source text the same comparison reaches 0.582: better, but still not a fingerprint you can draw provenance conclusions from.
Here attribution stops being a trick. A model's “voice” is not a stable property of its weights. It is the residue of training data, decoding settings, system prompt, and all the normalization the vendor does before you see the text. If two variants share one pipeline, the difference you want to detect may not exist in the output at all.
The reverse is also true. Some variants separate easily because their formatting and refusal style are distinctive. That isn't necessarily a deeper understanding of the model — it may just be a shortcut that vanishes when the vendor changes one template.
What changed
After this, my defaults for eval work changed:
- Split by prompt, not just by row. One prompt can generate texts that leak into each other. If the split doesn't respect the generation unit, the score is inflated.
- Report random seeds. One run is an anecdote, two are a direction, more are a result.
- Audit labels, not just models. A label is the publisher's claim; some are clean, some are marketing. A classifier learns that difference faithfully.
- Prefer boring baselines. A small model with a clean split is more of an instrument than a large one with a leaked split.
- Keep failed interventions. A feature that didn't reproduce is part of the result. Delete it and the research becomes a highlight reel.
Limits
This is a small study on a public data pool, not a conclusion about all LLMs. The 1.7B classifier is a probe, not a production detector. Labels are only as good as the source. Attribution is adversarial by nature: vendors change serving stacks, prompts change behavior, and a model that looks distinctive today can be indistinguishable after one template update.
What generalizes is the method: isolate the signal, measure uncertainty, and treat a clean negative result as information too. Models can be wrong; evals shouldn't be.
Method / Qwen3-1.7B + LoRA, 45,000 texts per round, 2,500 per label, two random seeds, 2,699 held-out real texts. The chart shows balanced accuracy; intervals reflect prompt-level resampling. The public pool and the exported chart are the source of the numbers above. This is a field report, not a peer-reviewed paper.