Last time I built a detector that counts four-letter runs (n-grams) and tells you whether Claude Opus 5 wrote something. It works well - about 91% on text it has never read.
It has one obvious hole: it only knows one robot (Claude Opus 5). Paste ChatGPT output into it and it says “meatbag”.
So I’ve fixed that. I’ve added a ChatGPT detector and you can try it here.
Getting the ChatGPT data
We did the same as last time, have the model rewrite the same 519 human passages, three ways. That’s 1,557 rewrites, plus 519 passages written from scratch that I keep out of training and use as a test. 2,076 generations in total.
Last time this went through Anthropic’s batch API and cost about $30. This time I just ran the command line codex using the subscription I’m already paying. So it didn’t cost anything extra (I’m wishing I’d done that with claude last time!),
There is a small issue with this - coodex is a coding agent, so it will have a system prompt that will influence the output, so our trained up detector might not do as well on text from the ChatGPT app.
The ChatGPT detector
We built this in exactly the same was as the Claude detector - and it works really well!
| Detector | Cross-validated | Held-out authors |
|---|---|---|
| Claude Opus 5 | 0.906 | 0.925 |
| ChatGPT | 0.930 | 0.935 |
ChatGPT seems to be easier to detect than Claude. Not by a lot, but consistently.
It also seems to handle shorter text better than the Claude detector.
| Letters | Claude detector | ChatGPT detector |
|---|---|---|
| 100 | 0.645 | 0.709 |
| 200 | 0.740 | 0.798 |
| 432 | 0.783 | 0.885 |
| 1,000 | 0.858 | 0.952 |
At a thousand letters the ChatGPT detector is on 0.952 against 0.858 for the Claude one, and it’s ahead on short text as well.
Do the two detectors work on each other?
| Detector | Reading its own model | Reading the other one |
|---|---|---|
| Claude | 0.906 | 0.792 |
| ChatGPT | 0.930 | 0.800 |
Each one is clearly best on the model it was trained on, but neither is terrible on the other - 0.79 and 0.80, against 0.5 for a coin toss.
A third detector
Can we build a detector that does Claude or ChatGPT?
We trained s third detector from scratch on all the data from both models.
| Detector | ROC AUC | Held-out authors |
|---|---|---|
| Was this a robot? | 0.897 | 0.911 |
| Does it look like Claude? | 0.906 | 0.925 |
| Does it look like ChatGPT? | 0.930 | 0.935 |
They’re three separate questions rather than slices of one pie, so they won’t add up to anything. A passage can read as 0.85 robot while both model scores sit lower, which just means it looks machine written without strongly resembling either one.
What it still can’t do
Everything from last time, plus a couple of new ones.
It knows two robots. Gemini, Llama, DeepSeek, anything else - it has never seen them. A low score is not evidence that a person wrote something.
It still detects a style rather than a model. The most useful result from last time hasn’t changed: a writing style held out of training scored 0.153, meaning the detector confidently rated it as more human than the humans. Both new detectors train on all three instruction styles for that reason, but there’s nothing magic about three - we need to add more data.
It’s still for education and amusement, and it’s still not evidence. Please continue not accusing anyone of anything based on my stupid website.
Try it out
meatbag.atomic14.com - paste something in and see which robot it thinks you are.