🌈 ESP32-S3 Rainbow: ZX Spectrum Emulator Board! Get it on Crowd Supply →
View All Posts
read
Want to keep up to date with the latest posts and videos? Subscribe to the newsletter
HELP SUPPORT MY WORK: If you're feeling flush then please stop by Patreon Or you can make a one off donation via ko-fi

Last time I built a detector that counts four-letter runs (n-grams) and tells you whether Claude Opus 5 wrote something. It works well - about 91% on text it has never read.

It has one obvious hole: it only knows one robot (Claude Opus 5). Paste ChatGPT output into it and it says “meatbag”.

So I’ve fixed that. I’ve added a ChatGPT detector and you can try it here.

Robot or Meatbag? The detector calling a passage "Written by a robot", with Claude Opus 5 at 1.00 and ChatGPT at 0.34 underneath

Getting the ChatGPT data

We did the same as last time, have the model rewrite the same 519 human passages, three ways. That’s 1,557 rewrites, plus 519 passages written from scratch that I keep out of training and use as a test. 2,076 generations in total.

Last time this went through Anthropic’s batch API and cost about $30. This time I just ran the command line codex using the subscription I’m already paying. So it didn’t cost anything extra (I’m wishing I’d done that with claude last time!),

There is a small issue with this - coodex is a coding agent, so it will have a system prompt that will influence the output, so our trained up detector might not do as well on text from the ChatGPT app.

The ChatGPT detector

We built this in exactly the same was as the Claude detector - and it works really well!

Detector Cross-validated Held-out authors
Claude Opus 5 0.906 0.925
ChatGPT 0.930 0.935

ChatGPT seems to be easier to detect than Claude. Not by a lot, but consistently.

It also seems to handle shorter text better than the Claude detector.

Letters Claude detector ChatGPT detector
100 0.645 0.709
200 0.740 0.798
432 0.783 0.885
1,000 0.858 0.952

At a thousand letters the ChatGPT detector is on 0.952 against 0.858 for the Claude one, and it’s ahead on short text as well.

Do the two detectors work on each other?

Detector Reading its own model Reading the other one
Claude 0.906 0.792
ChatGPT 0.930 0.800

Each one is clearly best on the model it was trained on, but neither is terrible on the other - 0.79 and 0.80, against 0.5 for a coin toss.

A third detector

Can we build a detector that does Claude or ChatGPT?

We trained s third detector from scratch on all the data from both models.

Detector ROC AUC Held-out authors
Was this a robot? 0.897 0.911
Does it look like Claude? 0.906 0.925
Does it look like ChatGPT? 0.930 0.935

They’re three separate questions rather than slices of one pie, so they won’t add up to anything. A passage can read as 0.85 robot while both model scores sit lower, which just means it looks machine written without strongly resembling either one.

What it still can’t do

Everything from last time, plus a couple of new ones.

It knows two robots. Gemini, Llama, DeepSeek, anything else - it has never seen them. A low score is not evidence that a person wrote something.

It still detects a style rather than a model. The most useful result from last time hasn’t changed: a writing style held out of training scored 0.153, meaning the detector confidently rated it as more human than the humans. Both new detectors train on all three instruction styles for that reason, but there’s nothing magic about three - we need to add more data.

It’s still for education and amusement, and it’s still not evidence. Please continue not accusing anyone of anything based on my stupid website.

Try it out

meatbag.atomic14.com - paste something in and see which robot it thinks you are.

Related Posts

It's Plausible, But Is It True? - Here's a wild ride with ChatGPT and other large language models! They're ace at cooking up plausible-sounding text, but they're not always the best when it comes to spitting out the truth - they've got a funky relationship with facts. One research paper showed they can come out with believable but totally fake answers to seemingly straightforward facts. But when I messed around with various models, there were a few discrepancies. Some got it right or plausibly wrong, but we humans are pretty gullible and tend to believe plausible-sounding info. So when it comes to using ChatGPT, make sure you fact-check, stay away from complex reasoning tasks, and don't try and solve maths problems - seriously, just stop. But it's a cracking tool for generating marketing copy, code (with a fact-check), finding bugs, and getting the creative juices flowing. Stay tuned though - tech's ever-evolving and these intelligent library computers aren't going away anytime soon!
Cocktail Bot Using Official ChatGPT API - In this post, I cover the migration of our cocktail bot to use the newly available ChatGPT API. Prompt engineering is performed to accurately guide the bot's versatility in cocktail recommendations. A Python application is provided via GitHub for testing the chatbot. To ensure appropriateness in user inputs, a moderation system from OpenAI’s API has also been integrated. However, readers are reminded to consume the AI's cocktail suggestions responsibly and remember that it's just a machine- its advice could potentially lead to less than satisfactory results.
Detecting Claude by counting letters - Anthropic now watermarks Claude, but I wanted a detector we can run ourselves—so I built one. It ignores words entirely, counts 4‑letter n‑grams, feeds them to a linear SVM, and reliably flags Claude Opus 5 (around 0.90+ ROC AUC, up to 0.977 on fresh writing), even on unseen authors and external benchmarks. It runs in your browser, and along the way it surfaces neat stylistic tells—think Germanic vs. Latinate vocabulary—so you can see why the call was “robot” or “meatbag.”
Why does ChatGPT make mistakes - a layman's explanation - In this enlightening blog post, I dive into the tantalizing world of ChatGPT and Large Language Models. Clarifying its operation, I unlock this enigma by comparing its mechanisms to a simple language model. However, Challenges arise due to the explosion of possible token combinations, leading to an inherent 'lossy' compression of our world's vast information. Surprisingly, even with such compression, these models can mimic human language in a compelling manner. I also investigate possible strategies to optimize this amazing technology - including zero-shot learning, one-shot learning, few-shot learning, and fine-tuning. Entering the era of prompt engineering and larger models, we're stepping into a thrilling future, so buckle up, folks!
I was wrong - we've not reached peak ChatGPT hype yet... - Strap yourselves in, folks, we're in for a wild ride! ChatGPT's new API has reignited my excitement for Large Language Models, just like the start of the dot com boom. With the pricing now 10 times cheaper, a flurry of creative and previously unthinkable use cases is within our grasp. Despite earlier doubts, I now believe we're scaling the peak of inflated expectations. Can't wait to see the innovative applications that will spring from this!

Related Videos

Forget the Arduino Serial Plotter - try this instead! - Hey everyone! So, you're probably familiar with the Arduino serial plotter, right? It's useful for basic stuff, but I wanted something better, more cross-platform, and runs right in the browser. Enter the web serial plotter, using the web serial API! It's super flexible with zoom, export options, and light/dark mode. You can even pan through history or switch to absolute time. Plus, it's all secure and runs locally. It's open source, and I'd love your contributions or bug reports on GitHub. It's still a work in progress, but I think you'll find it pretty handy!
AI Powered Raspberry Pi Home Automation - Is this the future? - Witness the power of ChatGPT controlling home automation lights through a Raspberry Pi, making life easier with plugins. Delve into the fascinating world of large language models, redefining interactions with APIs.
Unlocking the Power of ChatGPT: Effortlessly Generate Arduino Code for Your Projects! - Witness ChatGPT's impressive potential for generating working Arduino code, as demonstrated in a step-by-step ESP32-based project utilizing a potentiometer and dot star LED.
ChatGPT vs Stockfish: Can an AI Plugin Improve its Chess Game? - Watch as chat GPT takes on Stockfish, a world-class chess engine, in a thrilling match! See how GPT utilizes a chess plugin to improve its gameplay and compete against the best.
Surprisingly not CR*P - Vibing hardware - Well folks, what an adventure! After vibe coding our vibing button, I took up the challenge to vibe code the actual hardware for an ESP32 S3 dev board. Used a tool called at pile and gave it a whirl with some AI help from Claude. Interestingly, it handled everything from sourcing components with LCSC part numbers to wiring them up. Had a wee hiccup with missing capacitor in the RC circuit, but overall it was a thrilling ride watching the AI process! A bit of tweaking here and there, and we're good to go! Can't wait to see it in action.
HELP SUPPORT MY WORK: If you're feeling flush then please stop by Patreon Or you can make a one off donation via ko-fi
Want to keep up to date with the latest posts and videos? Subscribe to the newsletter
Blog Logo

Chris Greening


Published

> Image

atomic14

A collection of slightly mad projects, instructive/educational videos, and generally interesting stuff. Building projects around the Arduino and ESP32 platforms - we'll be exploring AI, Computer Vision, Audio, 3D Printing - it may get a bit eclectic...

View All Posts