Substack's AI Detector Isn't Broken. That's the Part Nobody Wants to Talk About.
It scored my last post at 100% AI, and it wasn't exactly lying.
The button is right there under the three dots. Scan for AI text. So I clicked it on my own post, roughly four hours after publishing it, the way you check your fly in a bathroom mirror after you’ve already given the talk.
100% AI.
I was mortified. Not because the number was wrong, exactly, but because of what it implies about me — that I’m out here breeding pure-bred puppies on the internet, clicking PUBLISH, then rolling around in cash I very much do not make. Which is, you know. A little more nuanced than that.
I’ve already established that I write with AI. That narrative is set. I’ve been refining what it means ever since the Spring 2026 Slop Incident™, which I am still not entirely over.1
So let’s talk about what that number is and isn’t.
What Substack actually shipped
On July 21, Substack turned on AI detection through a partnership with Pangram. Readers can scan posts, Notes, replies, and comments longer than 100 words, published on or after that date. Web and iOS now, Android later. The scan returns an estimate of how much of the text was human-written versus AI-assisted versus AI-generated.
You get some controls. There’s a “How I make this” statement you can write once, in Settings, that appears whenever someone scans you. You can disable detection on individual posts, which replaces your score with the words “AI detection unavailable” — a phrase that reads exactly like a lawyer telling you his client will be invoking the Fifth at this time. You can also report an error.
The announcement post took a swing at LinkedIn while coining a word: Claudefishing, for the con of investing your attention in text nobody meant.
Great word. Kinda shady as well. More about that in a bit.
The uncomfortable part: it mostly works
Here’s the deal: the standard move right now is to dunk on AI detectors, and a couple of years ago there was a lot of ammunition to do so. ZeroGPT flagged the U.S. Constitution. OpenAI killed its own detector for being bad at its one job.
And then the AI was all: “um, actually… that was three years ago. Detection really did get better.” Kind of. Let me explain:
Pangram got tested independently by researchers at Chicago Booth and came back near-perfect on 500-to-1,000-word samples, with a false positive rate the company puts around 1 in 10,000. Even critics who spent years telling people not to trust detectors have gone back and said, out loud, that this particular one is accurate on fully machine-generated text.
Well, damn. Okay then, I’m not going to tell you the detector is broken. Some of you were hoping I would.
But a caveat: Pangram’s own documentation recommends against scanning short text, bullet lists, outlines, and formulaic writing — and Substack won’t scan anything under 100 words, which is sitting right on the edge of that. It’s actually harder to prove that a person wrote something than to catch writing made by a machine, and the tools get it wrong much more often than the main numbers suggest, almost 1 in 70. Meanwhile the tool flagged a speech by the Pope at 46% AI, and a horror novel got pulled from a major publisher in March off the back of a scan.
With this scan, there are three buckets: human, AI-assisted, and AI-generated. Every working writer I know now uses AI to assist their writing, and it just so happens that the middle bucket is where Pangram’s AI confidence starts to die.
What the number can’t see
Here’s my actual workflow as abridged as humanly possible:
I journal daily.
At the end of the week, AI reads it and pulls out “story seeds” — a hook, a few lessons buried in there, questions for me to answer, and a pitch for why it could be funny.
I pick one, add notes, and have it draft in “my voice,” which it learned from a batch of my pre-2021 pieces.
Then I go line by line. Lately Claude has been writing in this weird essay-prose, stuff I would never EVER say, so it turns into a full rewrite. Sometimes with Grammarly, which is also AI, which is its own can of worms.
My rule is I keep going until every line sounds like it could come from me and I can co-sign all of it. Where I’ve screwed up before is when ADHD makes this part boring, or I’m rushed. Lessons are being learned. Keep me honest.
Then an AI skill for a title that’s SEO-friendly, and a prompt for OpenAI Images v2 in the brand art style, which I thoroughly enjoy doing.
All of it goes out the window for a different register. The eulogy for my dad — none of that was AI. But not every post is going to be a eulogy for my father. Cuz God, that would be intense, am I right?
A scan can’t tell you who decided what the piece was about. It can’t tell you whether anybody meant it. It can’t tell you whether the writer used AI to think or to avoid thinking, which is the only distinction I actually care about, and which has no technical answer at all.
Substack said as much in their own post, to their credit. The number can’t tell you if real human care went into a piece. But they shipped the number anyway, and put a text box next to it.
Why they did it
TLDR: Substack staff are kowtowing to anti-AI witch-hunters that may also be funders.
My slightly longer hot take, which I stand by 100%: somebody with a nice literary or political newsletter knows other people with newsletters, and they’re all low-key accusing each other of writing slop. Maybe their friend saw this slop and declared they’d never touch Substack again. Perhaps that friend was concerned, made noise, and the concern traveled up to somebody who could authorize a feature. Substack is still raising money to GeT BiGGeR. “We’re the platform that doesn’t tolerate slop” is a good line in a room like that. Coining a word for the enemy is what you do when you’re doing brand work, not when you’re quietly fixing a problem.
The more generous read is also true, and it’s structural.
Substack’s product is specific people’s minds, sold by subscription. Every other platform can survive being flooded, because their product is attention and attention doesn’t care where it came from. Here, if you can’t tell whether the opinion you paid for belongs to a person or AI, the subscription stops making sense. Slop voids the contract.
Detection is a defense of the pricing model. Fair enough.
But here’s the thing about a reader-initiated scan: almost no reader is going to run one. You have to already be suspicious, and once you’re suspicious you’ve mostly already decided to stop reading. Which means the audience for this feature isn’t readers.
It’s us. It’s every writer now aware there’s a number attached to their name, adjusting accordingly. And the 100% human-labelled ones—bless their perfect, English-native, time-having, literary hearts—get to be all twee and smug about it compared to the 0% human-labelled.
The box
I went into Settings and found the field for the “How I make this” statement, and I have been sitting with it ever since, which is embarrassing given I’ve been publishing on the internet since 1999 and this is a form.
The box wants a paragraph. The paragraph is supposed to explain a relationship I’ve spent three years failing to explain in longer formats, to a stranger who is already holding a percentage sign and squinting.
I don’t know how long to make it. Too short reads evasive. Too long reads guilty. There’s a version where I just paste “AI is the Adderall of the writing world” in there and let people sort themselves out.
Will Pangram and this implementation be useful? I have my doubts. I’m happy to be proven wrong, and I’d rather have the box than not have it, and I still haven’t typed anything in it.
But consider this a working draft:
I use AI. Substack will sometimes say 100% AI, 0% Human, and who am I to argue with a stupid counter? But know this: nothing gets published that I haven’t read out loud, nothing gets published that I don’t believe, and nothing gets published with a sentence I’d be too embarrassed to say myself. AI Slop is a real thing — I’ve read plenty of it. If this is that, tell me.
The Spring 2026 Slop Incident: several friends — yes, several, yes, still friends, yes, still loved — took turns “anonymously” ripping me a new one on social during a period of great stress. Bro. Just email me. See? Now I’m doing it too. Does anyone feel good? No? Okay then. Hit me up, it’s been forever. <3
Ahh, the classic sensitivity/specificity problem. It's very difficult to get a binary test to detect what you want it to detect. You have to decide. Do you want to affirm 100% human posts as often as possible? Or do you want catch AI as much as possible? Either way you are going to get false positives (or false negatives).