Living off the AI: why content-only phishing detection fails.

Every SOC briefing this quarter seems to include a slide on "AI-generated phishing detection." The pitch is uniform: large-language-model output has statistical tells, the vendor's classifier learns them, and your inbound rate of LLM-generated lures drops toward zero. It is a reassuring story. It is also, on inspection, a brittle one, because it rests entirely on the content of the message, and content is the one thing an attacker fully controls.

This piece is analysis, not a lab report. It walks through how content-only classifiers actually decide, why three cheap rewriting techniques defeat that decision, and the one class of signal that does not depend on the words at all.

1. How content-only classifiers decide

Text classifiers that flag "AI-written" messages are, at their core, distribution detectors. They tend to key on the statistical fingerprint of a given model's output, token-frequency regularities, predictable n-gram transitions, low perplexity, a narrow band of sentence rhythms, and score how closely a new message matches that fingerprint. This works reasonably well against default-configured model output, because default output is remarkably self-similar.

The weakness is structural: the classifier is keyed to a distribution, not to intent. Anything that moves the message off that distribution, without changing what it asks the victim to do, moves it out of detection range.

2. Three cheap ways the distribution moves

None of the following require novel research or a capable adversary. They are configuration changes and one-line prompts, and each attacks the same assumption from a different angle.

• Raise sampling temperature and widen top-p. Default generations are low-variance, which is exactly what makes them detectable. Turning up temperature and top-p broadens the token distribution and breaks the tidy n-gram regularities a classifier is trained to recognise, at little cost to readability.
• Back-translate through a low-resource language. Running the text EN → a low-resource language → EN passes it through a different model's priors and reshuffles the per-token distribution, while the semantic payload, the ask, the link, the urgency, survives intact.
• Style-transfer with a one-line prompt. "Rewrite this in the voice of a 1998 Slashdot user", or a support agent, or a named colleague, relocates the message into a register the detector has little signal for. The instruction is trivial; the distributional shift is large.

The common thread is that all three are content transforms an attacker applies for free, before the message is ever sent. A detector that only reads content is, by construction, one rewrite behind, and against transformed generations you should expect catch rates to collapse rather than degrade gracefully.

3. The signal that does not move: behaviour

Content is attacker-controlled; behaviour is not. The signals that survive a rewrite are the ones an attacker cannot edit from inside the message body:

• Response latency, the time between a message landing and the victim acting on it. Campaign traffic tends to cluster in ways ordinary correspondence does not.
• Device and client, the device and user-agent used to act on the message, and whether it matches the recipient's norm.
• IP-class of the interaction, hosting, VPN, or residential-proxy space around the click and any follow-on authentication.
• User-reported signal, a recipient hitting "report phishing" is a ground-truth label that no amount of prompt engineering can rewrite.

These live in telemetry you already collect, mail-gateway logs, endpoint and identity events, the report button, not in the prose of the email. They degrade slowly, if at all, when the lure is reworded, because they describe the campaign's mechanics rather than its wording.

4. So what?

If a detection strategy is content-only, it is one prompt-engineering revision behind, permanently. That does not make text classifiers useless, they raise the cost of the laziest campaigns, but they should be treated as one weak input to a behavioural model, never as the control. The telemetry that actually holds up is behaviour-side, and most of it is already sitting in your logs.

The most effective phishing defence in 2026 is still the user-reported-phishing button and a 24-hour SLA on the response.

Curious how your current stack holds up when the same lure is rewritten three cheap ways? That is exactly the kind of resilience OverWatch Labs can test against your controls, before an attacker does.