The Case Against AI Slop Reports

The report that made me a hardliner about this was one of ours.

Early on, we generated a batch of thirteen sample strategy reports on a real beverage-alcohol brand to see what our pipeline could do. Buried in one of them, in confident consultant prose, was a fabricated fact about an eighteen-year-old — on a product category where age isn't a detail, it's a legal line. Nobody asked the system to make it up. Nothing in the brand's data said it. The model needed a fact to finish a sentence, so it manufactured one.

The review that caught it closed on a verdict I still think about: "the thinking is agency-grade; the artifact leaks the machine."

That's the whole genre in one sentence. So before I make the case against AI slop reports, let me put my cards down: I sell AI-generated strategy reports. Ours nearly shipped slop. The difference between the report that invents an eighteen-year-old and one you can put in front of a board is not the model, and it's not the prose. It's whether the system around the model is allowed to lie. That's machinery, and machinery can be inspected — even from the buyer's side of the table, before any money moves.

What the twenty-dollar report actually is

Strip the landing page off a cheap "AI brand audit" and here's the mechanism: your brand name goes into a prompt, and a general-purpose model writes what is statistically plausible for a company shaped like yours. No retrieval. No verification. No source it could cite if you asked. The output is fluent because fluency is the one thing the model is guaranteed to have.

The problem is that a model without grounding doesn't know your business — it knows the average of businesses like yours. We measure this constantly. In our sanctioned public demo, we asked ChatGPT, Claude, and Gemini who founded a small CPG brand: one invented a founder's name outright, one admitted it didn't know, one got it right. One question, three answers — a lie, a shrug, and the truth. Now imagine forty pages of that, formatted as strategy, with your logo on the cover.

Whether AI-assembled strategy is worth buying at all is its own argument for another day — assume you're already shopping. The question here is narrower and more useful: how do you tell, from the outside, whether the confident PDF in front of you was researched or hallucinated? The answer is to ask about the machinery. Here's what the machinery looks like when it's real — gate by gate, from the pipeline we run.

Research happens before a single sentence

A real report starts by not writing. Ours derives research questions from the engagement, fans out to researcher agents that retrieve sourced material from the live web, and — this is the part that matters — persists the full retrieval set verbatim before any writing begins. The sources aren't decoration gathered afterward to make the prose look supported. They're the raw material, on the record, timestamped, in a store the writing step cannot edit.

The wrapper inverts this: prose first, then maybe a reading list. If the sources were assembled after the sentences, they're a costume.

Prose isn't allowed to know anything the claims don't

The most important design decision in our pipeline is an inversion most people never see. The writer doesn't write prose and cite as it goes. It first emits a structured array of claims — each one required to cite a source that was actually retrieved — and only then writes prose from those claims. The code comment states the intent plainly: this "kills the prose-first / cite-later hallucination surface: prose can only restate claims, and a claim can only cite a source actually retrieved."

A claim that cites a source the system never fetched gets its citation dropped; a claim left with no valid source is rejected. The eighteen-year-old can't happen in this architecture — not because the model got smarter, but because a fact with no source has no path to the page.

Verification checks the claim, not the link

Here's a tell that separates serious pipelines from theater: what does "verified" mean? In most of the genre, it means the URLs resolve. Ours runs verification at the claim level — an entailment check of each claim against the actual text of the source it cites. The comment at the top of our verification step is the sharpest one-line summary of the whole discipline I know: it checks "entailment, not link-validation — a URL can be real while the specific claim is unsupported."

A bibliography proves the vendor can search. Entailment proves the sentence in front of you is actually in the evidence behind it. Claims that fail go back to the writer, not into the report.

The system is allowed to say nothing

This is the gate buyers should care about most, because it's the one that costs the vendor money.

When our writer produces a well-formed but empty claims set — the research genuinely found nothing — the section renders as nothing. The comment on that step calls it "the suppress-don't-fabricate posture: no sources, no claims, no prose." On regulated territory it's stricter: age, health, legal, and financial claims on regulated products are suppressed entirely unless sourced and verified, and every suppression is written to an append-only audit log. The system keeps a record of what it refused to say.

The refusal reflex runs all the way up. Our capstone report — the one that synthesizes every vertical — is structurally forbidden from inventing an overall grade: the code rules that it "never re-scores, averages across verticals, or invents an 'overall brand score' — inherited grades stay inherited, cited to their own methodologies." One number summarizing a brand would demo beautifully. It would also be fabricated certainty, so it doesn't exist. Same posture at the language level: when a retail read comes from a sampled commercial data feed rather than measured panel data, our rule is to never say "measured" or "market share" — the honest phrase is "graded from a licensed shelf snapshot, dated." Honest systems downgrade their own vocabulary. Slop never does; padding is its business model. A wrapper that finds nothing writes the section anyway, because its unit economics depend on always delivering the same page count.

Provenance is a feature you can read

Every report we ship closes with a Methodology and Sources section — not as a flex, but because a strategy claim you can't trace is a strategy claim you can't act on. Underneath the visible layer sits a claim ledger: a table joining every claim in the report to the persisted sources that support it, with the audit log recording what was retrieved, what was verified, what was suppressed. When a client says "that's not right," the correction path starts from the ledger, not from vibes.

And because independently generated reports can each be defensible alone yet contradict each other across a bundle — describing what reads like two different companies — we built a coherence pass that extracts claims across the finished set, detects where reports assert incompatible things, and adjudicates each contested point against the shared fact base before delivery. That pass runs on every bundle we deliver; switching it off is the deliberate exception, never the default.

We ran the detector on ourselves

Everything above would be marketing if we hadn't pointed it inward, so here's the uncomfortable receipts section.

This month we ran ten independent audit passes over our own report catalog — fresh-context reviewers, framed to refute, read against the strongest productized competitors in each vertical. The verdict, recorded in the program doc: the writing is largely at bar, and where we fall short "is almost never the writing — it's that the pipelines already compute or could cheaply fetch more truth than the rendered report shows the reader… It's a plumbing-and-inputs problem, which is the good kind — it's buildable, gate-able, and honest." An audit that comes back "your gap is wiring, not intelligence" is only useful if you believe it — which is why the reviewers were framed to attack, not to flatter.

Earlier this month, we did the harder version: audited our own marketing numbers for reproducibility. Several didn't survive. Competitive win-rates we'd been circulating turned out to be internal ablations — the competitor's live product had never actually been run. Customer improvement figures couldn't be traced to a source on disk. All of them are now retired from every surface, permanently, and the standing rules they left behind are blunt: no competitive claim unless the competitor's live product was actually invoked, and no "lift" claim until a brand has been re-measured after a change — which, as of today, none has. The claims deck we operate from now carries the line I'm proudest of: "We publish the tests we lost, too."

I'm not telling you this because we're saints. I'm telling you because a vendor's relationship with its own numbers is the best proxy you'll ever get for its reports' relationship with yours.

The part that should make you nervous

Hold the tension honestly: AI reports can be slop. Ours nearly was. The walls I've described are not free, and every one of them makes the product look worse in a demo. Verification loops multiply model calls. Suppression produces visibly thinner sections on brands with thin public footprints. Refusing the composite score means no big shiny number on the cover. And the report takes real time to produce — deliberately; I've argued elsewhere that a premium report shouldn't arrive instantly, and I stand by it. The unpopular half of that conviction: the work has to actually be going in. Latency you can't explain is theater from the other direction.

The economics are the root cause. Teams ship slop when there is no point in their pipeline where refusing is cheaper than publishing. If the vendor's cost structure is one API call and a template, every incentive says: always deliver, never decline, keep the word count. You can't fix that with a better model. You fix it with architecture — and the buyer's job is to find out whether the architecture exists. The stakes stopped being hypothetical this year: in one widely shared investigation, the AI-detection firm GPTZero dissected an AI-generated KPMG report and found conflated sources, exaggerated claims, and AI-injected references — copied into the final report with, as far as the investigators could tell, no human double-checking a single citation. Their name for the disease: "vibe citations." The genre's credibility problem is no longer a niche concern; it's a news cycle.

One thing before the detector, so the frame stays honest: none of what follows is "buy mine." It's how to interrogate anyone in this category, us included, whenever you do buy.

The buyer's detector

Five questions, each keyed to a gate above. Ask them before paying. The answers are hard to fake because they're about machinery, not quality adjectives.

Buy the walls, not the words

The model that invented an eighteen-year-old wrote beautifully that day. Every slop report ever shipped was fluent — fluency is the free part now. What you're actually paying for, in any AI-generated report, is everything wrapped around the model: the retrieval that happens before writing, the claims layer the prose can't escape, the verifier that reads the evidence, the gate that would rather say nothing than something false, the ledger that remembers all of it.

The words are the demo. The walls are the product. Price accordingly.

Two status notes, so the frame stays honest: the report system described here is operator-run today, not something you can buy from a page. And the coherence pass was the last of these walls to be built; while it was still in progress we said so, because claiming a wall is standing before it is, is the disease this piece is about.