← blog / AI content agents

How to Fact-Check AI Content: A 20-Minute Checklist

Abstract Magic Share brand graphic: a field of dots with a highlighted cluster in green and teal on a near-black grid, with the Magic Share name and the label "ai-fact-checking".

Seventy-six percent of marketing leaders now spend at least three hours a week editing, fact-checking or correcting AI-generated output. Asked what creates the most extra work, 48% named fact-checking and hallucination review, ahead of every broken-tooling complaint on the list. Both figures come from Optimizely's 2026 global study, which surveyed 2,003 marketing leaders across seven markets in May and June 2026. "AI was supposed to give marketers room to think," said Optimizely's SVP of Marketing Tara Corey. "What most teams got instead was more to manage."

Three hours a week is the tax. This post is about paying less of it without lowering the bar — a repeatable 20-minute pass over a draft that catches the errors most likely to embarrass you, in the order that costs you the least time.

The organising idea is simple: most of the damage in an AI draft comes from its citations, not its opinions. A wrong-ish opinion gets edited in a sentence. A citation that points at nothing gets screenshotted, sanctioned or retracted. And citations have a property almost nothing else in a draft has — they are mechanically checkable. As one 2026 audit of hallucinated references put it, a reference "either resolves to a real scholarly work with compatible authorship, or it does not" (Russinovich et al.). That auditability is why verification should start with the links.

The 20 minutes, in short:

  • Minutes 0–5, resolve (machine): extract every URL, request each one, and sort the results into live, dead, likely hallucinated and unknown.
  • Minutes 5–15, match (human): for each quote, statistic and named work, find the sentence in the source that actually carries it.
  • Minutes 15–20, sniff: work out who runs each unfamiliar domain, trace aggregated numbers to their primary source, and read your sentence against theirs.
  • The copy-and-paste checklist is at the end of the post; everything in between explains why this order.

The three ways a citation fails

There isn't one failure mode. There are three, and they need different checks.

1. The URL never existed. Across commercial models and deep-research agents, 3–13% of citation URLs were hallucinated outright, and a wider 5–18% failed to resolve at all (Rao, Wong & Callison-Burch, University of Pennsylvania). The paper is precise about the relationship between the two numbers: "Hallucinated URLs form a strict subset of non-resolving URLs." Every fabricated link is broken; not every broken link was fabricated.

And the configurations you'd expect to be safest weren't: the single highest hallucinated-URL rate in the audit came from a deep-research configuration that reads the web before answering. Retrieval narrows the problem without removing it. The authors' own summary is worth pinning above a desk: "Generating more citations does not compensate for, and may exacerbate, per-citation quality problems."

2. The URL existed and died. A quarter of all webpages that existed at some point between 2013 and 2023 were no longer accessible as of October 2023, and 54% of English-language Wikipedia pages had at least one dead link in their references (Pew Research Center). The Wikipedia figure matters because Wikipedia is one of the most common sources in AI training and retrieval data.

The practical consequence: "verified once" is not "verified forever." A verification report has a timestamp for a reason.

3. The URL resolves perfectly and doesn't say that. This is the big one by volume, and the one a link checker will never catch. Wu and colleagues tested seven models on 800 medical questions, scoring every statement against the source cited for it. Their finding: "between 50% and 90% of LLM responses are not fully supported, and sometimes contradicted, by the sources they cite." Even the best performer, GPT-4o with web search, left roughly 30% of individual statements unsupported (Wu et al., Nature Communications, 2025). A separate 2025 audit of deep-research systems found the same pattern, with large shares of statements unsupported by the sources the systems themselves listed (DeepTRACE, Salesforce AI Research).

One case contains all three failures at once. In Whiting v. City of Athens, the Sixth Circuit counted over two dozen fake citations and misrepresentations in briefs across three consolidated appeals: a cited case that does not exist, quoted language that appears nowhere in the cases cited for it, and real cases cited for propositions they do not address. The two attorneys were ordered to pay $15,000 each (Sixth Circuit Appellate Blog). Invented cases, invented quotes, real cases that say something else. That's the checklist in miniature.

And the failure isn't limited to models being asked to find sources. A December 2024 letter in Intensive Care Medicine carried 15 references. Five could be traced to real articles, two of them with errors; the other ten matched nothing in the cited journal or anywhere else. Reference 11 cited a nonexistent paper supposedly published in the very journal printing the letter. According to the retraction notice, the authors said the references came from using generative AI to convert the PubMed IDs of cited articles into a structured reference list. The letter was retracted on 29 November 2025 (Retraction Watch). The model wasn't researching. It was reformatting, arguably the most constrained citation task there is, and ten of fifteen references pointed at nothing.

Minutes 0–5: resolve every URL

Start here because it's mechanical, fast, and it removes the errors with the worst consequences. Do not read the draft yet. Extract every URL and check what each one returns.

The cleanest mental model available comes from the Penn audit, which sorts every citation into four states:

State How it's determined What it means
LIVE HTTP 200 The page exists. That is all it means.
DEAD HTTP 404 and an archived snapshot exists Genuine link rot. Swap in the archive link or re-source.
LIKELY HALLUCINATED HTTP 404 and no archived snapshot The URL probably never existed. Cut the claim or re-source it.
UNKNOWN 403, 429, timeouts, connection failures Needs a human or a real browser. Do not auto-delete.

The 404-plus-archive test is the part most checklists miss, and it's the difference between "this link rotted" and "this link was invented." The Internet Archive exposes it as a single unauthenticated GET: http://archive.org/wayback/available?url=example.com, optionally with a &timestamp=YYYYMMDDhhmmss. A hit returns an archived_snapshots.closest object with available, url, timestamp and status; a miss returns exactly {"archived_snapshots":{}} (Wayback Availability API docs). The reference implementation from the Penn paper, urlhealth, is 83 lines of Python that issue an HTTP HEAD request (which fetches only a page's response headers, not its body) plus that Wayback lookup.

The 403 trap, and the soft-404 trap

Two traps sit on either side of this step, and both produce confident wrong answers.

Treating 403 as fabrication. HTTP 403 is the single biggest source of false positives in automated link checking. When the Penn team re-checked 403 responses with a headless browser (a real browser engine driven by a script, with no window), 99.7% of those pages were live — the blocks came overwhelmingly from academic publishers refusing bot-like traffic, not from dead pages. Cloudflare's own documentation describes the same class of false positive: bot-management rules routinely trigger on legitimate monitoring and checking services because of their user agent or cloud IP range (Cloudflare WAF troubleshooting). If your checklist deletes every 403, you will delete good citations. (We hit exactly this while verifying the Sixth Circuit link above — 403 to an automated fetch, fine in a browser.)

Treating 200 as verification. A server can return HTTP 200 on an error page, a paywall interstitial or an empty template. That is the classic soft 404: a page that shows the reader a "not found" message while sending the crawler a success code (Google's definition of a soft 404). A 200 tells you a page exists at that address. It tells you nothing about whether your claim is on it.

This whole step is a machine's job, and machines are now very good at it. When the Penn team gave models a URL-checking tool plus a self-correction loop, non-resolving citations dropped below 1% — GPT-5.1 from 16.0% to 0.6%, Gemini 2.5 Pro from 6.1% to 0.1%, Claude Sonnet 4.5 from 4.9% to 0.8%, reductions of 6× to 79×. The catch they flag honestly: the loop only works if the model actually acts on the tool's output.

That tool-plus-correction loop is how we built Magic Share: the agent checks its cited sources and fixes or drops the ones that fail before it hands over a draft, and the verification report ships attached to the draft so the reviewer starts from a list of what was checked rather than an empty page. Roughly 20 sources go into a post, which is about 20 URLs to check — a tedious five minutes by hand that the agent works through for you.

Minutes 5–15: match each claim to the sentence that supports it

Now open the draft. This is the judgement half, where the 50–90% unsupported-statement problem lives, and it cannot be fully automated. It can, however, be made fast — and the evidence on how to make it fast is unusually clear.

When researchers compared professional fact-checkers with PhD historians and Stanford undergraduates on live web evaluation tasks, the fact-checkers "arrived at more warranted conclusions in a fraction of the time." The advantage wasn't knowledge; it was technique. Fact-checkers read laterally: they leave a page after a quick scan and open new tabs to establish who's behind it, while everyone else read vertically, staying on the page and reading harder (Wineburg & McGrew, Teachers College Record). Applied to an AI draft: for each cited domain, spend seconds working out what it is, then go straight to the sentence that carries your claim. Do not read the article.

Use a text fragment to pin the sentence

Browsers give you a free verification tool here. A text fragment appended to a URL scrolls to and highlights an exact string:

https://example.com/report#:~:text=between%2050%25%20and%2090%25

The syntax is #:~:text=[prefix-,]textStart[,textEnd][,-suffix]. It's case-insensitive, needs percent-encoding for spaces and punctuation, and works only on user-initiated navigation. The useful part: when the text isn't found, the whole fragment is ignored and the browser links the top of the document (MDN reference). That makes it a self-testing citation. If the highlight lands, the sentence is there. If you get dumped at the top of the page, the sentence isn't. (Sites can opt out with a Document-Policy: force-load-at-top header, so a non-landing highlight is a strong signal rather than proof.)

Triage, because you have 15 minutes and not two hours

You will not check every sentence. Check in this order — the ordering is our reasoning, but each rung is justified by something measured.

  1. Quoted speech. Courts have effectively priced this: in Couvrette v. Wisnovsky in the District of Oregon, the sanction was built at $500 per fake case and $1,000 per fake quote, plus fees, reaching $110,204.38 combined (AI hallucination sanctions tracker). A fabricated quote costs double a fabricated cite, and it's the fastest thing on the list to check: paste the exact string into the page and use your browser's find function.
  2. Numbers with a source attached. These are what readers screenshot and what competitors check. Verify the figure, the year, and the population it describes.
  3. Named entities paired with titles. This is the signature fabrication pattern. In an analysis of ten AI-generated essays, about 30% of hallucinated-citation cases were duplicates of the same phantom reference — a paper titled "Education Governance and Datafication," attributed to two real and prominent scholars, that does not exist (Spennemann, 2026). Real people, plausible title, no such work. A link-resolution step catches that instantly; a plausibility read never will.
  4. Everything else, at whatever speed remains.

Watch for scope drift

The most common way a correctly sourced number becomes a false published claim is that it quietly grows. Two worked examples from this very post:

  • The Pew figure is 25% of pages that existed at some point between 2013 and 2023, measured as of October 2023. It is not "a quarter of the web is dead."
  • A widely quoted Graphite study puts AI-generated articles at parity with human-written ones: 49.9% of sampled articles in the first quarter of 2026, flat at around half since early 2025. But the sample is 55,400 English-language URLs from Common Crawl carrying article schema and 100+ words, scored by averaging three detectors, and the study adds a caveat most write-ups drop: those articles "largely do not appear in Google and ChatGPT" (Graphite). Graphite's own earlier release, built on a single detector, ran 3.3 points higher, which is exactly why the method has to travel with the number. Publishing volume is not visibility.

When you check a number, check its scope: US or global, one industry or all, one year or a decade, sampled or complete. Same number, wider claim, is a factual error.

Minutes 15–20: three sniff tests

The last five minutes catch what the first two steps structurally cannot.

Is the source actually a source? NewsGuard had identified 3,749 AI-generated news and information sites across 16 languages as of 23 June 2026 — sites with substantial AI-produced content, no meaningful human oversight, no disclosure, and a presentation designed to look like a legitimate outlet (NewsGuard AI Tracking Center). A live URL on a professional-looking site is not evidence of anything. If you don't recognise the domain, spend fifteen seconds finding out who runs it.

Does the draft cite the source or a copy of it? Aggregators, press summaries and roundups drift from the original. If a statistic matters enough to headline a section, follow it back to the study, the filing or the press release and cite that instead.

Does the draft claim more than the source claims? Read your own sentence and the source's sentence side by side and ask whether a hostile reader could say "that's not what it says." Hedges the source used, such as "associated with," "in this sample" or "self-reported," should survive into your version.

What 20 minutes does not buy you

A post that oversells its own checklist has the same problem as a draft that oversells its sources. So, plainly:

A complete link audit plus a triaged claim check is not a full verification. Fabricated URLs run 3–13%; unsupported-but-real citations run 30–50% and up even in the best-behaved configurations. Twenty minutes buys you the whole link problem and the highest-value slice of the claim problem. It does not buy you an academic review.

Some AI output can't be checked this way at all. Jonathan Bailey made the strongest version of this argument in June 2026: testing six platforms on a general research prompt, he found only two supplied citations at all, and argued that without sources you must "examine every single claim independently" — which makes the tool less useful than advertised (Plagiarism Today). He's right about sourceless generation. The fix is a different artefact, not a better checking habit. A draft that ships with a citation per claim converts an unfalsifiable output into a checkable one — which is why the checking step belongs inside the writing step rather than bolted on after.

And the SEO case is weaker than most posts on this keyword pretend. Broken outbound links are not a documented direct ranking penalty; Google's John Mueller said back in January 2016 that linking out to other sites doesn't help a page rank better, and nothing since has reversed that (Search Engine Roundtable). The real exposure is elsewhere. Google's helpful-content guidance asks self-assessment questions that fabricated citations fail directly — "Does the content have any easily-verified factual errors?" and whether the page shows "clear sourcing, evidence of the expertise involved" (Search Central). Google's spam policies separately target scaled content abuse (publishing many pages without adding value), which we unpacked in our close reading of the scaled content abuse policy. Verify links for credibility and legal exposure. Any ranking benefit is a side effect.

Why human review alone isn't the fix

The instinctive response to fabricated citations is "review more carefully." The evidence says that isn't enough on its own.

Papers at top machine-learning and security venues go through multiple expert reviewers (the audit notes three to five reviews per submission at ICLR 2026). In 2025, roughly one in twenty accepted papers at NeurIPS and USENIX Security still contained at least two likely hallucinated academic references — even though per-reference hallucination rates stayed below 1%, proceedings are large enough that paper-level failures show up (Russinovich, Siva Kumar & Salem). Their automated checker costs about $0.04 per paper to run. Expert human review, without a resolution step in front of it, is not a filter.

The wider trend runs the same direction. A Columbia University team analysing more than two million papers and 97 million citations found fabricated-reference rates climbing from roughly 1 in 2,828 papers in 2023 to 1 in 458 in 2025, and 1 in 277 in the first seven weeks of 2026 (STAT News on the Lancet study). "The more serious issue here is that citation practices are changing with the generative AI use," Northwestern's Mohammad Hosseini told STAT. "That means that the engagement with the literature is becoming increasingly more superficial."

It also isn't a bug waiting on the next model release. OpenAI researchers argue that hallucination persists because evaluations reward guessing: "Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty" (Kalai et al.). Their proposed fix is changing how benchmarks score abstention — a socio-technical change, not a patch.

Meanwhile, assuming a newer or paid model handles it is its own trap. The Tow Center tested eight AI search tools with 1,600 queries and found they "provided incorrect answers to more than 60 percent of queries," with rates ranging from 37% to 94% by tool. One tool returned error pages for 154 of the 200 citations it produced. And premium tiers "paradoxically also demonstrated higher error rates" than their free equivalents, because they answered confidently instead of declining (Columbia Journalism Review).

So the working order is: machine resolution first, human judgement second. Poynter's updated newsroom guidelines land in the same place: they treat human coverage as the preferred option where possible and emphasise the values of "transparency, fairness, human oversight and audience trust" (Poynter). That's the shape of a review that works: let software prove the links resolve, then spend your scarce attention on whether the pages say what the draft claims.

It's also the design behind Magic Share's approval gate. Drafts grow in a greenhouse, not in production: the agent researches, writes, resolves every cited URL and then stops, holding the post until a person approves it. Nothing publishes on its own. You stay in the loop; the point is that when you arrive, the mechanical work is already done and your 20 minutes go entirely to judgement.

The 20-minute checklist

Copy this into your review template.

Minutes 0–5 — resolve (machine):

  1. Extract every URL in the draft.
  2. Request each one. Sort into LIVE (200), DEAD (404 with an archived snapshot), LIKELY HALLUCINATED (404 with no snapshot), UNKNOWN (403/429/timeout).
  3. Replace DEAD links with an archive snapshot or a fresh source.
  4. Cut or re-source every LIKELY HALLUCINATED link — and check whether the same phantom source appears twice.
  5. Re-check UNKNOWNs in a real browser. Never auto-delete a 403.

Minutes 5–15 — match (human):

  1. For each quoted string, paste it into the page and confirm it appears verbatim, with the attribution intact.
  2. For each statistic, confirm the number, the year and the population — then check your sentence hasn't widened the scope.
  3. For each "study by X" or "paper titled Y," confirm the work exists and the named authors wrote it.
  4. Where possible, add a text fragment to the link so the citation tests itself on every future click.

Minutes 15–20 — sniff:

  1. For unfamiliar domains, spend fifteen seconds identifying the publisher.
  2. Follow aggregated statistics back to their primary source and cite that.
  3. Read your sentence against the source's sentence and delete any claim the source doesn't make.
  4. Date-stamp the review. Link rot means today's pass expires.

FAQ

How do I tell a hallucinated citation from a dead link? Check the archive. A 404 with a Wayback Machine snapshot is genuine link rot, so swap in the snapshot; a 404 with no snapshot anywhere most likely never existed, which makes the underlying claim suspect too.

Is a broken link in my post an SEO penalty? Not a documented direct one; Google said in 2016 that linking out doesn't help a page rank, and that still stands. The real exposure is credibility and, in regulated or legal contexts, liability.

Can I just trust a newer or paid model to get citations right? The measurements say no: paid tiers and deep-research modes have both posted higher citation error rates than cheaper configurations. The improvement comes from a verification step in front of the model, not from model size.

Twenty minutes for a 3,000-word post — is that realistic? Yes, if the link resolution is already done when you sit down; that step is the bulk of the mechanical work and the whole of the automatable part. If you're resolving 20 URLs by hand as well, budget closer to 45 minutes.

Wrap-up

Most AI citation errors are not arguments you disagree with. They are addresses that lead nowhere, or addresses that lead somewhere real and irrelevant. That's good news: both are checkable, one by a script and one by a reader who knows what to look at first. Resolve every URL, sort the dead from the invented, then spend your remaining minutes on quotes, numbers and named works — in that order.

If you'd rather not do the first half by hand, that's the part we automated. Magic Share researches your niche, drafts a 600–4,000-word post sized to the query, checks every cited source before it hands the draft over, and then waits for you — reviewing a draft takes a coffee, not an afternoon of fact-checking. Start free — your first three posts are on us.

More on AI content agents →

by email

Get new posts in your inbox

A short digest when new posts go up. Confirm once, unsubscribe from any email.

We use your address only for this digest. Privacy

Keep reading

  1. ·10 min read

    Blog Post URL Slug Best Practices: Pick Once, Keep Forever

    Blog post URL slug best practices for static blogs: short, year-free slugs pinned in front matter, and a 301 plan for each host if you ever change one.

    on-page seourl slugsredirectsstatic sitetechnical seo

  2. ·15 min read

    How to Mention Your Product in a Blog Post Without Pitching

    How to mention your product in a blog post so it reads as the next step, not an ad: where the mention goes, how often it shows up and what it can claim.

    content-strategyproduct-led-contentsaas-bloggingconversionseo

  3. ·11 min read

    Blog Attribution on Low Traffic: Did It Bring Signups?

    Funnel attribution breaks below a few hundred sessions a month. Three durable signals that show whether your SaaS blog is bringing signups on low traffic.

    content-strategyblogginganalyticsattributionsaas-marketingseo

Want posts like this for your site?

Magic Share researches, writes and fact-checks posts like this for any site — point it at your URL and review your first draft today.

Plant your first post