Hi, I'm Charles Costa, researcher, strategist, and founder of Lexora Labs™. Each week, I dig into research on AI, knowledge management, and the future of work so you don't have to.
EDITOR'S NOTE
Last week we sat with an old rule that suddenly feels expensive again: trust, but verify. The hidden cost was the human labor of checking what models produce. This week follows the tease. What happens when the reference layer itself starts to degrade? What happens to the people whose entire job is to be the thing everyone else checks against?
Editors, peer reviewers, fact-checkers, and preprint moderators are that layer. They are not a free API call. They are overloaded humans trying to keep the record trustworthy while AI floods submissions, invents citations that look real, and tempts authors to treat review as someone else's problem.
I keep returning to a framing I've used before. Treat AI like a new hire who is confidently wrong: quick, fluent, missing tribal knowledge. Accountability still sits with the humans who maintain the ecosystem. Critical thinking stays on the author. Peer review is not a full fact-check on your behalf. And as Tim O'Reilly has argued, AI writing is a new medium. Don't punish people for using it. Use it properly.
THIS WEEK'S READING:
Science Is Drowning in AI Slop | The Atlantic
Why It Matters: Journals and preprint venues are clogging with AI-assisted manuscripts, phantom citations, and reviews that look polished until someone actually checks. Editors and unpaid peer reviewers are the overloaded human check when the scientific record fills with slop. Volume and plausibility of bad work are rising together.
If you're tasked with peer review and you're not doing the review yourself, that's the failure. Using AI to retrieve information or surface relevant sources can be acceptable. Just running a paper through a model to "scan" it is ridiculous. It always returns to critical thinking.
Once phantom papers show up in respected venues, the question stops being "is there slop?" and becomes "how far did it travel before anyone noticed?"
Journal Submissions Riddled With AI-Created Fake Citations | Inside Higher Ed
Why It Matters: Fake references now look formatted, plausible, and correctly styled. They name real authors and real journals, then point nowhere. Editors report phantom citations surviving multiple review rounds and only getting caught in copyediting, after expensive review labor has already been spent.
We need to rethink peer review. Desk-reject papers with invalid references before thorough review. Most important: disclose AI usage, then gauge critical thinking versus AI use. If a tool hallucinates a cite, the author who didn't check failed first. Peer review was never meant to be the author's only fact-check.
Verification labor is not only an academic problem. Newsrooms face the same squeeze, often with tools built for a different workflow than the one reporters actually live in.
Why It Matters: A Princeton CITP report finds authentication and provenance tools racing into a market where fabricated media is cheap, while journalists who verify day to day are under-consulted in the design. Confidence scores, broken chain-of-custody manifests, and privacy risks collide with how reporters and audiences decide what is real.
Journalists are the right input for designing review and content processes. Chain of custody shows where things went off the rails and who failed. Enterprises already use audit trails in content and asset management. Newsroom-versus-authority battles are not apples-to-apples for corporate professionals, but privacy is still key to building trust in tools and standards so people actually use them. The real question is how these systems evaluate information, not whether a vendor stamped "AI authentic" on the box.
When institutions cannot keep up with review alone, some try hard rules on the people who submit. That is a reputation move, not a model move.
Why It Matters: arXiv is putting accountability on authors, not models. Submit AI-generated hallucinations or careless slop, and listed authors can face a yearlong submission ban plus a lasting requirement that future work clear journal peer review first. Humans maintain the ecosystem. Humans own the consequences.
For now, the key is establishing permissible uses of AI, clear frameworks, and proper disclosure. That matters more than debating the exact shape of a ban. Communities and entities determine how to ensure quality because they own their reputation. I am not here to set ban length for arXiv. I am here to notice who gets held responsible when the reference layer fails.
So what do the people whose job is verification actually say about generative AI? Not that the model owns the record. That they are trying to keep pace without losing the human infrastructure that makes "checked" mean something.
The Impact and Opportunities of Generative AI in Fact-Checking (Wolfe et al., FAccT 2024) | FAccT / Consensus
Why It Matters: Peer-reviewed interviews across 29 fact-checking organizations show GenAI opportunities in editing, investigation, and advocacy, alongside value tensions around transparency and reliability. The through-line is human infrastructure and labor. Verified information still comes from people collaborating with tools that remain unreliable if left unsupervised.
People are not pretending AI owns the reference layer. They are trying to keep pace with change and do what seems reliable. Same new-hire problem: confidently incorrect, fast, missing context. Authors of work like this can elicit more enthusiasm than warranted for GenAI in fact-checking. The hard parts remain contextual and portrayal answers versus boolean facts, data governance (garbage in, garbage out), and whether organizations can maintain the expertise to keep accurate standards.
AI augments human judgment. It does not replace it. Creating content still requires critical thinking end to end. Set-and-forget turns the model into a crutch. Challenging your mind while you author is the right use. Treat the model as a sounding board or thinking partner, not a ghostwriter you send off alone.
Last week, trust but verify was about the cost of checking AI outputs. This week, the same rule applies to the reference layer itself: journals, preprints, citations, provenance trails, and fact-check pipelines everyone else leans on. Accountability sits with humans because they maintain the ecosystem and develop standards accordingly, not with models that cannot own a reputation.
Walkaway: trust, but verify. Especially when the thing you are verifying against is under stress.
One thing I'm still turning over is who still carries the reference layer, and who gets to evaluate it. Companies chasing AEO often assume models look for the same things search engines do. They do not. Every AI system can be trained or prompted toward different criteria. Uniformity was never the default.
That's next week.
Know someone who still treats "the literature said so" as a finished sentence? Forward this to them.
Have a take on any of this? Reply directly. I read every response.
Charles Costa, MLIS