AI put “synthetic quotes” in his book. But this author wants to keep using it.

Slipping through the cracks

Rosenbaum used AI tools during his writing process, he told me, “to surface ideas, locate articles, summarize themes, identify people or papers I might want to look into.” He draws a hard line between this kind of research and the “actual reporting, narrative structure, interviews, arguments, and conclusions in the book,” which he says are “entirely mine… There was never a time when AI was writing the book.”

In addition to chapters based on transcribed interviews that Rosenbaum says he conducted himself, The Future of Truth also includes more research-based chapters in which Rosenbaum said, “We’re pulling facts and then knitting them together into a narrative.” Tools like OpenAI’s ChatGPT and Anthropic’s Claude were used heavily to gather information, he said, with any nuggets mined by those tools tagged with a “this came from AI” warning in his notes.

It’s strangely creative and crafty and unusual in all these ways … and then it betrays you in ways that are just really quite horrible.

Steven Rosenbaum

Those tagged AI-generated notes were then passed on to a fact-checker and two copy editors provided by the publisher, Rosenbaum said. Of the 285 outside citations in the book, six have been identified by the Times as problematic, including three so-called “synthetic quotes” that have no apparent source. (More examples could turn up as the book undergoes further review. And it’s worth noting that most writers manage to include zero made-up quotes when they write a book.)

“I think we did that [double-checking] incredibly effectively, but not a hundred percent,” Rosenbaum told Ars. “We’re doing the work, we’re doing the best we can. We look at it, it looks right. We double-check it, and then we made a mistake.”

But the significant failure here highlights how the traditional fact-checking process might be ill-equipped to handle AI-assisted research. In the past, a fact-checker could be reasonably confident that any author quoting cited written works had simply copied down those quotes directly. These quotes would need to be checked, of course, but the fact that they’re so easy to verify makes them less inherently suspicious. If AI tools are involved anywhere in the pipeline, though, that assumption goes out the window, and there needs to be an extra layer of skepticism that those quotes had been copied correctly or that they even exist at all.