80% of the articles published in AI & Society seem to be AI slop these days. OK, maybe that’s an exaggeration, but there are a lot. I hate wading through the slop to find the good articles they also publish. After seeing two “Curmudgeons’ Corner” opinion pieces yesterday that were very obviously AI-generated I fired off a Curmudgeons’ Corner of my own to the editors. It’s titled “AI & Society needs to stop publishing AI slop opinion pieces”. I hope they publish it – it’s certainly curmudgeonly, which is what they ask for. If they don’t I’ll send it somewhere else and publish it here.

I have an article of my own forthcoming in AI & Society, so I’m a little disappointed that the quality of the journal seems to be nosediving so rapidly. My article is precisely about AI slop. Here’s a link to the not-yet-resolving DOI in case you read this in the future and it’s already been published.

So I skimmed one of today’s freshly published LLM-generated articles on AI & Society, “Designing for rhythm: tempo-setting infrastructures and the temporal conditions of digital life” by Robert Atkinson, a professor of computer science at Arizona State University. The title made me suspicious: it has that dense rhythm that LLMs aim for in an academic article, but then again, that’s how we’re trained to write. The title of my forthcoming article might not be that much better, though I like to think it’s more concrete: “LLMs are metaphor machines: orbital argumentation, misaligned citations, and scientific fabrication in AI-generated writing”. It has a similar rhythm, unfortunately. Atkinson’s title has that evocative vagueness to it that LLMs love and, I suspect, that computer scientists think is the way humanities and social science research should be written. I ran it through Pangram, and unsurprisingly, Pangram says it’s 87% AI. That seems quite a generous assessment.

Anyway, I kept reading, and quickly saw exactly the same LLM-produced scientific fabrication in Atkinson’s article as I analyse in my forthcoming paper. Atkinson doesn’t cite well-known research on his topic, but instead cites an irrelevant paper by the author of one of the books that should have been cited. These are misaligned citations, and I argue that they’re a systematic error made by LLMs.

Here’s the abstract of my paper about this phenomenon:

Let me explain. Atkinson’s topic is digital temporality, and he starts off by saying there’s not much research on this, which surprised me, because I have been hearing about research on the topic for decades and it’s not even something I’ve directly done research on myself.

I read STS legend Judy Wajcman’s book Pressed for Time: The Acceleration of Life in Digital Capitalism a few years ago, and know many other scholars are publishing on the topic. So this seems like an odd claim to make. I did a quick Google Search to check. I’m in Greece right now on a writing retreat, so that’s why my Google Scholar interface is nice and Greek today.

OK, that’s just the first few hits. Let’s see if Atkinson cites Rob Kitchin’s book, the top hit, Digital Timescapes: Technology, Temporality and Society. Nope. But look, he cites a different article by Kitchin from 2017 about something not related to temporality. And wow, the sentence the reference is supposed to support is a great example of an LLM-generated not X but Y sentence too, just for good measure. With three references to articles that are all about digital media but not really specifically about the very vague claim made in the sentence.

OK, so the citations here are drive-by citations, performative decorations that make a text look like scholarship without actually doing the work of scholarship. The point of citing sources is to engage in a dialogue with previous researchers, to build on or challenge their work. Drive-by citations don’t do that. They’re performative rather than actually doing the work of research. And often, especially in LLM-generated texts, they’re completely irrelevant to the claim being made.

But this isn’t just an irrelevant citation. The reason the LLM picks Kitchin 2017 is that it should have cited Kitchin 2023, the book Kitchin wrote specifically about the topic of the whole article.

LLMs don’t pick references logically. They pick them based on statistical pattern recognition, they pick them based on proximity between tokens in the text in the context window (in this case the text Atkinson had already generated at this point of writing the article, or if he pasted the whole article into a tool like Grammarly and asked it to add references, the full text of the article) and tokens in the training data.

You could imagine a chain of connections, though that’s not quite what the multidimensional semantic space of an LLM really looks like. I think this might be a useful way for humans to analyse what’s going on, though.

My forthcoming article analyses equivalent examples in another clearly LLM-generated or at least heavily LLM-assisted paper, and I propose a theoretical argument for why this happens, building on John Gallagher’s concept of orbital argumentation, Anne Sigrid Refsum’s concept floating motifs and the narratological concept of unnarratability.

Atkinson’s article doesn’t cite Wajcman or the authors of other top Google hits for publications about digital temporality. This orbital argumentation where the LLM can’t directly name the originator of a term (as in the example I discuss in my forthcoming article) or the key scholars in a field, but instead pulls them in in the wrong way doesn’t happen every time. But I think every single irrelevant or half-relevant reference provided by an LLM (like Grammarly) is actually a misaligned citation, a citation generated through networks of similarity that are in the model but not direclty visible to us in the generated output. That means that AI-literacy requires a reader to evaluate those hidden connections – we can no longer assume there is a logical reason for a reference. We can’t even assume it’s sloppy scholarship (though it’s AI slop). It’s systematically wrong, and a symptom of something else having been left out. In this case, what’s left out is the extensive scholarship on the topic that is not mentioned in the article.

I have a lot more to say about this. I think it’s extremely important to understand. Please read my article when it’s out and tell me what you think! Oh, and if you see more examples of misaligned citations like this, please let me know in the comments!