Dorothy Bishop, a psychologist, wrote a useful list of what reviewers need to look out now that so many papers are bad science that looks good thanks to LLMs. Read her whole blog post, she explains it well, but here’s a brief version because I’m pretty sure we’ll be needing to refer back to this.

1. Assume what you’re reading might be fraudulent. Yes, it’s sad, but this might have to be the new starting point.
2. If data was analysed, insist on seeing the data and code before reviewing the paper. Dorothy gives an example of nonsense code and datasets that could not realistically be combined. Disheartening.
3. Do the cited references exist, and are the relevant? Are key works (e.g. coining central terms in the paper) referenced? This is the common LLM problem I wrote about the other day in my post on misaligned citations.
4. Does the article make sense? Are there tortured phrases? Do they use a method that isn’t appropriate for the research question, like a systematic review for a humanties question? Are headlines in strange places?
5. Is it plausible the researchers did what they say they did in the stated time frame and with the stated resources?
6. Are the researchers legitimate? If they don’t have a track record, or if they’re (say) a computer scientist or physicist suddenly publishing in the humanities, be a little extra vigilent.

Unfortunately all this takes a lot of extra time. But I agree with Dorothy that it seems to be necessary.