I gave a talk about AI today for a group of knowledge workers in the public sector – people who are asked to write reports summarising research on a specific topic, and who coordinate funding schemes and advise on policy – and I asked them how they use LLMs in their work. Based on their free-text responses on Mentimeter it looks like there are four types of things they have found LLMs actually useful for:

1. Translation and copy-editing – checking the comma rules was one example, another was revising a text to be easier to read
2. Summarising long texts
3. Structuring information, comparing data (but this was also in the “didn’t work” category)

I also asked for examples of things they had tried to use LLMs for that hadn’t worked:

1. Literature searches and finding factual information
2. Analysing a topic or texts, reflective analysis, complex topics, public consultations (høring, an institutional response to a policy proposal),
3. Summarising large or complex documents
4. Excel
5. Making a seating chart
6. Maintaining ambiguity or multiple perspectives, e.g. in an article with two argumments
7. Anything related to specialised expertise that is not heavily discussed online

These are slightly different types of tasks to the ones I wrote about yesterday, in the benchmarking test where LLMs failed at 70% of office tasks. My main advice in the talk today was to think carefully about whether or not a language model is actually the right tool for the task at hand – and to consider what is lost by not doing it yourself. Sometimes speed and a good-enough product is the most important thing. Other times the friction of working with a process is important in itself because that is where the thinking happens. And then there are tasks that are just better suited to other tools.