I came across Koenraad in the copy room yesterday photocopying a dozen or so copies of Christine WennerÂs and Agnes Wold’s dissection of the review process for Swedish post. docs (here’s a freely available copy for those without insitutional access), where they very convincingly demonstrate that to get the same score in the evaluations, female candidates had to have 2.5 times as many publications as men. I’d heard about the study while listening to Virginia Vallan’s talk Why So Slow? The Advancement of Women (highly recommended!) but hadn’t actually read it before. After reading, I’m shocked that the results aren’t more widely known and discussed. This paper is almost ten years old!

So in brief: WennerÂs and Wold noticed that while there is an almost equal number of male and female applicants for post docs in medical sciences in Sweden, twice as many men as women actually receive a fellowship. They wanted to know whether this was due to the female applicants simply not being as good, or due to a gender bias. Conveniently, there was central and consistant evaluation of these candidates, but the evaluations were not public. So WennerÂs and Wold went to court to gain access – and they gained it, as the Swedish consitution makes state documents that are not a threat to national security open to the public. Once they had the data, they analysed it in a number of ways. First they found that only the very best women were given a score as high as the average man, and that women with equal numbers of publications as men got lower scores than those men. Women would in fact have to publish 2.5 times as much to get the same score as men. Oh, having an affiliation with a committee member helped a lot, too, even though the person you were affiliated with was not allowed to be directly involved in assessing you. There’s lots more: you should read the paper, it’s only three pages long.

One thing I found interesting was that the quantitative methods WennerÂs and Wold use appear to cut through the gender bias inherent to more qualitative evaluation methods. In the talk I mentioned above, Vallan very engagingly describes dozens and dozens of studies showing that we all, men and women alike, have inherent prejudices and expectations to what men and women are like, and that without being extremely conscious of this, we will always overestimate men’s height, intelligence and leadership capabilities and underestimate those of women.

In the last years, many universities and countries have moved towards more quantitative measures of research production. Here in Norway there are now lists of approved journals sorted into level 1 (good) and level 2 (better). You’re assigned one point for each article you publish in a level 1 journal and five points for a publication in a level 2 journal. 0.7 for a chapter in a book with a level 1 publisher and one point for a chapter in a book with a level 2 publisher. Five points for a whole book with a level 1 publisher and seven for a book with a level 2 publisher. This is all logged electronically, reports are generated where articles in journals not on the list don’t show up at all, departments get extra money according to how many points they’ve earned and so on and so forth.

There’s been a lot of complaining about this. I hate seeing how some of my publications simply become invisible in this system – and it seems unfair when I know that some of my invisible publications are actually the ones being cited most often.

But listen: such a system makes the rules of the game absolutely explicit. I know which journals I should publish in for my university to approve of me, and to make sure I show up as productive in those reports. And other studies of bias in peer review show that women have less publications in high impact (level 2) journals. As Young Female Scientist points out, that might be because women, perhaps, read more widely, and therefore don’t care as much about whether a journal is high impact or not. I know I haven’t cared much about it. In terms of getting ahead career-wise, I guess I should.

Maybe quantitative measuring of research production is actually a way of fighting gender bias. It makes the game rules extremely clear in advance, so women (and men of course) are completely aware of what the criteria we’ll be measured by will be. It also skips the prejudiced evaluations we all make when we assess each other. Presumably, this would help avoid other biases, as well, like those based on ethnicity, age or citizenship. Of course, there’ll still be peer-review in the journals we try to get our papers accepted to, but at that level, perhaps the double blind review is more possible than in assessing the publications and other achievements of an individual. Until they placed a screen between the musician and the evaluations in auditions for orchestra positions, far more men than women were deemed good enough. With the screen in place and the anonymity it granted, women and men were accepted in equal numbers.

WennerÂs and Wold don’t suggest quantitative measure as the solution. They argue instead that evaluations should be public, and that this openness will force fairness. That might be, if checks are run on those evaluations often enough that our consciousness of these isues changes. Pharyngula, in a piece on this discussing the exact same tendency in European research grants (about 10% of male applicants and 5% of female applicants got a grant) notes that anonymous review masks systemic abuse and bias, which can be true. In the long run, though, I think that raising our awareness of our prejudices is the only real way of changing bias – and it’s going to be very, very hard. Perhaps quantitative evaluation and open review processes can help. No doubt, both have other problems.

Our university is striving to become the nation’s best university in terms of gender equality. A draft of the new strategy plan for equality (download as a word doc) has been sent out to all the departments for comments, and on Thursday there’s a conference to discuss it before it’s finalised. I haven’t finished reading the strategy plan yet, but will be at the conference and certainly plan to comment.

Hardly anyone’s signed up for the equality conference, though. So easy to just ignore.

2 thoughts on “perhaps we should be glad of quantitative measures of research productivity?

  1. Bradley Wentworth

    I read the Economist, and they just published an article that cites the same Swedish study as you do Jill. It goes on to summarise a new one conducted in England that further explores both the gap in salary and in ‘rising […] through the hierarchy’.

    It’s a compact but revealing article: http://www.economist.com/science/displaystory.cfm?story_id=7880036

  2. […] I’m rather shocked at this story, I must admit. Obviously it’s hard to know whether you’d have more success as a man when you always present yourself as a woman. I’ve never experienced obvious discrimination, though I’ve certainly felt uncomfortable in meetings where everyone else is male. Oh, and fumed at the difficulties of breastfeeding while travelling and noted that applications aren’t evaluated equally and that I have (had? have…?) a tendency to act like a little girl, a typical mistake women make. […]

Leave A Comment

Recommended Posts

Triple book talk: Watch James Dobson, Jussi Parikka and me discuss our 2023 books

Thanks to everyone who came to the triple book talk of three recent books on machine vision by James Dobson, Jussi Parikka and me, and thanks for excellent questions. Several people have emailed to asked if we recorded it, and yes we did! Here you go! James and Jussi’s books […]

Image on a black background of a human hand holding a graphic showing the word AI with a blue circuit board pattern inside surrounded by blurred blue and yellow dots and a concentric circular blue design.
AI and algorithmic culture Machine Vision

Four visual registers for imaginaries of machine vision

I’m thrilled to announce another publication from our European Research Council (ERC)-funded research project on Machine Vision: Gabriele de Setaand Anya Shchetvina‘s paper analysing how Chinese AI companies visually present machine vision technologies. They find that the Chinese machine vision imaginary is global, blue and competitive.  De Seta, Gabriele, and Anya Shchetvina. “Imagining Machine […]

Do people flock to talks about ChatGPT because they are scared?

Whenever I give talks about ChatGPT and LLMs, whether to ninth graders, businesses or journalists, I meet people who are hungry for information, who really want to understand this new technology. I’ve interpreted this as interest and a need to understand – but yesterday, Eirik Solheim said that every time […]