The REF is a flawed problematic technology that emerges in the context of the neoliberalization of higher education and is widely criticized by academics across blogs and journal pages. In my experience, most research and teaching active academics tolerate the REF, but not many of us embrace it. It detracts from our teaching and scholarship, it encourages disciplinary forms of performance management, it discourages long-term and ambitious monograph projects, and makes it difficult to embody alternative (non-REFable) academic subjectivities. As a result of the high social and financial cost of the REF, the government commissioned an independent review carried out by Nicholas Stern that has now been published. I was surprised to read in the review that “many respondents to our consultation stated that research and the HE sector would be poorer without it and that largely the benefits far outweigh the costs”. The review does not say who the respondents to the consultation were nor how extensive it was. So the review starts with an assumption that REF is mostly good and is here to stay but that it needs a little improvement here and there. The review identifies some of the flaws with the system and poses some recommendations in order to mitigate these. Since the review was published last week, a number of scholars have produced a range of critiques and endorsements that I have read with interest (see for example Campaign for the Public University 2016; Morrish 2016; Bhandar 2016; Wilsdon 2016). While a minority welcome the recommendations, the majority are more critical and have emphasized how the recommendations will be harmful to early career and BAME scholars. I think they are also potentially harmful to women and to scholars whose work is seen as marginal within their school or college.
The review identifies some of the well-known problems with the REF and there are three on which I wish to comment: the question of portability; the question of equality and diversity; and the question of interdisciplinarity. The recommendations seek to prevent the so-called “gaming” of the system, in particular where institutions hire high performing scholars just before the census date in order to enhance their REF return, by putting an end to the portability of outputs. They also seek to enhance equality and diversity at tertiary institutions. The review notes with concern how white men get submitted at a much higher rate than both women and people of colour. Finally, the review also wants to make it easier for interdisciplinary research to get included.
The section on portability is so flawed, it beggars belief. A number of commentaries have flagged its problem for early career scholars on short term contracts, pointing out how the university that has failed to give them a permanent contract should not be able to benefit from their publications and undermine their future career prospects. But the lack of portability is not just a problem for ECRs but for anyone. Indeed, it appears to be based on an assumption that good scholars only move jobs because they have been poached for REF purposes. They do not. They move to be closer to a partner (the nature of the academic job market and the absence of formal spousal hire policies means that lots of academic couples are living apart and commuting and seek as soon as they can to change that state of affairs) or elderly parent, to be able to see more of their adult kids or babysit their grandkids, because they want a job in a more affordable city where they can buy a house, because their head of school or dean is making their life a misery, and because they want to be closer to a fieldsite, co-author or collaborator. People who need or want to move should not be deprived of their intellectual property. The idea of date of acceptance is of course also profoundly arbitrary. Books can take many years to write and journal articles can be based on many years of fieldwork. Publications get started in one institution and finished in another. Heavy teaching and administrative workloads means that publications also get written in the evenings, at weekends and during annual leave. Imagine being told that the articles you wrote in your own time no longer belonged to you. Stern’s recommendations might terminate one kind of gaming but will lead to another. As a journal editor in the US, Neil Smith (2010) recalled receiving a phone call from a British scholar that urged rapid acceptance of a submitted paper “because our RAE submissions are due in two weeks” (RAE was the precursor to REF). In the future, scholars will be asking editors and publishers to post-date letters of acceptance so they can take forthcoming publications with them to a new position. And if you wrote the article in your own time, because 35 hours a week is simply not enough time to write and do other work, can an institution tell you that you can’t take it with you?
Equality, diversity and interdisciplinarity
The review seeks to enhance equality and diversity by insisting that all academics are returned. It also seeks to secure improved recognition for interdisciplinary research. But it instantly undermines these aims in two ways. First, it allows for a differential number of submissions per academic to be submitted. While an average of two per academic is the aim, some people could submit fewer (even potentially none which I am sure is no different than not being included) and some more, up to a maximum of six. Second, it calls for some measure of metrics to be added to the (secret[i]) peer review. I can see instantly how both of these measures could be highly negative, as a result of both gender and institutional location. I am a human geographer in a School of Geosciences, a school that must accommodate humanities scholars and geophysicists, critical social scientists and positivist ones. We have a College of Humanities and Social Sciences but we are not in it as the School of Geosciences is located in the College of Science and Engineering. The enormous differences in our epistemological and methodological approaches, not to mention in modes of writing, publishing, supervising, securing funding and co-authoring make REF particularly challenging. In the last REF, in order to maximize the financial return, the School submitted all academics to ‘Earth Systems and Environmental Sciences’ (Unit of Assessment B7) rather than to ‘Geography, Environmental Studies and Archaeology’ (Unit of assessment C17). Outputs authored by geographers were cross-referred to C17, but essentially we made a single submission to B7. But we all submitted four outputs so it was equalizing in that respect, even if as geographers we would have preferred to only have submitted to C17. If in the next REF, the School decides to submit once again to B7 and a differential number of outputs are allowed, we might find it is not only women and people of colour who are de-selected but scholars whose work is least like “earth systems”. It would certainly make the construction of a coherent narrative more straightforward if the work conducted by feminist scholars or queer theorists did not have to be included in the narrative. In other words, the serious risk exists that human geographers will be submitted at a lower rate than earth scientists or geoscientists. These inequalities as well as those of gender are likely to be exacerbated by the proposed addition of metrics to the existing system of peer review. A recent study (Writing for Research 2014) has shown that the citation rate in the natural sciences is six times greater than in the humanities. Other studies have revealed that men get cited far more than women. This is because men tend to cite mostly men, and women tend to cite both men and women (see Ahmed 2013; Ingraham 2016). Even in human geography, men get cited at five times the rate of women (of the 100 most cited human geographers on Google Scholar, only 21 are female). So if you were trying to maximize REF returns in B7, in a system where metrics matter, the article by a female humanities scholar with ten citations is going to look far less appealing that the article by a male geoscientist with 100 citations. The Stern review sheds no light on how these inequalities might be avoided. Maybe the new focus on interdisciplinarity will help scholars like me but I have no idea what Stern means by interdisciplinarity as no definition is provided. Is interdisciplinary work when a physical geographer works with a geologist (drawing on principles in geomorphology) or when a human geographer works with an anthropologist (drawing on feminist poststructuralist theory) or is interdisciplinary work when a positivist geophysicist works with a decolonial queer theorist? The first two examples are straightforward and commonplace theoretically and epistemologically, the final one is hard to imagine. Equality and diversity could be enhanced and the problem of interdisciplinarity minimized by allowing researchers to submit to the panel of their choice, as occurs in the New Zealand PBRF[ii], and then scores are aggregated at school or departmental level.
So Lord Stern, I feel totally underwhelmed by your intervention.
Ahmed S (2013) Making feminist points. feministkilljoys [blog] 11 September https://feministkilljoys.com/2013/09/11/making-feminist-points/
Bhandar B (2016) The Stern Review. London Review of Books 2 August http://www.lrb.co.uk/blog/2016/08/02/brenna-bhandar/the-stern-review/
Campaign for the Public University (2016) Let a hundred flowers fade … The Stern Review [blog] 29 July http://publicuniversity.org.uk/2016/07/29/let-a-hundred-flowers-fade-the-stern-review/
Cupples J and Pawson E (2012) Giving an account of oneself: The PBRF and the neoliberal university. New Zealand Geographer 68(1): 14-23
Ingraham C (2016) New study finds that men are often their own favorite experts on any given subject. The Washington Post 1 August https://www.washingtonpost.com/news/wonk/wp/2016/08/01/new-study-finds-that-men-are-often-their-own-favorite-experts-on-any-given-subject/?hpid=hp_hp-more-top-stories_wb-experts-1020am%3Ahomepage%2Fstory
Writing for Research (2014) Poor citation practices are a form of academic self-harm in the humanities and social sciences. Medium 27 October https://medium.com/advice-and-help-in-authoring-a-phd-or-non-fiction/poor-citation-practices-are-a-form-of-academic-self-harm-in-the-humanities-and-social-sciences-2cddf250b3c2#.zgh1cha9t
Morrish L (2016) A Stern talking to? Academic irregularities [blog], 28 July https://academicirregularities.wordpress.com/2016/07/28/a-stern-talking-to/
Smith N (2010) Academic free fall. Social Text 21 August http://socialtextjournal.org/periscope_article/academic_free_fall/
Wilsdon J (2016) The road to REF 2021: why I welcome Lord Stern’s blueprint for research assessment. The Guardian 29 July https://www.theguardian.com/science/political-science/2016/jul/29/why-i-welcome-lord-sterns-blueprint-for-research-assessment-ref-2021-stern-review
[i] I say secret because you don’t get to find out how your outputs were graded and what score you were awarded. Academics who tried to get their own scores through FOI requests were told to go away.
[ii] Before returning to the UK in 2013, I experienced three rounds of research audit in New Zealand under the PBRF (Performance Based Research Fund). The PBRF in many ways is an abhorrent, governmentalizing technology that like the REF subjects academics to disciplinary performance management regimes and heavy handed surveillance. It is however also a technology within which, as Eric Pawson and I argued, one can carve out subversive spaces of self-esteem and affirmation (Cupples and Pawson 2012). Since being exposed to the REF, I have to say I miss the PRBF (I can’t believe I am saying this), which is a fairer, more inclusive and more transparent system that overcomes some of the problems I have identified above and addresses many more. If we are to have research audit (and I wish we didn’t), it is a pity that Stern did not explore alternative audit systems elsewhere in the world. The PBRF is a better system than the REF for at least three reasons. First, rather than a system in which a unit of assessment/school narrative is authored in secret by a small group of academics, in New Zealand individual academics get to craft their own portfolios and write their own narratives. This means you can submit to the panel that most suits your work and don’t end up in a situation where human geographers are submitted to earth sciences panels as part of a school submission. You can be as interdisciplinary as you like and say so and cross refer yourself to a second panel. Second, you get to find out your own individual score. I know this is problematic in some ways, but in the REF you are asked to produce world class research without knowing whether your research in the last round was already world class. It is much more transparent than REF. Third, everyone is submitted, nobody is excluded on the basis of citation rates, gender or for any other reason, and you submit everything you published but nominate four outputs as being especially significant. It is OK not to have four if you are an early career scholar.