The Inexact Science of Research Assessments and Why Macroeconomics Should Worry About It

by Danielle Guizzo and James Walker

It is often said that academics and university research managers are obsessed with research rankings. Some see this as a malicious development; tilting academia towards managerialism, undermining academic integrity and leading to a stale research culture. Others say that rankings are critical to efficient decision-making about research quality, such as promotion and hiring, workload arrangements and allocating resources.

Indeed, it is rare to find an academic who does not have a strong opinion! But we need to go beyond anecdotes and really understand the mechanisms by which research evaluations might affect the working lives and behaviour of academics.

We are delighted to lead a Study Group for Rebuilding Macroeconomics to understand the impacts of the Research Excellence Framework (REF) on macroeconomics. We will analyse how the REF affects the production of knowledge in macroeconomics and whether or not it contributes to the ESRC’s question of a possible ‘monoculture of ideas’. We are interested in how it might influence working practices among economists at different stages of their career, as well as across different economics departments and higher education institutions.

Institutions, research clusters and individuals are often assessed by the quality and impact of their research, such as by peer-reviewed journal articles, scholarly books, book chapters and case studies. This is normally by governmental bodies or specific agencies at a national level, using rankings and citation metrics. The main objectives are two: first, to boost the country’s research performance and improve the quality of knowledge production; and second, to provide public accountability to taxpayers for the nation’s research spending.

The UK is at the vanguard of research evaluation. This follows some intense scrutiny during Margaret Thatcher’s era regarding the efficiency, accountability and value for money of British universities. The Research Selectivity Exercise (RSE), created in 1986 and administered by the University Grants Committee, was the first full-scale national exercise which aimed to base funding decisions on a wide-ranging assessment of the quality of their research.

After many criticisms regarding its criteria and metrics, the RSE was replaced by the Research Assessment Exercise (REA) in 1989, remaining until 2008, when it was replaced by the Research Excellence Framework (REF).

Instead of relying mostly on journal ranking lists or citation metrics, the REF uses an expert peer-review tool for assessing research strength. It assigns expert sub-panels across different disciplines, who evaluate research outputs produced by academics, including the impact of these outputs and the environment in which academics work with publications.

The REF poses several issues. For example, in terms of status of the field one potential indicator of institutional influence with respect to status and resources, may be the number of entries submitted to the expert panel (in economics the unit of assessment being Economics and Econometrics). On this measure, Economics does not appear to be doing well. The number

of submissions in Economics and Econometrics has fallen from 60 in 1992 to 28 in 2014, and will fall further in the upcoming 2021 evaluation.

Does this mean that there is less work being done in economics? Not necessarily.

In the REF, economic outputs can be submitted to a number of different sub-panels (such as Economics and Econometrics, Anthropology and Development Studies) and then distributed to the panel where expertise resides (i.e. Anthropology and Development Studies). This partially reflects a redistribution of departments to form or support business schools and management centres, but also that individuals’ entries in economics departments are sometimes being distributed elsewhere, such as in business, management, geography and environmental studies, agriculture or development.

A reason for this strategy may be that institutions are tactically trying to maximise their overall outcome and seem to consider that submitting economics to the economic unit of assessment to be a risky business.

And they seem to have a point. Being ranked 10 in Business and Management, where there are about 100 entries, puts that department in the ‘top 10%’. However, an economics department with an identical rank of 10th out of 30, and thus ranked in the ‘top third’, may not maintain the same gravitas with Vice Chancellors who often want to claim to be a “top institution”.

The outcome may be that it is more difficult to fulfil resource requests and, as has been the case in the past, that economics departments get absorbed into business schools or other departments.

At the more micro level, evidence suggests that institutions’ strategic approach to choosing the outputs that academics produce might also play an important role. Because the REF is retrospective and aggregated to department level, it provides an incentive for indirect proxy measures of research quality.

For example, more than 90% of economists working in business schools perceived that their school used the journal rankings “to decide which individuals to submit to the Research Excellence Framework (REF)”, whereas others noticed a substantial shift in submissions to 4-rated outputs as institutions attempt bolster their submissions.

As a consequence, this ‘one size fits all’ logic, research assessment conditions the activity of academics by suppressing the diversity of topics and methods thereby limiting innovation and critique. In the case of sub-disciplines such as macroeconomics, this can significantly narrow the breadth of knowledge, limit the type of research questions asked and methods used.

Another possible factor relates to the characteristics of academics and how they interact with their institution. There is growing evidence that individuals with different characteristics - be it gender, where they undertake their PhD training, or point of careers (such as being an early career scholar) – perceive and engage with research evaluations differently.

For example, in some top institutions to obtain academic rank in economics departments, it is common for probation requirements to include expectations for publications in specific grades of journals, such as one or more ‘Top 5’ articles plus several leading field publications. Papers

in ‘Top 5’ for junior faculty are also a matter of career sustainability. Failure to deliver in the early years of one’s career could lead to a downward shift to lower-ranked institutions.

To get to the bottom of how the REF might impact macroeconomics we are carrying out a multi-disciplinary, multi-source, mixed-method study, combining quantitative and qualitative data collected from a large-scale survey of academic economists, in-depth interviews with a purposive sample of macroeconomists, and publication data.

Given the extent of criticism the REF has faced in the past (see Lord Stern’s review in 2016), unpacking its structures and influences might be critical to reshaping macroeconomics. If you have suggestions for other incentives we should consider in our Study Group, please do reach out to us. We would be delighted to hear from you.