A Marketplace of Economic Ideas

by Carlo D'Ippoliti

Citations are the new metric in the marketplace of economic ideas. They are part of parcel of meeting the requirements for promotion in most major economic research departments. Some young researchers may end up conforming to whatever research agenda brings them as many citations as possible. But are citations really a measure of scientific quality, or might they instead reflect personal and professional connections?

Some economists like to think of any social interaction, from the division of household chores in the family to organ donations, and from organized crime to finding a sexual partner, as if it was a market. Actually, as if it was a competitive market: there are no price leaders; no barriers for competitors to entry or exit the sector; no problems for anybody to know everything about the product; who offers it, who is willing to buy, and at what price.

Science is no exception – including economics. One could think of it as if scientists (or universities) competed in the production of knowledge. But there is a catch. Except for the vehicles of dissemination of this knowledge (books, journals, but increasingly also blogs or papers posted online), there is no price to pay for accessing the product in this market. In some sectors, patents allow keeping valuable information private, but in most cases, and certainly in the case of economics, everybody can easily know whatever is written on a certain topic.

Therefore, to a large extent, economics knowledge is what economists call a public good: there is a cost for producing it, but there are no customers willing to pay a price for it.

This does not mean that there can be no market: only that this market is special. People who think in these terms often say that it could be the students who pay a price for economics teaching, and the scientific community itself could give a value to economics research.

This latter value is often taken to be citations in the scientific journals. Somebody who is highly cited in the economics literature will surely be saying something of interest, so the thinking goes. So, for example Daniel Hamermesh concludes: “Fortunately, the market for economists implicitly recognizes the importance of citations, since they help determine economists’ pay”.

Other sciences, however, have since long abandoned the idea that citations really measure scientific quality only. After all, citations are correlated with the number of authors, the year, language and kind of publication, the reputation of the journal, the number of pages, and even the title length and the fact that the title does or does not contain an hyphen!

Citations also incorporate bias and discrimination against authors, for example in terms of age, field of specialization, and gender; not to mention the incredulous lack of self-awareness in excessive lists of self-citations, citation inflation (increase in total number of citations over time) or the fact that citations are very unequally distributed, so that averages and mean values… mean very little.

This is why when evaluating the quality of economics research within the UK’s “Research Excellence Framework” (REF, on which funding of universities heavily depends), the panel in charge of assessing economics research “will make use of the [citation] data supplied by the REF team where it is considered appropriate as an additional piece of supplementary evidence to support the initial assessment of outputs, not as a determining factor”.

Reality may be different though. Previous research on Italy and the UK has shown that economics referees tend to consider citations even when they are not asked formally to do so. This creates a perverse incentive for economists, especially younger ones or those at risk of discrimination, such as women or minority scholars. They have an incentive to conform to whatever will bring them as many citations as possible or perish trying.

The citations game has a dramatic impact on the content of the economics debate. Research evaluation may seem a narrow and dry technical debate for academics who must look after their own pockets, but economics has of course a much wider impact on policy and society. That the economics debate takes place in an open and level field should be a political concern for everybody.

In our project for Rebuilding Macroeconomics we will look at how personal and professional connections among fellow economists determine who they cite, and who ends up unheard of in the discipline. We hope to show that citations indeed measure scientific quality to some extent, but also many other things that we summarize as social connections. The use of citations as the measure of quality in the marketplace of ideas may, in fact, be misplaced.

If we are correct, we must find better ways to ensure that everybody pursues research that they consider will be most rewarding across a number of measures and not just our own networks. Alas, this is far from what happens today.