Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework

We use the UK's 2014 Research Excellence Framework (REF) to study the attributes of top-scoring (four-star) publications in Economics and Econometrics. Although official documents contain aggregate scores for each institution, we show how these aggregates can be used to infer the score awarded by REF panellists to each publication. We demonstrate that this score responds to journal prestige as measured by the Thomson Reuters Article Influence Score. Several econometric analyses confirm the limited contribution of other publication attributes, such as the citation counts, to the awarded score, and publications in the top generalist and top five economics journals are awarded four stars unambiguously. We conclude that in large-scale evaluations such as the REF, peer reviews and bibliometrics should be viewed as complementary modes of assessment; the time-consuming task of peer reviews would be more cost-effective if targeted at publications whose quality cannot be classified unambiguously using bibliometrics.