The 2015–2019 VQR, Italy’s Third National Research Assessment: an intended informed peer review or a DIY bibliometric exercise?

Aiming to enhance university performance and maximize public investment returns, many governments conduct national research assessment exercises to allocate core funding based on research quality, in turn measured by ‘informed peer review’, i.e. combining quantitative indicators with peer reviews. This paper examines Italy’s latest VQR, intended to be based on such an approach, in STEMM fields. We analyze the final report of the 2015–9 VQR exercise and metadata of products submitted by universities to address key questions about the role of metrics in informing peer reviewers. In STEMM, 99% of submitted research products are indexed in bibliometric repositories; the few products not indexed in WoS/Scopus obtained very low quality scores. On average, each reviewer evaluated the originality, methodological rigor and impact of 600 products. Correlating the VQR outcomes with those of a similar purely bibliometric exercise based on an experimental dataset reveals the predictability of the outcomes of the ‘official’ exercise, although ‘locally’ there are important differences between the two rankings. We provide insights for reflections that are all the more urgent, given that the next Italian research evaluation exercise is already underway.