On the agreement between bibliometrics and peer review: evidence from the Italian research assessment exercises

Two experiments for evaluating the agreement between bibliometrics and informed peer review -- based on on two large samples of journal articles -- were performed by ANVUR, the Italian governmental agency for research evaluation. They were presented as successful and warranting the combined use of bibliometrics and peer review in research assessment exercises. This paper aims to analyze in full the two experiments and to draw the definitive evidence from them. First, we have provided the correct design-based environment for the inference, since data were collected by means of stratified random sampling. Thus, the design-based estimation of the weighted Cohen's kappa coefficients and the corresponding confidence intervals is developed and adopted for assessing the agreement. In both the experiments, the upper bounds of the confidence intervals for the weighted Cohen's kappa coefficients are smaller -- in most cases strictly smaller -- than 0.40 (a threshold indicating at most a weak agreement) for each scientific area and for the aggregate data. Therefore, given such a low level of agreement, it is likely that the combined use of bibliometrics and peer review might have introduced uncontrollable major biases in the final results of the Italian research assessment exercises. In addition, as to the second experiment, we have also addressed the problem of missing proportion homogeneity between the scientific areas. We have assessed that the data are missed with unequal proportions between the areas -- a further drawback which may invalidate the conclusion carried out by ANVUR. Hence, from the point of view of the academic discussion about the agreement between bibliometrics and peer review, this paper documents that the two ANVUR's experiments do not bring a valid contribution to the debate, since they were designed in a largely unsatisfactory manner.