Evaluating scientific journals: properties, limits and conditions of effectiveness of classification methodologies

During last years a large debate was developed among academics about research evaluation procedure. Approaches proposed in literature considers quantitative and/or qualitative aspects of each journal but each methodology has limits and characteristics high heterogeneous. The meaning and implication of results achieved with a ranking procedure is strictly influenced by the approach selected for the journal evaluation. The paper presents a literature review of the main qualitative and quantitative approaches proposed for journal ranking focusing the attention on the main differences of approaches. The study is completed with an empirical analysis on the database Thompson Scientific, one of the main provider of quantitative rankings. The analysis considers the characteristics of quantitative rankings proposed, look at the qualitative characteristics of best and worst ranked journals and compares results obtained with those achieved by an international qualitative survey (Harzing database). Results obtained demonstrate the low degree of coherence of ranking based on different approaches and point out some risks related to the use of these approaches to evaluate research.