Event Details

Why AI cannot assign Quality Scores to Mathematical Sciences Journal Articles

This talk will summarise the results of a project to investigate whether machine learning techniques could replace or support the peer review process currently used to evaluate journal articles submitted to the Research Excellence Framework (REF). For this, the REF team gave us provisional REF2021 scores from March 2021 for 148,977 journal articles from all 34 Units of Assessment (UoAs) to assess the accuracy of various machine learning methods. We consulted with REF2021 assessors about the results through a series of focus groups and produced a set of recommendations and additional analyses of the scores (http://cybermetrics.wlv.ac.uk/ai/). This talk will summarise the methods used (inputs, algorithms, strategies) and explain why we did not recommend any strategy, even the one that was technically capable of improving REF2021 score accuracy. The talk will also investigate UoA 10 Mathematical Sciences in more detail, comparing it with other UoAs in terms of the accuracy achievable, the (limited) role of journals and the (limited) value of citation counts as quality indicators.