Multi-criteria benchmarking: ARWU 2020

Published: 25-05-2021| Version 1 | DOI: 10.17632/7cbsmm5sx9.1
Jean-Philippe Hubinont,
Yves De Smet


This paper proposes a novel framework to support the generation of strategies for multi-criteria long-term improvement. It can be applied to general preference models but it is illustrated in this article on a Multi-Attribute Value Theory model. The novel contributions to the literature are twofold. Firstly, the framework addresses the issue of resistance to change that may arise during the implementation of a strategy. It constrains a step of improvement to be focused on a single criterion, and minimizes the intensity of operational changes. Secondly, it addresses the realism of the improvement scenarios by treating three types of structural dependencies differently: the positive synergies, the negative synergies and the bottlenecks. The scenarios are generated by finding a set of efficient solutions to a shortest path problem in a graph whose edges represent possible steps of improvement. Each edge is characterized by an increase of rank or level and two penalty functions relating to the difficulty of its execution, one representing a risk associated to bottleneck mechanisms and the other to operational change relative to a previous edge. A case-study using the Shanghai Academic Ranking of World Universities is presented in order to illustrate how this framework could be useful to generate a sequence of strategic actions for the Université libre de Bruxelles. The framework based on multi-criteria benchmarking in order to help generate strategic action plans developed in the article (J.P. Hubinont & Y. De Smet, Long-term Multi-Criteria Improvement Planning - Decision Support System) is provided as an executable file : "MCDMBM.exe". It should work fine on Windows OS 64 bits. The case-study presented in the article uses parameters values (contained in a configuration file: "Article_cfg.txt") and a data set (ARWU2020.xlsx). It is possible to load the configuration file into the software with the menu :File>Open. Then one can go in the tab 'Paths' and click on 'Generate paths'. It allows the user/scientist to reproduce the results of the article in a few clicks. Moreover, the python files used to generate the executable files are also provided for any scientist to be able to adapt it and be helped in its research work.


Steps to reproduce

The software steps are the following: 0) Launch the MCDMBM.exe file. Select the only option available for the moment: MAVT model. 1) Load an .xlsx file whose format follows that of ARWU2020.xlsx and choose a folder to save results 1.bis): In order to reproduce the results presented in the article: Click on File > Open, and select "Article_cfg.txt". It will automatically load the parameter values used in the article. Go directly to step 4) 2) Select which alternative must be improved, the direction of optimization of each criterion and transform the performances into partial value functions using a simple bisection method. 3) Parameters tab: Select a statement about the penalty weights used in the article, insert the worst and best performance reachable as well as the scaling factors for the aggreagation of the partial values. 4) Click on generate paths. Depending on the size of the graph generated, the process can take a while. A progress score (in percentage of the total process) is printed. 5) The scenarios can be observed in a table as well as graphically. 6) A visualization of the optimality of each efficient path is made available in function of the penalty weight values. 7) It is possible to save a config (the parameters values) into a text file to be able to load it another time. 8) The software is made to be interactive. Hence, when paths of improvement are generated, it is very easy to change one parameter's value or another and re-generate the paths to study the deviation of the outcome. Note that if the graph is too large, it might take hours to produce results. In such case, we advise to tighten the range of possible reachable performances and revise the set of observed alternatives to keep only comparable alternatives which could be potential sources of inspiration. For instance, while studying (in the article) the strategy of Université libre de Bruxelles in order to improve in the ARWU 2020, we considered only universities from Western-Europe that are sufficiently similar to ULB. We did not consider universities with a very large tuition fee for instance. Moreover, we advise to save a config file when all parameters are typed in just before clicking on 'generate paths'. It allows the user to shut down the software if it takes too much time and recover all the parameters by loading it after reopening the software.