In this section, we tell you more about how SDU Analytics work with rankings and how we use the results to produce meaningful decision intelligence about the university’s performances. Hopefully this will give the reader insights to how SDU Analytics process ranking results and why is it important to understand the methodology behind it all.
Why is it important to understand the methodology?
The methodology behind each ranking can be difficult to understand and sometimes the scoring systems in the various rankings is not exactly transparent either.
But why is it even important to understand how the ranking measures performances and scores are calculated?
Overall, it is important to understand the methods if the ranking results are to be included as meaningful management information.
The purpose of ranking is to provide the right insight into what the university can do better. Therefore, it is important to be able to interpret ranking results correctly, having the methods in mind, which affects how the university's performance is perceived.
The advantage of using quantitative indicators is that they can be easily applied and compared across the university sector. In addition, the results in the form of a ranked list of universities, are relatively easy to understand. On the contrary, there may be restricting factors in the quantitative setup, which can be significant for the interpretation of a result or performance.
For example, the difference in weighting of the individual indicators. If the university performs poorly on an indicator that is highly weighted, it can impact the overall score, without it meaning that the university performs poorly on all the indicators.
As another example, there may be cases where the university's performance appears to be lacking according to the scores given, but it may not necessarily mean a lack of quality. Instead, the cause may lie in how the indicator is measured, or rather what is being measured. Some indicators are not doing a great job in capturing outcomes caused by the national context or educational settings.
How do we process ranking data and the result?
As explained above, it is important to know what exactly is being measured in rankings, especially the ones using submissions from the universities themselves. Understanding of methods thoroughly prevents the university from “missing out” on points that would otherwise be justified.
When reporting to the ranking agency it will always be a balance between optimizing the points the university may be given for its practices, but still being true to the meaning of the ranking indicator.
That is, one should not refrain from reporting matters that seem common in a national context. This can easily be high standards when comparing internationally. Conversely, one should not take advantage of the knowledge asymmetry that exists between the individual universities and the ranking agency.
Therefore, it is important to familiarize yourself with what exactly the ranking agency is interested in knowing.
In some cases, it is necessary to look at the intention behind the individual indicator, to avoid submitting information based on misunderstandings. In these situations, it makes sense to reach out to the ranking agency with clarifying questions. The ranking agency will mostly likely be able to help, and this will make the submission both better and easier in the future.
As mentioned above, rankings can be important tools for development. And if the results are evaluated in the right way it can be avoided that the ranking becomes the end goal.
This also means there’s a need to focus on the purpose and choose which institutions it makes sense to use for benchmarking in different aspects. At the same time, one should always bear the method and context in mind, to the extent that make sense, before deriving a particular conclusion based on a ranking result.
Therefore, used correctly, ranking can provide insight into both the university’s own practice and "best practice", providing the necessary drive and inspiration to go in the directions that increases quality, in a way that it makes sense for the individual university.