Are university ratings and rankings credible?

 

There is significant debate on the credibility of Malaysian universities and in particular the ratings and rankings used to evaluate them.

After a decade of trying the ratings and rankings show almost all Malaysian universities in the lowest echelons compared with their international peers, with the majority in the bottom 500.

The default to international systems, especially the easiest QS Ranking and their QS Stars Rating systems, suggests that the national regulations based on local criteria and performance metrics are inadequate to measure and signal quality or worse, that Malaysian universities have to buy their credentials from foreign commercial marketing companies.

The credibility of the QS Stars audit process is now in serious doubt with QS extending their ratings for two years without verification for a fee of US$33,000 (RM154,000). It already costs around US$55,000 (RM257,000) for the three-year baseline audit.

A university given five stars in 2020 based on pre-Covid data would still have five stars six years later no matter how devastating the Covid-19 lockdowns have been on international and local students, university finances or even the delisting of their programmes.

The audit standards also give rise to questions. A 5-star rating in the online learning category can be achieved using freeware from the internet. Universities can claim points for their facilities based on subscriptions to high-street fitness centres.

Audits are based on documents sent to junior analysts overseas with no direct knowledge of Malaysia. There are no site visits or similar verification events.

Problems in auditing were highlighted in a recent case where an internal QS investigation involving the founder Nunzio Quacquarelli and the CEO Jessica Turner unearthed multiple audit errors which were not corrected in the final assessment. Allegations of bullying and threats were also not addressed transparently.

There is a clear disparity between the ratings and the rankings. Two Malaysia universities are rated “5-Star Plus – Outstanding” benchmarking them among only 23 global peers but one is ranked below 600 in the QS World Rankings and the other has no ranking at all in the QS World or QS Asia rankings. The University of Bristol which is also rated “5-Star Plus – Outstanding” is ranked 55 in the World.

The founder of the formerly top-ranked Malaysian university complained publicly that QS changed its ranking methodology, described as their “largest-ever methodological enhancement”, without adequate engagement with universities which fell in the rankings as a result.

Growing international dissatisfaction with QS for the same reason caused the withdrawal of all 52 universities in South Korea and many other leading global institutions. Even Malaysian universities have stopped using it.

Ranking is a brand recognition measure with 50% of the score based on employer and academic surveys. Malaysian universities with strange acronyms or branded with amusement parks often do badly in these measures. There is no measure of student satisfaction or independent stakeholder verification.

The QS system is essentially a marketing tool with associated marketing collaterals and is big business. It is estimated that the ratings and the recent international conference has cost RM10 million at least in the last few years.

As a signal in the marketplace, ratings and rankings are now confusing students and parents and the consensus of international academic opinion supports this. A statement by the United Nations University this year roundly exposed the flaws in ratings and rankings.

As the commercial brands of the ratings sellers become damaged  so does the academic and commercial brand of universities associated with them. From an economics perspective chasing these ratings and rankings at a huge financial cost will likely prove a bad investment as the market signal first erodes and then turns negative. - FMT

The views expressed are those of the writer and do not necessarily reflect those of MMKtT.