What makes a college or university "great"?
We all debate endlessly university rankings or "league tables." And a recent e-mail exchange with colleagues from US and UK institutions started me thinking about how we determine an institution's quality.
We readily refer to "elite" universities and if asked to list, say, the top 25 schools around the globe most of our lists would overlap significantly, if not completely. You and I might disagree about whether a particular school should be in the top 5 or not, but I bet we'd agree on 20 of the names we'd list in our top 25.
But how do we "know" which schools are great? Along with whatever appears on the cover of US News or Times Higher Education, some combination of faculty notoriety, research output (including technology transfer), and admissions competitiveness play a large part.
And then of course there is the graduates' success. But while we can read the annual rankings, and measure research output, patents and citations, how do we assess alumni success?
Dan Guhr of Illuminate Consulting Group has mentioned to me that he sees alumni outcomes as a likely (if not inevitable) component of university ranking systems in the future. Any metrics around this would have to provide a quantitative (or rank ordered) proxy for categories such as
- alumni achievement
- alumni influence
- alumni network value
We can probably propose arbitrary benchmarks for these kinds of categories in the form of ranges and ratios. For example, if "PhD completion" is an achievement metric, then we can measure the approximate fraction of our alumni who have completed a PhD within some finite period after we granted their bachelor's degree. This is just an example off the top of my head.
One survey that has started down this road is the Professional Ranking of World Universities (PRWU), established by Mines Paris Tech (also known as the École nationale supérieure des mines de Paris). They've selected a single criterion for ranking the "professional effectiveness" of institutions: the number of alumni from a single school who are CEO (or equivalent) at the "500 leading worldwide companies" as measured by annual revenue.
They also quickly acknowledge the limitations of this specific model, but claim it is the only practical one that they can currently apply. The authors further admit that there is no way to measure, say, successful entrepreneurship among alumni:
The creation of companies by graduates could also be a relevant criterion to be taken into account but, once again, the data on an international level do not exist, and achievements (companies actually created) are difficult to compare.
This single criterion for business is no more flawed than the Jiaotong University Shanghai rankings', [link opens two windows] which award points for alumni receiving the Nobel Prize or the Fields Medal. But saying "our survey is equally flawed, but in a new and different way" is hardly a defense of a system such as the PRWU. (A flaw of the Shanghai ranking is that they consider the total number of prizes won by a school's alumni – without accounting for the size of the alumni population. At tiny Caltech the chances that a particular grad has won a Nobel are about 1:1300; what's the ratio at other research institutions?)
So an obvious problem with the overall approach of using alumni achievement is that it will be extremely difficult to assess the prevalence of a particular outcome for an institution with poor data about its alumni. In the long run, however, this could exert pressure on institutions to invest in improved data collection, internal data sharing and data maintenance.
The email exchange that started me thinking about alumni achievement as a ranking criterion did indeed move on to a discussion of internal data processes and the lack of will that many staff members exhibit when presented with a chance to reveal meaningful truths about alumni by sharing and analyzing data. Perhaps that will provide me with a topic for a future blog post.
Do you think alumni outcomes can and should be measured as a proxy for institutional success? What should we measure, and how?
Also: Wikipedia article about rankings with links to a number of such schemes and outlets
Image of university rankings from Flickr user Mathieu Thouvenin via Creative Commons.
