Lord of the Rankings

00:00:15
Speaker 1: Pushkin. Once upon a time in the middle of the nineteen eighties, when the Cold War still raged and a man named Ronald Reagan occupied the White House, there were three major weekly news magazines in the United States. Time Magazine was number one, with millions upon millions of copies in circulation, Newsweek was number two, and What News Magazine came in third place, a sleepy little publication based in Washington, DC known as US News and World Report. One of the magazine's managing editors in those years, Peter Bernstein, remembers the day when everything changed.

00:00:58
Speaker 2: It was a very exciting time. I mean, because Moore had just taken over. There were a lot of people buzzing.

00:01:05
Speaker 1: Around mort being Mortimer B. Zuckerman, a Canadian billionaire who bought US News and Will Report with dreams of becoming a media baron.

00:01:15
Speaker 2: There are a ton of really good ideas sort of in the magazine, and I think it's fair to say our feeling was that the execution could have been a lot better. And one of the things we wanted to do was distinguish ourselves from Time and Newsweek, because we were the third.

00:01:33
Speaker 1: And how do you distinguish yourself if you're always coming in third. If a Canadian billionaire with visions of grandeur is breathing down your neck, you do something audacious. You create something that no educated parent could possibly ignore, a scheme to examine every American college and university and rank them from top to bottom.

00:01:59
Speaker 2: We were looking to reinvent the magazine, and the college rankings became part of that.

00:02:06
Speaker 1: So Bernstein and his colleagues began collecting a mountain of statistics on the universities and colleges of the United States. To make sense of those numbers, they came up with an algorithm, a secret algorithm that could assign every school a score of one through one hundred, regardless of its size, or mission or complexity, an algorithm to capture the elusive quality of academic excellence.

00:02:31
Speaker 2: Finding those points of data and putting them into a ranking, as you can appreciate, is a complicated task. Yes, full of value, judgments, full of problems, challenges, dangers, But we were undaunted and precede it.

00:02:55
Speaker 1: I love that you're laughing about this.

00:02:56
Speaker 2: It was a very heady time, and we thought, and we did a rarely challenge time in news. We get their game.

00:03:06
Speaker 1: The rankings became an institution. This news surged, careers were forged, fortunes made. I mean, what's fasci today is that time and newsic have basically disappeared. The rest of US News has basically disappeared. But the rankings are as strong as ever.

00:03:22
Speaker 2: Well, it's nice to know in this world of effemera that something can last. But and as a journalist, you know who you know when your work is used for for old fish after a day, it's nice to see that something hasn't impact for a while.

00:03:44
Speaker 1: Peter left US News long ago, but he and his colleagues had added something new to the American academic calendar beyond freshman week, finals commencement. There's now the day the new rankings come out.

00:03:57
Speaker 3: The new thirtieth edition of Best Colleges was just released this morning, and we've got it.

00:04:02
Speaker 4: You're seeing the two thousand.

00:04:03
Speaker 5: And the list for twenty nineteen was Princeton University, followed by Harvard Tied.

00:04:09
Speaker 6: Scholars of America's best colleges have been revealed. The US News and World Report is out with its angle list, and we've got the inside scoop on how.

00:04:17
Speaker 1: The algorithm that's developed is undergoing a constant state of evolution over the years. How often are you tweaking it?

00:04:24
Speaker 2: I thought you might ask about this. I mean I did go online and look at it, and I was glad to see they are tweaking it on a regular basis, at least it looking if you look at their methodology. And of course they've also been reacting to competition. I mean, you know a lot of people have been critical of the rankings and whether this mythology really this methodology?

00:04:48
Speaker 1: Excuse me, that's it, Peter. That is like the definition of a Freudian slip. You're talking about your you're talking about your brain child, and you refer to it as mythology.

00:05:01
Speaker 2: That's true. I couldn't have come up with that one consciously.

00:05:06
Speaker 1: At what point did the US news methodology become a mythology? Let's find out. My name is Malcolm Gladwell. You're listening to my podcast, Revisionist History about Things Overlooked and Misunderstood. In this episode, I ask, how is it that a promotional gimmick from the nineteen eighties turned into a fetish object for American higher education? How do these rankings work and why does everyone take them so seriously? We're going to launch a revisionist history investigation, one that will take us to all kinds of strange places, to the bond markets, to Portland, Oregon, to New Orleans. We'll meet a mysterious college administrator who asked to be identified only as Dean, as well as a band of hackers who agreed to join our cause. All right, all right, so we're ready to go this very now. Before we start, can both of you guys introduce yourselves.

00:06:15
Speaker 5: So I'm Kelly McConville, and I'm an assistant professor of statistics at Reed College.

00:06:21
Speaker 4: I'm playing too, and I'm an international student who is currently studying at UNC's High school.

00:06:29
Speaker 1: So these are some of my accomplices, and here's why I needed them. Us News will tell you what variables they use to determine whether a school is good or mediocre, but they won't disclose exactly how they combine those variables into a score. I mean, why would they. It's their secret Sauce they charged forty dollars a year for premium access to their findings. This is a big business. So I figured my only chance at really getting under the hood of the ranking algorithm was to find out if anyone had in effect hacked the algorithm, and it turns out someone had. It was a group of students at Reed, a small liberal arts college in Portland, under the leadership of their professor Kelly mcconvill. Tell me how on earth this whole project of looking at US News rankings? How did it begin?

00:07:21
Speaker 4: So?

00:07:21
Speaker 5: I teach a course called Statistics Practicum, and in that course, the students complete a semester long empirical research project, and I solicit those projects and the data from other READ faculty and staff.

00:07:36
Speaker 1: Every year, US News asks for a lot of data from schools, but the Read administration had stopped supplying that data. They decided they didn't believe in the rankings anymore. But still the magazine kept generating a ranking for Read anyway, and someone in the Read administration came to Kelly McConville and said, ever since we dropped out, our US News ranking has plummeted. We think they've been punishing us. Can you double check their math?

00:08:03
Speaker 5: That was our task. We were tasked with recreating the model and then seeing what our ranking would be if we did supplier data.

00:08:11
Speaker 1: Yeah, So tell me, the two of you walk me through how does one go about conceptually trying to answer this question.

00:08:19
Speaker 5: They use a model that's called a weight in some model that means they take about twenty or so different factors about a school, like class size and graduation rates. They wait each of those factors based on how important they think the factor is, and then they sum up all those pieces and then calibrate it so that the highest scores at one hundred, and then based on those scores, they rank as schools.

00:08:47
Speaker 1: The Red team had to build their own version of the US News algorithm from scratch, trying to get as close to it as possible.

00:08:54
Speaker 5: We grabbed all the variables we could from open government sources, and we built a model that isn't trying to recreate the US newswait in some model. Instead, it's just trying to predict the US New score of a school with a high degree of accuracy.

00:09:13
Speaker 1: Whying how close did you get?

00:09:15
Speaker 4: I'll say fairly close. The only school that's outside of like a reasonable range of predictive score in the test status read oh really, And all other schools are like within a reasonable prediction interval of their scores.

00:09:33
Speaker 1: Meaning the Reed students identical twin algorithm is so good that when they run their rankings of American colleges. Their list looks almost exactly like the US News list, with one exception Read College. Wait, so they were messing.

00:09:49
Speaker 4: With you based on our finding, it seems reasonable they this something.

00:09:56
Speaker 1: In twenty nineteen, US News put Read in ninetieth place. Where do you think you should be ranked?

00:10:06
Speaker 4: Thirty sixth Oh?

00:10:09
Speaker 1: Wait? What that much of a difference? Yeah, that's bananas. Oh my, oh, they really did a number on Reed. Now we don't need to go much further into the ensuing beef between Read and US News. The magazine calls the undergraduates analysis quote inaccurate. But just imagine my joy when I ran across Swaying's analysis. Here. I was searching for some way to understand the secret algorithm cooked up by Peter Bernstein and his pals back in the nineteen eighties, and lo and behold, a bunch of undergraduates at Reed College had cracked the code. And were they motivated to help me? Absolutely? Because US News dissed their school. My first question to Kelly and her team was where should I start? And they said start with a little something called the peer Assessment score. Can you give me a more precise sort of statistic sense of just how important is it?

00:11:12
Speaker 4: Like, okay, So we find that the pure assessment score has the largest effects, So it has a coefficient of like six point five eight. In other words, like, if you hold everything else constant and you like increase the peer assessment scorer by one unit, your overall score will increase by six point five to eight.

00:11:39
Speaker 1: The peer assessment score was at the heart of the US News secret sauce, a measure of reputation, one of the most powerful elements in the whole ranking system. So if I'm a school and I'm anxious to move up the US News rankings, the thing I should really be focused on is trying to boost my reputation score.

00:12:02
Speaker 4: Yeah.

00:12:02
Speaker 1: Yeah, I had my marching orders, and I knew where I had to go. Next to the guardian of the US newsgates that dragon at the mouth of the cave. Robert Morse. Robert Morse, the man who runs the US News rankings, is an older man, white hair, very preppy, nice, if a bit lugubrious, heavy hangs the head that where's the crown? I called him up. I wanted to understand the mysterious variable that the read College hackers had told me was so important? Describe again, how is the reputation score generated?

00:12:52
Speaker 7: So we send out three surveys per school, and then they respond. They rate the schools on a scale of five to one or no response.

00:13:04
Speaker 1: So who are the three people at each school who are asked to fill out the survey.

00:13:09
Speaker 7: In the provos and the enrollment manager or the head of admissions.

00:13:15
Speaker 1: The surveys arrive via email once a year to everyone who matters in American Higher ed. If you're at a liberal arts college, you're given a list of all two hundred and twenty two other liberal arts colleges in your category. If you're part of a big university, you get a list of the three hundred and eighty eight other big universities in the United States. Your job is to rank the schools on your list on a scale of one to five, five being amazing, one being there's a serious problem here. And on what basis are people making those judgments?

00:13:50
Speaker 7: Well, that's a good question. I mean, we ask them to base it on their view of the school's reputation for undergraduate academic quality.

00:14:00
Speaker 1: But I mean, how would you know.

00:14:03
Speaker 7: I mean, we believe that to aggregate sum of all the raiders who are a leader is in higher education. You know their presidents and provosts and mission deans. So they're not rookies. When you aggregate their views at any given time, they're representing where those schools stand in the marketplace.

00:14:25
Speaker 1: I mean, so it's it's an assessment of their feelings what other schools.

00:14:33
Speaker 7: Well, they're leaders in higher education, so it's based on their knowledge of the other schools that they've gathered or through meetings or for exposure. Do they necessarily know a lot about all the schools they're rating. I don't know.

00:14:52
Speaker 1: If I rate a restaurant on Yelp, it's because I've eaten there, right, yes, But here, when I rate a university on the US News ranking, it's not because I've attended that university or taught there.

00:15:09
Speaker 7: It's just, well, you may have. We shouldn't say that you didn't. But you're saying, have the people who rated the schools been on every campus or taught at that school or no? In depth their course catalog?

00:15:23
Speaker 4: No.

00:15:23
Speaker 7: Do some of them have greater degrees of knowledge about the schools they're rating, yes? Do some of them have lesser degrees of knowledge? Yes? So I think it varies from having substantial knowledge to far less knowledge.

00:15:39
Speaker 1: In my naivete, I had imagined that there would be some level of rigor to this crucial variable in the US news wait, in some model a methodology that I could study, analyze, learn from. But Morse wasn't being very helpful, so I tried to draw him out. I mean, I was fascinated that there's a little clump where BYU Bring them Young, Yeshiva and Gonzaga are all ranked right in the same roughly equally in the National University's rank. So a Mormon school, Conservative Jewish school, and the Jesuit school. And I was trying to think, what is the rabbi who runs Yeshiva know about bring them Young? And what does the Jesuit priest who runs Gonzaga know about Yeshiva? And it's a good point, like is Gonzaga giving George ten of five and giving Boston College a five?

00:16:36
Speaker 7: And we're not looking at how people are rating other schools. We are doing a trim means, so we take off a few of the highest ratings and a few of the lowest ratings in an effort to eliminate some of the strategic voting that you're alluding to well, I.

00:16:56
Speaker 1: Don't know if that's strategic voting. I'm wondering if it's just that. If I'm president of Gonzaga, I'm powerfully motivated to believe that the Jesuit education is the best education out there. So when I look at Georgetown, I think they're doing the right thing. And when I look at Yeshiva, I say, where's the New Testament? Why are they not using the New Testament? I mean, this is just natural human biases. I just don't know what they have to do with the quality of the institution.

00:17:29
Speaker 7: Well, I don't have a different answer.

00:17:35
Speaker 1: Devoted listeners of revisionist history. Well, remember how years ago I fell for Rowan University in New Jersey. I love Rowin, But Rowan is ranked one hundred and eighty seventh in the National University's list dismal mediocre, and why because its peer assessment score is a forgettable two point four? How on earth have the University presidence of America come to the collective decision that they hate Rowan University way way down in South Jersey. I call up Ali hush Mind, the school's president. How many university presidents have visited Roan in the last five years from other colleges.

00:18:16
Speaker 8: Oh, two or three, one from Canada, one came who was part of an accreditation and I believe another one was a friend.

00:18:26
Speaker 1: That's it. Yes, So when they're giving you a score, on what basis are they using forgiving you? How do they know anything at all about Roan?

00:18:35
Speaker 7: Again, I don't know.

00:18:36
Speaker 8: I can only speak for myself. I can assure you that I really don't know much about the institutions that I'm asked to rank or score.

00:18:45
Speaker 3: I just don't have enough tools.

00:18:48
Speaker 1: Now. It's not like university presidents don't know anything about grading. They were all teachers themselves once. But when it comes to grading schools, it seems like they might just be mailing it in. Next, I called up the head of admissions at a major university. You've heard of it, trust me, but I can't tell you what it is. And we altered the voice to protect this person's identity. Of course we did. We had to look how US News treated read college. My source told me to address them only as dean.

00:19:20
Speaker 9: I am certain and I am far more qualified to be a member of the Baseball Writers Association of America and vote on the Hall of Fame than I am to assess other institutions for the US News and World Report.

00:19:35
Speaker 4: Yeah.

00:19:35
Speaker 9: Yeah.

00:19:36
Speaker 1: I proposed a Dean that we do an exercise. I'd name a school at random, and then the two of us would try to peer assess it. Florida State. Never been there. I've not been there, I think three point two. Syracuse. Yes, I I stopped for coffee there once in a way driving back to my mom's house in Canada. I don't know if that counts. I don't think it does. I think I think it counts account experience. I got to say experience the weather, but there's no there were no students there. It was Christmas. But I will say this, my cousin went to Syracuse and had a good time.

00:20:19
Speaker 9: Oh h and so you would rate it based on that.

00:20:23
Speaker 1: Let's give Syracuse some love. Three point nine, No, four point zero. University of Washington in Seattle. Yes, you've been there, Yes I have. What'd you do when you were there?

00:20:38
Speaker 9: I walked around?

00:20:39
Speaker 1: I think that counts.

00:20:41
Speaker 7: It counts.

00:20:42
Speaker 1: I said to Dean, you can see the mountains from there, the space needle. Let's go big. University of Washington at Seattle four point two. Have you ever been to William Mary.

00:20:56
Speaker 7: I have yeah.

00:20:58
Speaker 9: I love history, I love colonial Williamsburg.

00:21:02
Speaker 1: Do you remember what score you gave William and Mary.

00:21:06
Speaker 9: I probably gave them a four or five.

00:21:09
Speaker 1: Really you went for five. It's the second oldest college in the US. Do you think that affected your judgment on that?

00:21:17
Speaker 9: I'm sure it did.

00:21:19
Speaker 1: Dean and I went on for quite some time. We had a long chat about Southern Methodist University in Dallas and whether or not the great football team the school had in the nineteen eighties should count in their favor. Dean said no. I said yes. Now was all this a helpful exercise? Not really? But then again, now that was my conversation with the Wizard of the Rankings, Robert Morse. One suggestion that has been made to me by people who do the rankings is that the thing they rely on to make their reputation scores for other universities is the only source they have of reputation scores for the other universities, which is the previous year's US News rankings. Is this circular?

00:22:05
Speaker 7: If they're telling you that, then that's not our expectation of how people are doing their ratings. Versus looking at the ranking table and looking at the score. We believe that there's more thought going into it than that, but we haven't done any kind of social science research to prove or disprove that point.

00:22:28
Speaker 1: You haven't done any right, Yeah, they haven't done any social science research. Only one person gave me much insight into the mechanics of the peer Assessment score, and that was Ali Huschmand, president of Rowan. His advice was, you have to take matters into your own hands. So what does he do? He makes his own hot sauce. And he told me that Rowan sends out free samples to every college president in the country. That's not too really he did, and he.

00:23:01
Speaker 8: Did no that I make hot sauce. Well, that would have been that's scoring.

00:23:08
Speaker 5: I hope.

00:23:09
Speaker 1: Yes. I want to know about your hot sauce. What kind of hot sauce do you have?

00:23:13
Speaker 4: Oh?

00:23:13
Speaker 8: I make serious, serious hot sauce. But I actually grow from seeds all the way to the packaging. And there are three levels of heat. The lowest one is called Ali's nasty, which is.

00:23:24
Speaker 4: Kind of mind.

00:23:26
Speaker 8: And then the one next to it, it happened, was as a heat it's called a nasty delicious, And then the one with Carona Reaper is called nasty vicious, and I will send you a package the nasty divisious can get you.

00:23:41
Speaker 1: But this isn't gonna help your cause. This is gonna hurt your cause. If you're if you're giving some some university president in the middle of the Midwest, it's like it happens to have a little bit of vicious. They're gonna they're gonna curse your name.

00:23:54
Speaker 8: Probably will, but I but I figured that no, no publicity is about bad publicity.

00:24:01
Speaker 1: Did I try the nasty vicious?

00:24:02
Speaker 2: Oh?

00:24:03
Speaker 1: Yes, I did, on a scale of one to five five point oh. In the middle of my investigation of the pure assessment score and its many implications, I had a conversation with a professor at Duke named Bill Mayu. Mayu is friendly with some bond traders, and he was talking with some of them one day when they mentioned that they were having difficulty with bonds issued by historically black colleges HBCUs as they're known.

00:24:38
Speaker 3: If you're a bond trader, you worry about moving the bonds, that you have an inventory, and they just mentioned, you know, hey, we're having a really hard time placing these HBCU schools because people don't want to hold them.

00:24:50
Speaker 1: The way the bond market works is that an institution needs money. Let's say a university wants to build a dormitory that costs fifty million dollars. The school can't raise all that money on its own, so it goes to a Wall Street bank, which purchases a bond a promise of repayment from the university. Then the bank breaks that bond into many hundreds and hundreds of pieces and sells those pieces to wealthy investors or insurance companies or hedge funds. So in May you hears that banks are struggling to sell bonds for historically black colleges, he wonders, is this just some apocryphal Wall Street story or is it true? Luckily, there's a simple way attests this. The price the bond trader charges is called the gross spread. If it's super easy to find buyers for the bonds, the gross spread is likely to be low. If the trader has difficulty finding buyers, the spread can get quite pricey. So if the bonds from black schools are harder to move than bonds from white schools, the spread should be higher. And are they Yes, they are May you and his colleagues compared black schools with non HBCUs that were otherwise identical. The spreads for the historically black schools were twenty percent higher in the bond market. A twenty percent difference in spreads is huge. Mayo and his co researchers then went one step further. They zeroed in on three states in the Deep South, and those three states were so.

00:26:26
Speaker 3: Those are Louisiana, Mississippi, and Alabama.

00:26:31
Speaker 1: And what's the difference in spread in those states? Not twenty percent? Thirty seven percent. Mayo and his colleagues have given us a scientifically rigorous measurement of just how much of a burdened black institutions face in the real world. You've put a number on reputation. What is the reputational cost of being a school that overwhelmingly accepts and educates black kids in Louisiana. The reputation cost is thirty percent.

00:27:01
Speaker 3: That's one way to think about it. But I would say it's the racial component of reputation.

00:27:07
Speaker 1: A spread is a reputation in score, isn't it. Investors don't know the precise ins and outs of all the institutions they're asked to buy how could they. There are literally hundreds of thousands of bonds on a market at any given time, So the investors proceed on a feeling, a feeling about how worthy and how safe and how legitimate the institution getting the loan is. You see a black school and you say to yourself, I don't value that HBCU quite as highly as the white school across town. So you give the black school a discount, which is what prejudice is, by the way, a discount. The black life does not matter as much as the white life. And who are these people imposing a discount on black schools. They're wealthy, successful people who are simply checking a box. I like this bond, I like this one a lot, I don't like this one. They are, I suspect, not a whole lot different from the college administrators who assign a number to their peers for them. The US News rankings every year motivated by the same unconscious biases. I like the school four point two, I like this one a lot four point five. I don't like this one two point four, even though I don't really know anything about it. So if you were the president of a historically black college in one of those States with a reputational discount for being black is thirty seven percent. How do you think you'd feel about your peer assessment score? I thought I'd call Walter Kimbro, president of Dillard University, a historically black college in Louisiana, And.

00:28:47
Speaker 6: So when I start to look at the formulas and dig into it, I was like, this isn't a real good measurement of what we do. In fact, it's a measurement of privilege.

00:28:56
Speaker 1: Dillard's reputation score is two point six. That's not good. You can't ever get to the top of the US News rankings with a peer assessment score of two point six.

00:29:06
Speaker 6: Maybe that sounds defeatis, but I mean.

00:29:09
Speaker 1: It's it's not defeat its it's realistic, verble living it.

00:29:14
Speaker 6: Nobody's going to just go and say this school, with all these black people, is great.

00:29:19
Speaker 1: In their haste to compete with Time and Newsweek way back when, maybe all US News did was create a system that allowed the presidents and deans of every college in the country to assign a number to their prejudice to disguise mythology as methodology. Next week on Revisionist History, we take up the cause of Dillard University, and I enlist my old friends at Reed College to help me out.

00:29:49
Speaker 5: So that is how we can get Dillard into the tippy top tier of the liberal arts colleges.

00:29:56
Speaker 1: So it's Williams Amherst Dillard swarthwik Yes Division's history is produced by mel Belle, Leeman Gesteu and Jacob Smith, with Eloise Linton and on An eyem Our. Editor is Julia Barton. Original scoring by Luis Kira, mastering by Flonn Williams, and engineering by Martin Gonzalez. Fact checking by Amy Gaines. Special thanks the Pushkin crew had a fine Carl MCGLIORI, Maya Knick, Maggie Taylor, Eric Sandler, Nicol Morano, Jason Gambrell, and of course Jacob Weisberg. I'm Malcolm Glaba. If you love this show and others from Pushkin Industries, consider becoming a Pushnik. Pushnick is a podcast subscription that offers bonus content and uninterrupted listening for four dollars and ninety nine cents a month. Look for Pushnik exclusively on Apple Podcasts subscriptions with three flavors Ali's Nasty, Nasty, Delicious, and Nasty Vicious. Okay, guys, but you hold on the websites to do simultaneously. Ye okay, one, two, three, God oh, I like that it comes on, Oh, go Ali, go Ali, because it's really good. It's good.

00:31:43
Speaker 3: Hold on I okay, it's very spicy for a very long.

00:31:49
Speaker 1: Time, because am I, it's not eating me as hard as it did. I'm taking a second bite. Martine's struggling, By the way, where's my Harvard hot saice? I don't. I don't see any Harvard hot sauce. No one's sending me hot sauce from New even, are they.

00:32:10
Speaker 3: No