The University of Iowa Task Force on Graduate Education faced a daunting task. Its charge was to evaluate around 100 graduate programs offered by the university, in an enormous range of disciplines, in 10 months. That’s, on average, one program every three days, counting weekends. It produced a 681-page report.
How was this task accomplished? It’s a bit hard to say. UI Graduate College Dean John Keller has said the treatment of departments was “equitable” — I suppose he means something like “balanced and fair” — but it’s not easy to imagine a methodology that would enable a committee composed of 18 faculty members, one student, and two administrators to achieve that.
If there’s anything about the task force’s report that leaps to the eye, it is this: Of the graduate programs of the 19 departments represented by the faculty and student members on the committee, all but two were classified as either “exemplary” or “high quality” (the remaining two were rated “good”).
More specifically: Among the committee membership, the representation of “exemplary” programs was 10 programs, or 53 percent. But in the pool of all programs evaluated, the percentage given “exemplary” ratings was a mere 14.4 percent. This discrepancy can hardly be a statistical fluke. What could account for it?
Maybe the provost knew ahead of time what graduate programs were “exemplary” and chose to select committee members from that pool. But that can hardly be accepted: If the provost knew ahead of time which departments have excellent graduate programs, why all the expenditure of time, money, and effort on discovering this?
Perusing the written summary evaluations of the programs (a short paragraph is devoted to each) is an exercise in bewilderment. I cannot begin to summarize all the puzzling discrepancies, nor do I wish to single out departments.
Suffice it to say that there is no visible correspondence, in numerous cases, between the justification given for a rating and the rating itself. Ratings were ostensibly based upon (among other things) selectivity of admissions, GRE scores of students, retention rates, average time to graduation, and job placements. Different measures were cited for different programs, with no rationale for this. If a department got a high rating, it appears that often the committee searched for positive numbers and ignored numbers that were extremely poor. And for poorly rated programs, the reverse appears to be the case.
In all of this, there were glaring omissions. Foremost of these is that there was no assessment of faculty quality. It is hard to conceive how this can fail to be a major factor in the quality of a program. Second, such numbers as time-to-completion took no account of the fundamental differences in the kinds of research carried out in different fields (or even within a field); nor did it take into account the fact that some programs admit only — or largely — students who have earned M.A. degrees from other institutions. Third, there was no protocol for determining how various measures of success were to be weighted, nor any rationale for such a protocol.
Thus, the committee’s evaluations give every appearance of being largely seat-of-the-pants judgments. And indeed, it’s hard to see how it could possibly be otherwise. If people think that such a complex thing as a graduate program can be evaluated formulaically, they are laboring under a delusion.
The committee’s task was hard. Given the constraints, I would say it was impossible. Nevertheless, it came up with a set of ratings. What explains why those ratings so conspicuously favored departments represented on the committee? I shall leave that as an exercise for the reader.
Evan Fales is a UI associate professor of philosophy.