RateMyHeuristic.Com

RateMyHeuristic.Com

Mark Morton, Instructional Program Manager, Centre for Learning and Teaching Through Technology

Everyone loves a heuristic: a handy, tidy, bulleted, get-down-to-brass-tacks distillation of the best practices for navigating your way through a complex system or situation. In higher education, one of the best known heuristics must surely be the “Seven Principles for Good Practice in Undergraduate Education,” devised in 1987 by Arthur Chickering and Zelda Gamson. As you probably know, Chickering and Gamson’s seven principles encourage things like “student/faculty contact,” “prompt feedback,” and so on. Other heuristics have also been proposed as alternatives to that of Chickering and Gamson. For example, the National Survey of Student Engagement (NSSE) advocates “Five Benchmarks of Effective Educational Practice,” such as “level of academic challenge” and “active and collaborative learning.” Likewise, Patrick Terenzini, the author of the award-winning How College Affects Students, identifies “Six Characteristics of Learning and Development,” including “real world activities” and “unbounded by time or place.”

When I first learned of the trio of higher education heuristics mentioned above, I wondered whether they were simply three different ways of conveying the same pedagogical ideas, or whether they were fundamentally and conceptually distinct from one another. Finding out was fairly straightforward: I simply laid out all three heuristics in front of me, and tried to map the principles of one heuristic against the principles of the other heuristics. There was, it turned out, some degree of overlap – for instance, all three heuristics affirm the value of active learning and collaborative learning – but overall I was struck by their lack of correspondence. As a single example, Chickering and Gamson’s “prompt feedback” has no counterpart in either the NSSE or Terenzini heuristics.

All this got me thinking that if these three heuristics couldn’t agree, then maybe I should try to invent my own.

I decided that my heuristic would be based upon the attributes that students themselves identify as contributing to effective instruction. To this end, I collected and analyzed a thousand comments from a website known as RateMyProfessors.Com. As you may already know, RateMyProfessors.Com is an advertising-supported website that allows students from any university in North America to rate their instructors on a five point scale. Students also have the option of supplementing their numerical rating with written comments. These ratings and comments are then collated so that a user – let’s say a student considering a Biology course offered by Professor Nemo – can check out what other students have said about Professor Nemo, and factor that into his or her decision to take the course.

Let me now elaborate on my method. First of all, rather than randomly choosing a thousand comments from RateMyProfessors.Com, I instead selected ten top-ranked and ten bottom-ranked instructors at each of five Canadian universities, including my own. For each of these instructors, I chose ten cogent student comments. By “cogent,” I mean that the comment identified at least one meaningful attribute about the instructor. Thus, a comment such as “She explains difficult concepts really well” is a cogent comment, whereas “He sucks” or “She rocks” is not. I then did a qualitative analysis of the thousand cogent comments, tabulating and categorizing the instructor attributes that students cited. I ended up with thirty categories of attributes, such as “good sense of humour,” “passion for teaching,” “helpful outside of class,” and “fair assessments.”  I then converted the raw numbers that each category received into percentages of the total: thus, for example, “passion for teaching” represented 4% of all the comments, while “good lectures” represented 9.5% of the total. Finally, I collapsed most of the thirty categories into broader categories. For example, the categories of “friendly,” “good sense of humour,” “passion for teaching,” and a few others were all subsumed into a broad category that I called “Character” – that is, attributes that pertain to the personality or demeanor of the instructor. Once I was done collapsing my thirty original categories, I was left with a heuristic comprising four “master” categories, namely, Character, Transmission, Course Design, and Assessment. In other words, those four categories are the ones that students tend to identify as most important to them in effective instruction. Let me say a bit more about each one of these four categories, and also about their importance in relation to one another.

First, the category of Character, as I’ve already said, pertains to an instructor’s personality or demeanor, including whether he or she is friendly, has a good sense of humour, is enthusiastic, and so on. Character represents a whopping 45% of all the comments that I sampled from RateMyProfessors.Com.

Second, Transmission pertains to how an instructor conveys content to the students. I phrase it this way because that, judging from the student comments, is how students think about their learning, that is, as the transmission of ideas and information from instructor to student. To put it another way, students don’t see instructors as facilitators of learning whose job is to devise ways to get students to interact with content. Rather, as far as students are concerned, instructors are transmitters of content, which to them means giving good lectures, making available good lecture notes, responding to student questions clearly, using good examples and demos, and so on. Transmission represents 31% of all the comments.

Third, Course Design pertains to how well prepared and organized an instructor is, whether there is congruence between what is taught in class and what is tested during the exam, whether the pace and workload are reasonable, and so on. Course Design represents 7% of all the comments.

Fourth, Assessment pertains to whether an instructor grades fairly and whether he or she provides timely and useful feedback. Assessment represents 6% of all the comments.

Clearly, there are some striking findings here, not the least of which is that what students identify as having the biggest impact on their learning is the character of the instructor. To put it another way, if an instructor wants to do one thing that will help students learn, then it appears that he or she should be friendly and tell the occasional joke. The importance of this to students shouldn’t be underestimated: from a percentage point of view, attributes pertaining to Character were mentioned in the student comments almost seven times more frequently than attributes pertaining to Assessment.

Another salient conclusion would seem to be this: that students think about learning in terms of a transmission model, the “sage on the stage” approach, as opposed to the active learning or “guide on the side” strategies that are currently being propounded by experts in higher education. In other words, the comments from RateMyProfessors.Com indicate that as far as students are concerned, good lectures do make good learning.

In a nutshell, that’s what my heuristic-driven analysis of student comments in RateMyProfessors.Com turned up. But of course there’s a huge caveat that must be addressed: namely, should we give any credence to these findings? Do students know what they are talking about? Are students credible authorities on what makes effective instruction? When it comes to rating instructors, are they actually able to distinguish what is effective from what they like or what they are used to?

My own response to these questions is mixed. I think, first of all, that we can unreservedly accept the students’ high valuation of Character. In other words, if students say that they learn better from an instructor who is friendly and has a sense of humour, then I think we really need to take this at face value.

However, with regard to high valuation of a transmission model of education, I think we need to be a bit more cautious. On the one hand, I share this high valuation: I love listening to a good lecture, either in a classroom or on my iPod. But on the other hand, I’m also convinced that learning activities that are more interactive than a lecture are a powerful resource for a instructor to draw upon. I also think that learning activities that are collaborative in nature – which the traditional lecture is not – can contribute greatly to learning outcomes. University students, however, don’t seem to recognize yet that replacing or supplementing the traditional lecture with more active  forms of learning can be a good thing. This is due in part, as I suggested above, to the fact that lectures are what students are used to, and straying from this familiar sine qua non into collaborative work, class presentations, reflective learning, task-based learning, one-minute summaries, and other forms of active learning can feel disorienting and annoying, like being forced to switch from Fahrenheit to Celsius, or from a high-fat diet to a low-fat diet. But I think the reluctance of students to abandon a lecture-centered model of education also results from their tendency to see themselves as consumers, and learning as a product rather than a process. Their attitude toward education is like mine when I hire a plumber: I just want him or her to fix the damn leak under my sink, not to facilitate a collaboration between myself and my spouse in which we reflect on our household water consumption. The difference, of course, is that learning really is a process, a personal and also social construction of meaning, not the acquisition of a commodity or even of a service.

In any event, what seems ineluctable is that there is a significant disjunction between student expectations and the active-learning pedagogies that experts in higher education are encouraging instructors to adopt. Students expect course content to be transmitted to them via lectures and lecture notes, and the extent to which this expectation is fulfilled correlates with how they rate their instructors; yet many instructors are attempting to supplement or replace traditional lectures with learning activities that are more interactive and collaborative, and in so doing they end up frustrating their students’ expectations and potentially sabotaging their own course evaluations. The solution, perhaps, is not to abandon active-learning strategies, but to educate students about their education. That is, instructors who are shifting away from the traditional lecture model need to spend some time explaining to their students the reasons for the shift. They need to show their students the evidence, which abounds, that active and collaborative learning activities result in improved learning outcomes. And in order to make this case a convincing one, the instructor’s explanation needs to be ongoing: not just a fifteen-minute spiel given on the first day of a course, but a recurrent theme that is woven into or around the activities that are built into the course. Changing attitudes and assumptions may be difficult, but it is, after all, what learning is all about.

Further information

A PowerPoint presentation that provides much more detail regarding the methodology and conclusions of the study described in this article is available via a link at the following URL: http://lt3.uwaterloo.ca/resources/papers.html (Look for the link entitled “‘Even Your Pillow will Need a Pillow’: The Extent to which Student Comments from RateMyProfessors.Com Map Against the Best Practices Proposed by Experts in Higher Education.”)

Seven Principles for Good Practice in Undergraduate Education:
http://www.msu.edu/user/coddejos/seven.htm

National Survey of Student Engagement:
http://nsse.iub.edu/pdf/nsse_benchmarks.pdf

Six Characteristics of Learning and Development
http://www.educause.edu/eliweb061