This article was published in CAUSE/EFFECT journal, Volume 21 Number 3 1998. The copyright is shared by EDUCAUSE and the author. See http://www.educause.edu/copyright for additional copyright information.
Browser-Based Class Evaluations
by James HefferonOne of the main tools used by colleges and universities for assessment and development of instruction is student evaluations. At Saint Michael�s College in Colchester, Vermont, we are experimenting with enhancing this process using Web-based tools. This is a faculty-initiated project that aims to use the flexibility and ease of browsers to get more useful information out of the process than is possible with static paper forms alone.
The problem and the approach
For at least a decade, Saint Michael�s has asked all students in all classes to fill out paper class evaluation forms that are provided by a vendor and then returned to that vendor for processing off-site. Although outsourcing the job has advantages, two disadvantages are that all classes fill out the same form--the one that is given to all of the vendor�s customers--and that it takes too long to get the results back (about six months). Instructors object to both of these drawbacks, saying that they reduce the value of the information as feedback for continuous improvement and instead make it more like a grade for the class from the students. One solution is to leverage the existing information technology to address the disadvantages of the current system--to do it online.
In a new system under trial at Saint Michael�s, students use a Web browser to fill out a form for each class. These forms contain both Likert items ("rate one to five") and open-ended items ("type your comments"). The forms differ from class to class, except for four questions that are common to all classes, which are useful for comparisons. Students say that the forms are easy and intuitive.
The core of the project is that the instructors compose the forms. So an instructor in a class without a laboratory simply does not include the question asking students about the lab, and an instructor who tries something new can include a question targeted at this innovation.
To make construction of these forms as easy for the instructors as their use is for the students, we have developed a "make-a-form" form. After picking the class and section from a list, the instructor chooses a list of questions for the students in that class, selecting from about two dozen prepared questions, and also from ones that the instructor may compose. For each chosen question, the instructor clicks to include either a Likert component or an open-ended component, or both.
In the last week of class, the resulting evaluation forms are opened for access and left open until the day that finals begin. Then, on the day after grades are due, the summary reports are sent out.
The entire process is more flexible and faster than with paper, and the cost of outsourcing will be eliminated.
Feedback instead of criticism
Learning research has established that in the performance-improvement process, feedback must be prompt to be effective. This is one major advantage of our new system, since the responses reach the instructor at the earliest possible time.
Another advantage is that students give many more open-ended responses than they have given on the paper forms. For example, in past years the students in Elementary Statistics have usually given fewer than ten such comments, so that the information received was mostly restricted to averages. Under the electronic system, there were over forty comments. Instructors have said that this is the main reason that they prefer the new system. The process is now more of a dialog.
Where the bits go
After the instructor selects the questions to be asked and submits the form, a CGI script turns the submission into the class evaluation form that will be seen by the students. At the same time, a small database is created. During the comment period, the class�s database is filled by another CGI script that processes the students� feedback. At the end, the data is turned into a report. Because this report is locally generated, we can put in exactly what we think is most useful (for instance, input from a member of our Faculty Council, which is responsible for promotion and tenure decisions, prompted us to add the ability to give department-wide averages for the four in-common questions). These reports can be sent on paper or electronically.
Instructors want as many students as possible to submit a form, but they also want to be sure that each student can submit only one form. For the first concern, we have found that if students are simply told the address for the form, then the response rate is not adequate (below half), but if students are sent an e-mail allowing them to click to get to the form, then the response rate is strong (about three quarters). For the second concern, the system currently requires students to give their ID number, which is checked against the registrar�s list before the response is accepted. A better solution, using a Private Key/Public Key client verification system, is under consideration.
Where the bytes are
Our software was developed on a Linux system using tools available from the Open Source software community (Apache as the Web server, Python for the scripts, and TeX for the reports). In the same spirit, our software is also freely available (see http://joshua.smcvt.edu/evaluations), and because Python and TeX run on most platforms, our solution should be transportable.
Conclusion
The Web offers many chances to improve the way we generate and use information. Saint Michael�s looked at the paper-based class evaluation system and decided that we could do everything that the current system does, and some of it better, with the Web systems that are newly available. Our students, faculty, and staff have been pleased with the results.
Jim Hefferon ([email protected]), Ph.D., is associate professor of mathematics at Saint Michael�s College in Colchester, Vermont.
