and
Lieutenant Colonel Eugene K. Ressler, Ph.D.
Associate Dean for Information and Educational Technology
Building 600, Attention: MADN-IETD
U.S. Military Academy
West Point, New York 10996
ressler@usma.edu
914-938-4189 (assistant)
914-938-5141 (fax)
http://www.dean.usma.edu
The most useful and believable studies of courses and academic programs are those that instructors and directors design themselves, focusing on those questions about which they care most. We present two systems that provide such tools to support the scholarship of teaching. First, a general purposes system for collecting information from students at the end of courses. This was developed by the U.S. Military Academy at West Point to coordinate academy, department and faculty inquiry and curriculum assessment. Second, we present Flashlight tool kits and training that focus on improving instructional uses of computing, video, and telecommunications.
When the United States Military Academy developed a model for centralized end-of-course feedback a decade ago, we eschewed the evaluative mode that has become so problematic in higher education (Gerstman, 1995). We opted for a purely formative system that provides information about an individual instructor’s class solely to that instructor. Program directors, department heads, and the Dean all receive only aggregate information at their respective levels, information that is helpful for program assessment, planning, individual faculty development, but that does not allow for evaluation of individual instructor performance. In its original paper and pencil format, student feedback was limited to responses to a standard set of questions, and the scoring of surveys delayed instructor access to survey information. Our new electronic version is substantially more flexible and capable. It has expanded our ability to provide useful student feedback information at every level easily and quickly. Originally begun as a cadet design project in a computer science class and further developed by one member of the design team as an independent student project, in 1998, the authors, drawing on the cadet work, took the system to this "next generation" providing its current flexibility.
In brief, the system protects the anonymity of instructors through a system of PINs used to control views of information. It operates in a once-per-semester cycle. During phase I, designated individuals in each department, the "trusted agents," organize the department’s academic program as a hierarchy that reflects its organizational structure by creating "nodes" for designated units; that is, its programs, courses, and sections. They determine "who sees what" by setting access lists for each node. A faculty member with access to a node may write survey questions and receive a tailored report for that node. For example, the English Department head may write questions, and the report he receives will represent what every student taking an English course responded to those questions, as well as the Dean’s. The course director for freshman composition writes questions, and her report will contain the response of every student enrolled in freshman composition, both for her questions and those "above" her in the hierarchy---perhaps the core course program, the department head, and the Dean’s questions. Each composition instructor can write questions for his or her sections, and the reports will contain the responses of only the students in those sections, but for questions at all levels of the hierarchy. However, although the instructor can see how his or her students responded to the department head’s questions and the course director’s questions, neither the department head nor course director have access to information at levels lower than their own. The faculty members have most of the semester to create the department hierarchy and write questions because phase 2, student response, does not begin until the final week of class.
During phase 2, students respond to a unique, personalized survey for each class section they attend, created by the system on-the-fly by joining questions of relevant nodes. For instance, a student in the freshman computer science core course, CS105, hour D, section 1, will receive a survey potentially formed from questions written by the instructor for the section, by the instructor for all his/her sections, by the course director of CS105, by the computer science program director, by the department head for all non-elective courses, and for the department as a whole. There is also a short list of standard questions for the entire Academy. While this may seem a daunting list, we have cautioned our faculty to make sure that individual student surveys not exceed a total of fifty questions and be closer to the thirty-question limit of our earlier paper version. Questions can be multiple-choice with or without a scale, with exactly one or any number of responses, and they can be "rank ordering" questions. Faculty members can ask for free text responses, which are delivered either only to the instructor (maintains high confidentiality) or "rolled up" at all nodes below the one where the question was asked (maximizes information). Questions can be "mandatory" or "optional" in the sense that surveys cannot be submitted while missing answers to mandatory questions. A sample of this system can be viewed at http://www.dean.usma.edu/feedback.
The system is in Perl for CGI over Microsoft IIS. Source codes are available on request. Mail ressler@usma.edu.
In its first two semesters of operation, the system collected about 20,000 responses each semester of 22,000 possible, about 91 per cent overall. We judge this a successful response rate. Note the system informs instructors of students who have and have not responded during Phase II, allowing for in-class reminders.
End-of-course feedback used as an evaluation of teaching is much criticized in higher education and remains largely because of what has been identified as the principle of faute de mieux, loosely translated as "lack of anything better" (Gerstman, 1995). A major problem is the tendency to mistake the numerical nature of the data with objectivity, ignoring the multitude of variables that affect student evaluation of instruction. However, when considered from the formative perspective, there is more of a tendency to see the feedback contextually, understanding it as the student impression of the learning situation and appreciating its value for certain types of information. For example, in our first semester core math course, the course director composed questions to obtain feedback on how well instructional intentions were meeting student learning needs, and he reported satisfaction with the results:
The feedback let us know that most of the cadets (71%) felt that technology helped them understand mathematical concepts, 77% indicated that the emphasis we place on concepts rather than formulas helped them learn mathematics, and the requirement that they communicate mathematics by writing and briefings helped them learn the mathematics (76%). We take that as an indication that we are doing what we hope to do. (e-mail message from MA103 course director)This same faculty member asked different questions of his own sections, seeking information about his individual classroom practices. He reported that he confirmed the value of cadets "going to the boards" to work problems but also students advised that he should be "more patient in waiting for cadets to finish problems on the boards," something he probably would not have learned without this type of feedback that allows fee text responses.
We conclude that the system has value for those who choose to employ it. It also imposes no administrative burden on those who do not. Future work will be directed at guiding faculty in question writing and in interpreting results. The academy level questions are also a subject of study by the faculty teaching committee.
Angelo, T.A. (1991). Classroom Research: Early Lessons from Success. San Francisco: Jossey-Bass.
Angelo, T.A. and Cross, K.P. (1993) Classroom Assessment Techniques: A Handbook for College Teachers. San Francisco: Jossey-Bass.
Cross, K.P.(1996) Teaching and learning for the twenty-first century. Keynote address at conference for federal service academies, West Point, New York, September 1996.
Gerstman. B.B. (1995) Student evaluations of teaching effectiveness: the interpretations of observational data and the principle of faute de mieux. Journal of Excellence in College Teaching. 6, 115-124.
Menges, R.J. and Rando, W.C. (1998). Feedback for enhanced teaching and learning. Teaching on Solid Ground: Using Scholarship to Improve Practice, ed. R.J. Menges, M. Weimer and associates. San Francisco: Jossey-Bass. 233-255.
Shulman, L.S. (1988) A union of insufficiiencies: strategies for teacher assessment in a period of educational reform. Educational Leadership. 46, 36-41.
Upcraft, M.L.(1998). Teaching and today’s college students. Teaching on Solid Ground: Using Scholarship to Improve Practice. 21-41.
This essay contains a brief history of the Flashlight Program, summarizes its current operation, and suggests some future directions. Flashlight's goal is to help institutions evaluate and improve their educational uses of technology. The program is a self-supporting, non-profit operation, part of the TLT Group. The TLT Group is the teaching, learning, and technology affiliate of the American Association for Higher Education.
Flashlight was created in 1993 by the Annenberg/CPB Projects, where I was a program officer. We got initial funding from FIPSE and then Annenberg/CPB itself put up some substantial development money. That's why copyright for the tool kits is still held by the Corporation for Public Broadcasting. We moved to AAHE in 1996 and then helped found the TLT Group when it began operation separate from but affiliated with AAHE in 1998.
Our initial notion with Flashlight was to create a suite of standard survey and interview forms that could help departments and institutions evaluate and improve the usefulness of various technologies for supporting access and the seven principles of good practice (e.g., student-faculty interaction, student-student collaboration, active learning, time on task, etc.). The tools were (and are) designed for studies both of on-campus and off-campus learning.
Our work with our five institutional partners quickly convinced us that the thematic emphasis was right but that tool kits were more needed than tools. So we began painfully slow development of tool kits, starting with an indexed library of pre-tested items for gathering information from currently enrolled students (the Current Student Inventory or CSI). This came out in Dec. 1997 and has been site licensed by about 180 institutions around the world so far.
Next came our Cost Analysis Handbook (actually a methodology for studying the use of all resources, including time, along with case examples); that was published just a few weeks ago and already seems quite popular.
One of our institutional partners, Washington State University, volunteered to incorporate the CSI into a Web-based authoring and data collection/analysis system they were building. That was the genesis of Flashlight Online, a Web-based service for creating surveys, gathering data, and sharing studies that we rolled out about a month ago, thanks to a huge amount of work by Gary Brown, Peg Collins and others at WSU. Subscribers to Flashlight Online can currently use it to create surveys using CSI items; we'll add faculty items to the pool, too, later on.
We're also using Flashlight Online to return to our roots: creating standard surveys that could be widely used, with pooled data. In other words, users can use our survey, as is, and, if they're willing to share data, they can compare their students' responses with those from students in other institutions. The first such surveys are now under development: one in nursing for distance learning, one for use of presentation software such as PowerPoint, and a suite of surveys for use with course management systems.
We learned along the way that tool kits and tools were important but not enough. Our guiding philosophy has two important components (at least). First: what's most important is how a technology is used - it's the activity that should be the focus of the inquiry. The second point emerges from the first. If the particular use of technology is what determines benefits, costs, and problems, then evaluators face a problem. Their findings can improve those benefits, control costs etc. ONLY if their findings influence how lots of people use technology. But most users seem to pay little attention to evaluations and, if the findings reflect badly on what they do, they don't believe the findings. Solution: involve users in the design of studies so that the study actually focuses on issues about which the users already feel some uncertainty and concern - so that they are eagerly awaiting the data. But that requires attracting their attention so that they'll help design the study (we try to help local evaluation leaders attract their colleagues' attention by giving talks, writing articles, briefing people, etc.) and some training (where we try to help with workshops, face to face and, soon, online). We just ran two national workshops that people seemed to find wonderful, one at IUPUI on cost analysis and a second at RIT on evaluation of Web-based courses. If anyone on this list was at one of those workshops, they might want to comment. Those were nationally advertised workshops - we run 2-4 of those a year - but most of our workshops are on campuses and serve that campus, or its system, or perhaps also folks from neighboring institutions. (If you'd like to host such a workshop, please contact my colleague Amanda Antico at Antico@tltgroup.org or 202-463-1784.)
Finally, in addition to toolkits-tools and events-training we also offer active assistance, e.g., creating instruments for institutions or even doing studies for institutions (e.g., external evaluation of grants, external assessment of programs). For example, we're about to begin a fascinating study for Southern Cal on their interdisciplinary efforts to teach undergraduates more visual and interactive forms of academic expression.
We bundle those three kinds of help (tool kits and tools; events and training; active assistance) as the Flashlight Network program. Institutional members get all our tools, inexpensive services, and the opportunity to play a leading role in Flashlight development. Members are quite varied in their technological and evaluation background - some wanting help to get started, some national leaders already. A few of them: University of Iowa, Massachusetts Institute of Technology (MIT), Fort Berthold Community College, University of Hong Kong, Regis College, UC Santa Cruz, Georgia Tech, Sinclair Community College, National University of Singapore, Stanford University, Regents of the University System of Georgia, Mount Royal College (Canada), Michigan Virtual University, University of Kansas Medical Center, Cal State Sacramento, Rochester Institute of Technology, Appalachian State University, Virginia Tech... You can get information on all this, including more information on Flashlight Network members and Flashlight licensees, starting with our home page at http://www.tltgroup.org/programs/flashlight.html
Our current thrusts will continue at least for the time being: helping as many institutions as possible develop the capacity to improve education with technology because they can evaluate what they're already doing. That institutional capacity is a combination of what faculty, staff and students can find out on their own and what they can find out together. We are especially excited about using Flashlight Online as a vehicle for helping faculty and institutions pool data and compare notes.
Flashlight's basic ideas emerged from talks and articles about "what works with technology" - my observations about areas where technology seemed appropriate enough, cheap enough, and easy enough for lots of institutions to use it. I've continued to watch the passing scene and our observations of current trends will shape new generations of evaluative tools. That's because we see evaluative tools in part as ways of measuring progress and investigating barriers to progress: for the country, for institutions, for programs and services, and for individual courses. That's how Flashlight works: describing key goals and concerns about technology use and then helping people decide how their own work can be gauged against those goals and concerns: For examples of a couple of recent articles that will shape our new work, see the September issues of Educom Review and Academe. Drafts of those articles are on our Web site at http://www.tltgroup.org/resources/varticles.html. One is called "Access and/or Quality? Redefining Choices in the Third Revolution." The second is called "Grand Challenges Raised by Technology: Will This Revolution be a Good One?" We look forward to hearing your comments about what we are doing and what we should be doing. Please send them to me or to Flashlight@tltgroup.org.