CAUSE/EFFECT

Copyright 1998 EDUCAUSE. From CAUSE/EFFECT Volume 21, Number 2, 1998, pp.62-64. Permission to copy or disseminate all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, the EDUCAUSE copyright and its date appear, and notice is given that copying is by permission of EDUCAUSE. To disseminate otherwise, or to republish, requires written permission. For further information, contact Jim Roche at EDUCAUSE, 4840 Pearl East Circle, Suite 302E, Boulder, CO 80301 USA; 303-939-0308; e-mail: jroche@educause.edu

Readers Respond

Question:
Has your institution attempted to assess the impact of information technology on instruction, student life, and/or administrative services, or do you have plans to do so? Please comment on your progress in one or more areas.

I would encourage anyone interested in assessing the impact of IT to look at the results of a project sponsored by the Coalition for Networked Information (CNI), a partnership of EDUCAUSE and the Association of Research Libraries (ARL). A complete description of this project, called Assessing the Academic Networked Environment, can be found on the CNI Web site at http://www.cni.org/projects/assessing/. In particular, the manual by Charles McClure and Cynthia Lopata gives excellent guidance on what needs to be thought through before implementing campus surveys, as well as providing a sample survey.

Seven institutions are working with CNI to continue work on assessment and to develop instruments that address a number of aspects of network and networked information services use. Progress reports are available from seven of these schools: Brown University, Dartmouth College, Gettysburg College, King's College London, Mary Washington College, University of Washington, and Virginia Tech. In addition, Christopher Peebles of Indiana University, has done some impressive work on assessment, which was presented at the 1997 CAUSE conference, and to which the CNI assessment site provides a link. Other links are provided to sites that offer very rich resources in this area, including the Flashlight Project (see http://www.learner.org/edtech/rscheval/).

Joan K. Lippincott
Associate Executive Director
Coalition for Networked Information
joan@cni.org

At The California State University I am currently directing a two-year project entitled Case Studies in Evaluating the Benefits and Costs of Mediated Instruction and Distributed Learning. The project was first funded in 1996 by the U.S. Department of Education, with additional support from The California State University. We have data from nine campus case studies, six involving various forms of television delivery and three based upon computer applications. Five of the cases are published and available at the project Web site http://www.calstate.edu/special_projects/. Other cases and other project reports will be available at the Web site as they are completed.

Case Study Findings:

Benefits --(1) Learning outcomes for students enrolled in electronically mediated courses, as measured variously by course grades, exam scores, student surveys, faculty surveys, field supervisor surveys, and alumni surveys are essentially equivalent to learning outcomes for regular classroom instruction. These results are consistent with other comparative studies (see Thomas Russell, "The No Significant Difference Phenomenon," http://tenb.mta.ca/phenom/phenom1.html). (2) Student access is improved for cases that involve delivery of instruction to sites remote from the campus classrooms. "Improved access" for students is another way of describing "increased convenience" for students.

Costs-- The estimated direct costs of distributed instruction differ from classroom instruction in two basic ways: (1) The fixed or "start-up" costs of mediated instruction are greater than the fixed costs of classroom instruction. (2) The incremental or marginal costs of mediated instruction (the costs of adding more students to a given course in a given year) are less than the incremental costs of classroom instruction. Many forms of distributed instruction are subject to economies of scale, and at some level of annual course enrollment, distributed instruction can be less expensive than classroom technology. Also see: A. W. (Tony) Bates, Technology, Open Learning and Distance Education, 1995, Sir John Daniel, Mega-Universities and Knowledge Media, 1996, esp. p. 62; and Greville Rumble, The Costs and Economics of Open and Distance Learning, 1997.

Cost Simulation Model --The project has also developed a cost simulation model (BRIDGE) that is being used to compare the costs of a campus that is offering a substantial amount of its FTE in distributed instruction with the costs of a campus that is offering all of its instruction in classroom (lecture-laboratory) mode.

Frank Jewett
Research Project Director Information Resources and Technology
Chancellor's Office
frank_jewett@calstate.edu

Seton Hall began with a simple, three-pronged approach to assessment: understanding and improving the learning, teaching, and the materials with respect to information technology.

The university has been engaged in major information technology initiatives since the mid-nineties when it launched a Mobile Computing Pilot Project, a cross-department seed grant Curricular Development Initiative, and rolled out its Information Technology Strategic Plan. Through an external grant and continued efforts in the Teaching, Learning, and Technology Center, the University has woven assessment and quality assurance, with technology, into the fabric of its teaching and learning.

To understand the learner, the university administered a nationally normed survey to gather information on student learning preferences, followed by a baseline technology proficiency inventory. This provides a snapshot of a student profile that could be used for formative course development as well as to understand the impact of technology in a summative way.

As the technology is brought into the courses and curricula, a number of faculty keep journals to document their experiences. The journals focused on the cognitive, behavior, and affective changes occurring with the use of technology.

At the end of the semester, students participate in focus groups and complete a survey based on the work of Chickering and Gamson detailing if and how the use of various technologies were used to enhance best practices in teaching and learning for the purpose of improving student learning outcomes.

The thread that ties Seton Hall's assessment with technology is communication. Efforts are made to communicate assessment results widely via faculty technology training and program enhancement, throughout University Computing and to the executive level. The latest in Seton Hall's assessment efforts is to assess the learning environment as it is impacted by technology beyond the classroom, such as student services and student life.

For more information contact Heather Stewart at stewarhe@shu.edu

Phillip D. Long
Director, University Computing, TLT, and Support Services
Longpd@shu.edu

Over the last ten years there has been growing interest in universities and colleges across the U.S. in the use of technology for supporting learning in higher education. Evaluation of the contributions and limitations of technology in supporting and improving learning are sorely needed for all participants -- university administrators, faculty and students.

The Stanford Learning Laboratory (SLL) was established in 1997 by Stanford President Gerhard Casper and the Commission on Technology in Teaching and Learning (http://learninglab.stanford.edu/). Its mission is to enhance the personal learning experience of all Stanford students and to create a model for the judicious use of pedagogically-informed learning technology. It is SLL's conviction that in-depth assessment is critical to achievement of our mission. Ad hoc experiments and flirtation with emerging technology will not yield the kind of systematic understanding required for efficient deployment of new technical and behavioral learning models.

In all its projects, the Lab conducts a comprehensive assessment using both qualitative and quantitative methods to evaluate the project's effectiveness, utility, impact and deployment barriers. The projects continue for several iterations to refine courses and learning activities, and feedback from intensive formative evaluation is used to guide changes in course design.

One example of a Stanford Learning Lab project exploring at a fine-grained level the interactions of students and technology, is a humanities course entitled "Introduction to Humanities: The Word and the World." In the Fall of 1997 SLL assisted with the piloting of an innovative course that focused on five significant texts in order to introduce freshmen to the methods of inquiry that researchers and scholars in the humanities use. The course consisted of lectures and discussion sections but also featured a Web-based discussion "backbone." The course's web environment provided content, supporting and enrichment materials on all five texts; posed and collected short-answer questions prior to each lecture; and provided both lecture and section forums.

In parallel with the creation of the web "backbone" during the summer of 1997, SLL funded the work of an "Assessment Team" (A-Team), consisting of six core individuals and five advisors. Working in partnership with the teaching and technology teams, the A-Team defined teaching and learning issues, goals, and questions that motivated the investment in Web technologies for the course, and an assessment plan that would address questions being posed by the various stake-holders -- SLL, the teaching team, and the faculty senate.

The assessment plan used a variety of tools (e.g., surveys, interviews, video taping, Web-statistics) that allowed for direct evaluation of utilization and effectiveness of particular elements of the web backbone, the direct addressing of questions posed by the stake-holders, and a broader window into the utilization of technology more generally by the 90 Stanford undergraduates in the course.

Sheri D. Sheppard
Co-Director, Stanford Learning Lab and Professor of Mechanical Engineering
sheppard@cdr.stanford.edu

Michele Marincovich
Director, Stanford Center for Teaching and Learning
marin@stanford.edu

The College of Wooster is a national liberal arts undergraduate institution of 1,700 students in northeastern Ohio. In May 1996, May 1997, and January 1998, the College offered week-long Web tools workshops to a total of 40 faculty members, funded by grants from the College's Hewlett-Mellon Fund for Institutional Renewal. (For more information on Wooster's faculty Web workshops, see http://www.wooster.edu/acs/workshop/.) Following the workshops, dozens of courses were offered that incorporated LISTSERV email discussion groups and Web pages. Partially in response to a requirement of the grants to submit reports on the outcomes of the workshop, we designed and conducted surveys in 1996 and 1997 to gather student responses to the use of these technologies in their classes. Faculty members distributed and collected surveys in class; nearly 100% of the students in the 20 classes surveyed participated for a total of 339 responses.

The instrument gathered background information including class year, the student's rating of their computer skills, and whether the student owned a computer. For LISTSERV email discussion groups the survey asked students to indicate for what functions LISTSERV was used, how students rated the number of messages, how difficult it was, and what impact they believed it had. Likewise for Web pages the instrument asked for what purposes the Web was used, how difficult students found it, and what impact it had. In 1997 we added several questions that asked whether the LISTSERV groups and the Web pages helped in the development of particular skills such as writing, critical thinking, discussion, research and computer skills.

Our results can be summarized as follows:

A possibility for further research would be to investigate how different faculty teaching styles impact student perceptions of the value of these technologies.

A full report on this study is forthcoming in Campus Wide Information Systems (MCB University Press). Philip Harriman Director of Academic Computing pharriman@acs.wooster.edu.

I have also presented sessions on our research at the last two Consortium of Liberal Arts Colleges (CLAC) annual meetings: Web pages from those presentations are available at http://pages.wooster.edu/pharriman/clac97/ http://pages.wooster.edu/pharriman/clac98/ And I will be presenting on this topic at the EDUCOM'98 post-conference session this fall being hosted by Rollins College.

Philip Harriman
Director of Academic Computing
Pharriman@acs.wooster.edu

Virginia Tech has embarked on an impressive number of instructional technology projects over the past few years. For example, the internally-funded Center for Innovation in Learning has sponsored 66 grants to faculty for over $1.5 million. These projects typically involve the use of new technologies in the "transformation" of a course or set of courses. The Instructional Services division, which administers the grants, is providing consultation on assessment and also coordinates the assessment of these and a number of other technology-based initiatives. Until this spring, they relied for quantitative evaluation on a set of student surveys that had been developed internally for a project sponsored by the Alfred P. Sloan Foundation's asynchronous learning initiative. In January, the Flashlight Project (sponsored by Annenburg CP/B and the AAHE) released their Current Student Inventory (CSI), which Virginia Tech adopted and is in the process of integrating into their assessment program. In addition to its solid research base, the CSI should provide the opportunity to compare findings with other institutions who also use Flashlight instruments. Instructional Services is also using qualitative methods, primarily semi-structured interviews of faculty developers and students who are enrolled in the "transformed" courses. Many of the interviews are videotaped so that segments can be used to communicate the results and impact of the course transformations to interested outsiders.

An example of an assessment report (for the Sloan-funded ACCESS project): http://www.edtech.vt.edu/access/ For information about Flashlight: http://www.tltgroup.org/ Virginia Tech: http://www.vt.edu/

C. David Taylor, Ed. D.
Assessment Manager/Instructional Designer Educational Technologies
David.Taylor@vt.edu

At the University of British Columbia, Vancouver, Canada, we are nearing the end of a three-year project studying the costs and benefits of telelearning, that is, the use of computer communication technologies for teaching.

The project is looking at the costs and benefits of telelearning in seven different projects, using seven different software applications for telelearning (HyperNews, Virtual-U, Web CSILE, Lotus Notes, NetMeeting, Visual Page, and one still in development). The cases analyse the costs and the benefits, using my ACTIONS model (access, costs, teaching functions, interactivity and user-friendliness, organizational issues, novelty, and speed). The study is also looking at indirect benefits, such as the opportunity for increased enrolments, campus traffic reduction, and internationalization. Part of the study includes the development of cost models, to predict costs and benefits of different combinations of face-to-face teaching and telelearning for various enrolment targets.

Data for most of the studies have now been collected, and the case studies are currently being written up and should be released between September and December this year. The reports on indirect benefits and costs models are due between January and April 1999. The study is funded by the Canadian Federal Government and is part of a $13 million National Centre of Excellence-Telelearning project headed by Dr. Linda Harasim and Dr. Tom Calvert at Simon Fraser University.

For more information, contact Sylvia Bartolic at bartolic@cstudies.ubc.ca.

Tony Bates
Director, Distance Education and Technology, Continuing Studies
bates@mail.cstudies.ubc.ca

Next Readers Respond Question:

Have you developed or are you in the process of developing Service Level Agreement (SLAs) with the customers/departments that your IT organization supports at your institution? If so, please briefly describe your experience and provide the URLs for the SLAs or related documents that are sharable on your Web site.

Send your response via e-mail to Elizabeth Harris, CAUSE/EFFECT managing editor, eharris@educause.edu.

...to the table of contents

Menubar Imagemap