How Do We Know IT's Working?
Assessing The Academic Effectiveness Of Information Technology
Access And Use
Peter V. Deekle, Ed. D.
College Librarian
Wheaton College
26 East Main St.
Norton, MA 02766
Tel. 508/286-8225 � Fax 508/286-8278
[email protected]
Kathleen Ebert-Zawasky, Ed. D.
Director, Academic Computing
Wheaton College
26 East Main St.
Norton, MA 02766
Tel. 508/286-3926 - Fax 508/286-8278
[email protected]
Cause 98, Wednesday, Dec. 9, 1998 - 2 p.m.
Wheaton College benefits from a 10-year progressive history in its curriculum-related use of electronic information resources. The faculty has endorsed the integration of electronic information technology into the academic program of the College in a manner consistent with Wheaton�s emphasis on learning as an evaluative and reflective process. A divergent assortment of pedagogically-based technology projects is underway stimulated by faculty development workshops, competitive funding opportunities, faculty partnerships with library and computing staff, and, more recently, a faculty mentor program. This paper describes how early project evaluations have evolved into a more cohesive assessment of how this pedagogical endeavor has changed teaching and student learning.
How Do We Know IT's Working?
Although this is Wheaton College�s information technology story during the past decade (1988-1998), the authors have told it with the intention of universal application and relevance to any liberal arts undergraduate college. The technology story for every institution begins much earlier than ten years ago, but the past decade at Wheaton (and at other colleges, too) has been marked by significant events which have helped to define a context for the thoughtful application and assessment of information technology. Wheaton�s individual story, therefore, reflects several themes common among many institutions of higher learning.
In 1988 New England�s first women�s college was preparing to become coeducational. It was also planning for a campus wide telecommunications network which would unite all on-campus members of the academic community. This information technology story begins with a consciousness of the College�s academic and pedagogical traditions and distinctions. In telling this story of the last decade the authors began with the premise of assessing the success from the viewpoint of the information service provider. It soon became apparent, however, that Wheaton�s significant achievements in integrating technology broadly throughout its curriculum have had as much to do with the sense of ownership and efforts of its creative and energetic faculty, as with the dedicated pursuits of academic computing and library professionals. IT professionals may well have prepared the way, but the faculty have ensured the success of information technology in teaching and learning.
For several years prior to 1988 Wheaton had recognized the need for leadership in information technology, creating an academic computing center to prepare the infrastructure and the faculty for the thoughtful use of computing and interactive media. Wheaton�s library had anticipated the benefits to faculty and students of remote access to its catalog of holdings and was preparing by 1988 for the eventuality of an online public access catalog.
Like other liberal arts colleges, Wheaton emphasizes its teaching mission above all other academic pursuits. Information technology was embraced as a promising tool for the advancement of teaching and learning at the College. In 1988-89 a grant from the Charles Dana Foundation enabled the Library and librarians to partner with the faculty in an ongoing program for the deliberate integration of electronic resources into the curriculum, the goal of which was to promote information for students. This program, begun ten years ago, has continued in the form of (1) library user instruction programs at the entry, mid, and capstone levels of the curriculum and (2) semi-annual faculty technology workshops which have increasingly involved collaboration with academic computing in carrying out its goals.
The promise of electronic access to the Library�s catalog was realized in 1993-94 when the prevailing CD-ROM catalog became an OPAC (online public access catalog). 1994-95 extended the networking of academic and office sites throughout the campus to individual students in the campus residence halls by means of a grant from the Davis Educational Foundation.
The completion of the campus wide telecommunications network within the past five years encouraged both the Library and Academic Computing to provide an increasing variety and number of electronic information resources and application programs for remote access to faculty and students. Complementing the networking of the campus was a program consolidation grant in 1995-96 from the Andrew W. Mellon Foundation which supported the installation of a video distribution network to twenty-two primary classrooms and other academic sites on the campus.
The existence and use of Wheaton�s campus network has enriched the partnership between the Library and Academic Computing over the past decade. The faculty and student constituencies of educational information technology users have offered compelling justification for these two agencies to collaborate on the design and equipment of electronic classrooms, campus wide technology delivery and support services, and the shared cost of acquiring and collaborative distribution of both instructional and applications software. Meanwhile the Library has continued to expand its licensing of electronic resources for remote access, both on and off campus.
Assessing the impact of information technology on the academic program has been a recurring Wheaton theme throughout the past decade. The initial information literacy program grant included pretest and post-test assessments by librarians of student skills. The semi-annual faculty development workshops have fostered an increasing number of technology centered "course transformation" proposals, each with specific provisions for the assessment of the impact on student learning.
The remainder of this paper described several models and means which have grown from the deliberate information technology planning at Wheaton, and which the authors believe have a general appeal to other liberal arts colleges.
PROGRAM DESCRIPTION
The heart of our program focuses on the role of computing and information technology in the educational experience of our students and consists of three progressive components.
First Component: Learning in a Networked Community-An Introduction
Students will develop computer skills throughout their careers at Wheaton and beyond, but the most intense part of this phase should be at the beginning of their careers so that they can begin immediately to use computers as productively as possible.
Recommendations:
Students will learn how to use email and drop boxes, use the Wheaton library OPAC, be introduced to word processing, spreadsheet, and database software. Students may be encouraged to develop a "product" for their First Year Seminar (possibly getting information from the World Wide Web and integrating it into a document using presentation software). Students may take placement and assessment exams by computer.
The First Year Seminar Library component already introduces students to using OPAC and other computer-related reference skills. We believe that this component also gives us an opportunity to give our students a more sophisticated introduction to the powers and limitations of the Internet.
Second Component: Communicating and Computing Across the Curriculum-Day One through Year One
It is important that the skills described above be exercised as soon as possible, or they will be lost. Even more importantly, we believe that faculty teaching introductory courses will discover that they can teach more effectively now that they can count on all students having a solid computer skills background.
Recommendations:
We expect that, as a result, students should recognize the role of computer technology as a material part of their learning experience and should be able to take fuller advantage, both inside and outside the classroom, of their membership in an electronically-enhanced learning community.
Third Component: Creative Contributions to a Networked Community
The College should offer students a broad range of curricular and extra-curricular opportunities to continue to develop their computer skills creatively, and, as they become more proficient, independently. Such opportunities should be an integral part of courses in every area of the curriculum, and should be made widely available outside the classroom as well.
Instead of imposing an additional requirement on students, we strongly urge departments to incorporate computing into upper-level courses as a natural part of their curriculum.
Recommendations:
In summary, there are really two different but related kinds of advanced uses of computers-one that teaches students to work with state-of-the-discipline programs and products in a classroom setting, and one that allows students to go even further and create material of campus-wide- and even world-wide value.
IMPLEMENTATION OF THE PROGRAM
Implementation of the program is guided by the newly transformed faculty committee, the "Library, Technology and Learning Committee" or LTLC. Specific duties of the committee are:
1. to invite, develop, and support the implementation of initiatives that advance curricular and pedagogical uses of library resources and technology and enhance the quality of teaching and learning at the College;
2. in collaboration with the College Librarian and the Director of Academic Computing, to review, develop, and support academic services and policies that advance appropriate uses of library resources and technology in the academic programs of the College;
3. to facilitate the ongoing collaboration between the Library, Academic Computing Center and Information Technology and Services.
The LTLC actualizes these goals primarily by overseeing the design and delivery of semi-annual faculty technology workshops and by eliciting faculty technology proposals. Typically, a one-day workshop in January and a two-day workshop in May are designed by staff from Academic Computing and the Library on carefully selected, pedagogically significant topics. During the workshops, the faculty are encouraged to submit proposals for incorporating new uses of technology into their teaching.
Conducting Faculty Workshops
Faculty technology workshops organized by collaborative efforts of Academic Computing and the Library over the last few years include:
Visual Elements in Teaching and Learning: Trends, Tools and Techniques (May 21-22, 1998);
Two Faces of Technology: A Look at the Effectiveness of Technology in Teaching and Its Potential in Classroom Assessment Techniques (January 14, 1998);
Technology and Learning Workshop: Exploring Alternate Forms of Interactivity (May 21 and 22, 1997); and,
Pedagogical Issues of Technology in Writing and Text Presentation (January 13, 1997.)
Encouraging Faculty Technology Projects
Corresponding with each faculty technology workshop is a call for proposals. Faculty members are encouraged to design projects that thoughtfully include the use of technology in their teaching. The Committee suggests using the following format for proposal development:
A. Brief report on previous technology projects
B. Pedagogical goals and problems that the current proposal is addressing
C. Specific technological strategies that will be used to address these issues
D. Needs (estimated time required, mentor, equipment...)
E. Assessment plan
F. Dissemination plan
Upon review of the proposals, the LTLC determines the appropriateness of the project, funding levels, support to be provided and makes decisions on stipends and awards. The instructor is also asked to submit a final report based on guidelines which include parallel elements.
INITIAL PROGRAM ASSESSMENT
Integration of this initiative across the curriculum is evidenced by the involvement of faculty from all areas of the Liberal Arts. Between 50 and 60% of the full-time faculty typically attend the Faculty Technology workshops and 65 projects have been endorsed and funded by the LTLC since May, 1995. The breadth of involvement across the three main areas of the curriculum can be seen in Figure 1.
A clearer picture of the impact, success and depth of the program emerges if we organize technology projects by categorizing them into the original three components of the program and discuss individual assessment protocols. Goals for Component One (Introduction to Learning in a Networked Community) are, however, targeted by members of the community during Orientation period and before as well as by Faculty and Librarian collaboration in First-Year Seminar classes.
First Component: Learning in a Networked Community-An Introduction
The first and most fundamental component of the program is accomplished by providing a variety of opportunities for students to interact with the technology when they first arrive at the College. For example:
A. Approximately 20% of the incoming class attends one-day summer workshops. Many students are introduced to word processing (for academic papers), spreadsheet, and database software and the Wheaton Residential Network (ResNet) before the semester begins. Student employees and professional staff from Academic Computing and the Library design activities and exercises to acquaint incoming students with the resources and electronic services available.
B. Orientation, including pre-orientation programming, has been modified to include basic computer skills training. Students are introduced to navigating the campus network and the Internet within a few days of their arrival on campus. They become familiar with the use of their email accounts and personal dropboxes. Activities during Orientation, organized and executed by the professional staff, are supported and augmented by Student Preceptors who are supervised by the Academic Dean and serve as peer advisors for incoming students. Within a few weeks, all first-year students are familiar with the network resources of the community.
C. Students preparing to study foreign language courses take online placement exams during Orientation. Results are emailed to faculty and placement into appropriate courses takes place smoothly and quickly.
D. The Library’s introduction of students to networked information resources begins with information literacy as part of their First Year Seminar course. The model (below) represents one of those courses.
From the inception of the program, there has been a continual effort to assess the effectiveness of Library instruction or users of networked resources. Beyond merely quantifying actual resource usage, however, the Library’s 1989 initial model for its information literacy program included both pre- and post-testing of students in technology-centered user instruction sessions. The College's Technology report has borrowed from this early instruction and assessment model, particularly for its level one student proficiency (embodied in the College’s First Year Seminar).
The present model for assessment of the FYS Library component consists of questionnaires distributed to librarian instructors and to FYS faculty which serve to qualitatively evaluate the information literacy assignment, planning process, classroom teaching methods, and student learning outcomes. A single question on the FYS evaluation form distributed to students at the end of the course serves to assess their perceptions of the Library component's usefulness in enhancing their information literacy skills.
In the past three years this pivotal first semester course (which introduces many of the research and information literacy skills upon which liberal education depends) has grown from a single class session to, often, two sessions devoted to information literacy. The College finds itself, now, at a critical juncture where the successful assessment of information literacy depends not only on models for use by librarians, but also a process which actively engages faculty and students participants.
The authors now look beyond the First Year Seminar to assessment models for other technology-centered teaching and learning experiences which Wheaton librarians, increasingly, are supporting.
An example of a faculty technology project designed for First-Year Seminar students in Economics follows:
Use of simulation models and presentation software. (Gordon Weil, Professor, Economics)
Students in my First Year Seminar formed teams and used the simulation program International Futures Simulation (IFS) to learn about the operation of complex system. The exercises began when they changed a parameter in the model then observed, and considered the impact on several economic variables. They then exported the variable values (under the base case and for the changed model) into the spreadsheet program Excel where they made them into graphs. Next they exported the graphs into PowerPoint to make slides of the graphs. Finally they presented the slides to the class and discussed the meaning of graphs. The project went remarkably well. I think they learned some real economics through our class discussions of why the variables they observed changed as they had. I'm certain that they learned a good bit about PowerPoint, and I'm sure several will make use of it in the future when they make classroom presentations.
Second Component: Communicating and Computing Across the Curriculum-Day One through Year One
In achieving Component Two, introductory courses provide opportunities for students to apply the skills they have learned earlier. In addition, the faculty strives to expose students to representative applications in specific disciplines and encourage students to develop skills in those applications.
Examples of abstracts from three project reports describe applications and at this level:
TakeNote! For Students and Faculty. (Alex Bloom, Professor, History)
The aim of this project was to integrate the use of the program TakeNote! into my senior seminar as well as to create a guide and serve as a mentor for faculty wishing to use this program. This involved creating an exercise to teach TakeNote!, doing the exercise with my senior seminar, offering a faculty workshop on TakeNote!, and making the exercise available to anyone via the Wheaton Server.
Email Submission of Reading Assignments in Calculus I and II. (Tommy Ratliff , Asst. Professor, Math/Comp. Science)
Very few calculus students have any experience reading a mathematics text before the material is discussed in class. In order to help students develop this skill, I gave specific reading assignments for most class meetings which included several basic questions that they should be able to answer after completing the reading. The students emailed their responses to me before class, which gave me valuable insight to the students level of understanding before class. The student response rate to the assignments was very good, and most students indicated on supplemental evaluations that the reading assignments were a worthwhile part of the course.
Electronic discussion. (Susan Dearing, Writing Coordinator, English)
The following report describes my experience with adding an e-mail discussion list to Education 025, "Introduction to Tutoring Writing," in the fall of 1997. My goal was to provide a vehicle for student interaction outside of class and to engage students in an additional form of informal writing as part of the writing-intensive component of the course. Although student participation in and evaluation of the listserve was not uniformly positive, I was more than satisfied with the results of the experiment given my own sense of its efficacy in stimulating student interaction. Two-thirds of the class reported an essentially favorable response to this component of the course.
Third Component: Creative Contributions to a Networked Community
All major programs seek to add significant discipline-specific computer applications and content to at least one 200- or 300-level course. Students are expected to use the computer as a creative tool, reflecting the state of the art in their respective field. Students are encouraged to pursue advanced or independent study using computer technology in the field.
Projects at this level include:
Construction of visual archives for teaching and research and developing the model for anthropology/sociology research on the Internet. (Department of Sociology/Anthropology )
The Department received a grant in the summer of 1997 to begin implementation of a long-term plan to integrate learning and technology in curriculum revision to meet the goals and objectives defined in our mission statement. The longitudinal plan for curriculum transformation includes four components: construction of a visual archives for teaching and research; development of a model for upper-level majors to use for research on the Internet; the development of a model for making social indicator data bases accessible and interactive for use in introductory and other courses; and construction of guidelines to improve student presentations utilizing a multi-media format.
Internet Resources in Chemistry (Elita Pastra-Landis, Professor, Chemistry)
The goal is to use a portion of the Chemistry Senior Seminar this year to create a new section in the Chemistry Department home page that will contain links to Internet Resources in Chemistry. The linked sites will be carefully examined and then described and critically evaluated for the potential user: an " annotated electronic bibliography" is what I have in mind, but it would be best if the descriptions are written by chemistry students for chemistry students.
In an earlier project (report not currently available), Prof. Tim Barker assigned advanced Astronomy students research projects which became reading and teaching materials for students taking the Introductory Astronomy course. They created and published the materials making the subject area more accessible to less experienced students.
CURRENT ASSESSMENT METHOD
Individual Faculty members assess their own projects and report them to the committee. Assessment reports vary from quantitative to anecdotal. Interesting examples include:
Conversion of Class Lecture in PowerPoint presentation. (Barbara Brennessel, Professor, Biology)
Now that I have seen the course evaluations for Bio 303 (Biochemistry) I can add the following information with respect to assessment of my project: N.B. class size = 8 students, all completed the evaluations.
On a scale of 1-5, with 5 being the highest, I asked students how valuable the PowerPoint presentation was in terms of their learning the biochemical principles. 4/8 rated it 5, 3/8 rated it 4 and 1/8 rated it 3.
When I asked students if they would like access to the presentation via a course folder or other mechanism, 6/8 responded yes.
When I asked if it would be useful to have other topics covered in a similar format, 4/8 replied "yes", 1/8 replied "no", and 1/8 replied "possibly". In addition, students wrote comments about the PowerPoint presentation such as "organized", "concise", "visual aids are useful".
Internet Resources in Chemistry (Elita Pastra-Landis, Professor, Chemistry)
Evaluation of this component of the course by the students was already carried out in the course evaluations of the spring term. Students commented that the systematic searching experience was invaluable and great fun. They also enjoyed the fact that in many cases they were finding some material about which I was entirely ignorant, the "Alchemy" website being a case in point.
Introduction to Women's Studies Internet Resources Project (Kathryn Tomasek, Asst. Prof. History)
I have no formal way of assessing the success of this project. I do, however, think that the project was a success because it gave students and discussion leaders an opportunity to learn about resources available on the Internet, to think about evaluating webpages, and to participate in a collaborative writing project that resulted in a webpage that adds to the existing Women's Studies webpage.
CONCLUSION
The authors have described a decade of history for an IT program which continues to develop. It began through the deliberate leadership of information service providers, most notably academic computing professionals in collaboration with librarians. Its present state and continued success have been in large measure due to the contributions of individual faculty and their work as a technology planning and implementation group. A similar variety and curricular breadth at the third or capstone level of the curriculum must complement the variety of successful models of instruction for the first and second levels of the program. Throughout the course transformation or development process and during the implementation of the instruction model, assessment by the individual faculty member and by those associated with IT support is of central importance.
The Library, Technology and Learning Committee is currently considering a more cohesive approach to assessment of the technology program. Goals for this year include refinement of the proposal guidelines suggesting more explicit learning goals and development of an assessment tool that would facilitate project comparison and compilation of evaluative information. Preliminary indications suggest that we are doing very well in accomplishing Components One and Two but that we may want to focus our attention now on support of Component Three projects (the creative contributions to the networked community) or perhaps redefine or broaden the meaning of participation at this level. Enhancement of the Faculty Mentor program that evolved as a support structure is likely to accelerate future developments.
References
Adalian, Paul T., Jr., , Irene M. Hoffman and Ilene F Rickman, "The Student-Centered Electronic Teaching Library: A New Model For Learning," Reference Services Review, 25:3-4 (1997): 11-22.
Angelo, Thomas A. and K. Patricia Cross. Classroom Assessment Techniques: A Handbook For College Teachers, 2d ed. San Francisco: Jossey-Bass, 1993.
Banta, Trudy, et al, eds. Assessment in Practice: Putting Principles To Work On College Campuses.. San Francisco: Jossey-Bass, 1996.
Brown, George. Assessing Student Learning in Higher Education. London: Routledge, 1997.
Cross, K.Patricia., & , M.H. Steadman. Classroom Research: Implementing the Scholarship of Teaching. San Francisco: Jossey- Bass, 1996.
Davidson, Jeanne R. "Computer Technology: Pandora’s Box or Toolbox?" Research Strategies, v. 12 (Summer 1994), 182-6.
Falk, Dennis R. and Helen L. Carlson. "Learning to Teach with Multimedia," THE Journal, 20:2 (Sept. 1992), 96-101.
Gardiner, Lion F., Caitlin Anderson and Barbara L Cambridge, eds. Learning Through Assessment: A Resource Guide For Higher Education . Washington, DC: American Association for Higher Education AAHE Assessment Forum, 1997.
Lloyd, L. (ed.) Technology and Teaching.. Medford, NJ: Information Today, 1997.
Martorana, Janet and Carol D. Doyle " Computers on, Critical Thinking Off: Challenges of Teaching in the Electronic Environment," Research Strategies, v. 14 (Summer 1996), 184-91.
Pascarella, E. T. and P. T. Terenzini. How College Affects Students. San Francisco: Jossey-Bass, 1991.
Prestamo, Anne M. Tubbs. "Development of Web-Based Tutorials For Online Databases," Issues in Science and Technology Librarianship, no. 17 (Winter 1998).
Smith, Nathan M., Mary I Piette and Betty A Dance. "Effective Computer- Based/Hypermedia Instruction for Libraries: The Design, Evaluation, Implementation and Challenge!" in Working with Faculty in the New Electronic Library. Pierian Press, 1992.
Figure 1. Supported Technology Projects Showing Breadth of Program
Humanities |
Social Sciences |
Science and Math |
First Year Seminar |
First Year Seminar |
First Year Seminar |
Art |
Economics |
Astronomy |
Classics |
History |
Biology |
English |
Psychology |
Chemistry |
Religion |
Sociology/Anthropology |
Mathematics |
Foreign Languages French German Hispanic Studies Italian |
Computer Science |