Developing Effective User Training Programs at the University of Michigan Copyright 1994 CAUSE. From _CAUSE/EFFECT_ Volume 17, Number 2, Summer 1994. Permission to copy or disseminate all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, the CAUSE copyright and its date appear, and notice is given that copying is by permission of CAUSE, the association for managing and using information resources in higher education. To disseminate otherwise, or to republish, requires written permission. For further information, contact Julia Rudy at CAUSE, 4840 Pearl East Circle, Suite 302E, Boulder, CO 80301 USA; 303-939-0308; e-mail: jrudy@CAUSE.colorado.edu DEVELOPING EFFECTIVE USER TRAINING PROGRAMS AT THE UNIVERSITY OF MICHIGAN by Elaine Cousins ABSTRACT: The University of Michigan's computer training program serves over 10,000 faculty, staff, and students each year. Critical components of the program are solicitation and use of user feedback and the partnerships formed with administrative offices, managers, and faculty to ensure their needs are met. Listening to users' needs and comments helps trainers design and improve the training offered, which in turn enables users to leverage the available technology to do their jobs more effectively and efficiently. Designing effective end-user training for computing at the University of Michigan is a result of user involvement, a responsive and talented training staff, and diversified formats for different subjects and audiences. This article provides an overview of the goals and scope of Michigan's training program, focusing on organization, training formats, staff preparation, how users help shape the program, and some of the challenges we are facing in the future. It is intended to share with other training providers and designers some of our experiences, especially ways our program actively solicits user feedback on workshops, and works in partnership with managers and faculty to design training that reflects their needs--two elements we think have contributed most to the program's success to date. Program overview and organization The University of Michigan recognizes the need for all members of the University community to have ongoing access to training in computing technologies. Such training is relied upon each year by approximately 10,000 faculty, staff, and students representing all academic disciplines and administrative units. End-user training is designed to enable these constituents to effectively use the technology available on campus. This might include information about the campus computing environment as well as training in specific technologies. Training is provided in a variety of formats, including: * Regularly scheduled workshops (approximately 650 hours per term and 100 course titles) * Ad hoc workshops by special request of faculty or staff groups * Computer-based tutorials/simulations (commercial products as well as some developed by our own staff) * Special programs for students (residence hall instruction and introductory classes at campus computing sites) * Short presentations as part of special campus-wide events on a particular topic This training is an important component of the User Services program of Michigan's Information Technology Division (ITD). User Services is responsible for training, consulting, documentation, campus computing sites, computer classrooms, publicity, and needs assessment activities. In addition to User Services, ITD has five other units: --University Information Systems (administrative systems development) --Research Systems (computing services for researchers and academic users) --Instructional Technology Systems (instructional uses of technology, including courseware development) --Resource Administration (ITD personnel, space, and budgets) --Network Systems (voice, data, and video networking services) Instructors for workshops are drawn primarily from the User Services staff, but also from other areas of ITD. In particular, we depend upon staff from our Network Systems group to teach networking classes, and upon staff from the Instructional Technology Systems group to teach courses in authoring systems. The education function draws on approximately thirty individuals, representing seven full-time equivalents (FTE), including workshop developers, instructors, a registration and clerical support staff person, program manager, teaching facility network administrator, and a half-time coordinator of the residence hall computer support program. The ITD education manager, who is a part of the User Services project management team, functionally manages the instructional staff, who report administratively to other managers. The main advantage of this "functional management" approach is that it enables us to better draw upon the expertise of the entire organization and select the best person for a particular teaching assignment. We are also able to provide staff with a more varied set of responsibilities. Workshop instructors participate in topic area groups with a coordinator for each area. The topic areas are: * Systems use (Macintosh, DOS, UNIX, and MTS, our academic mainframe operating system) * Administrative data * Electronic mail and conferencing * Database management * Desktop publishing * Graphics * Spreadsheets * Word processing * Statistical computing * Networking and data communications * Authoring systems and instructional technology * New technologies Topic coordinators are charged with the continuous improvement of the curriculum in their areas. They meet with the other instructors in their area and with each other to assure that the curriculum is timely and meets the needs of participants, and that courses in one area complement and support courses in other areas. The entire education group meets monthly to discuss common issues related to teaching and workshop design, and to identify and provide guidance on any problems that arise. Training program curriculum planning Curriculum planning is an ongoing process of the training program, which culminates each semester in an assessment of what classes should be taught, how frequently, and when in the semester each should be offered. To meet deadlines for publication of the course schedule, the curriculum for each semester is finalized approximately two to three months in advance of the start of the semester. In instructor meetings and discussions, care is given to make sure that curriculum changes are thought through, including their implications for the entire program. Instructors of courses in a sequence pay particular attention to how their workshops complement one another. When changes are introduced, we are careful not to duplicate efforts or assume that someone else is covering an important subject. Modifications to classes might be made on the basis of participant feedback or because of changes in the technology or the campus computing environment. Instructors review summaries of workshop evaluations and other sources of data about participant needs. They also stay abreast of changes in commercial software products and new programs and services being developed on campus for which training will be necessary, and work with other consultants to identify common problems of users that could be addressed in training. Other information about user training needs comes from campus-wide forums (both electronic conferences and actual focus group sessions with users) and by working in partnership with faculty, staff, and students in the community. These partnership efforts are discussed later in this article. Training formats While hands-on workshops have been the mainstay of our training program, self-paced, computer-based tutorials/simulations are becoming an increasingly attractive alternative or supplement to workshop training. There are a variety of reasons for this. * Individual conflicts with workshop schedules may make it inconvenient to attend a workshop. With personal tutorials, training is always available "on demand." There is no need to wait for a workshop. * Users of a computer-based tutorial/simulation can work at their own pace. * Unlike printed tutorials, computer tutorials provide corrective feedback when a mistake is made. * Users appreciate the portability and ease of use of computer-based tutorials. Unlike video, which requires a special setup, the computer- based tutorials can be used just about anywhere a computer is located. In the fall of 1992, we introduced commercially available tutorials, for which we acquired campus-wide site licenses. These tutorials cover DOS, Windows, dBASE, Lotus, and WordPerfect. All tutorials are available for use in campus computing sites and for staff and faculty use in their offices. We are additionally requesting that our campus computing sites routinely install the vendor tutorials that accompany the software. We are experimenting with using computer-based tutorials in a workshop setting. In training new users of our academic mainframe system, the computer-based tutorial received very high praise from users in a classroom setting. They liked working at their own pace, always getting feedback or help when needed, and having all their individual questions answered without worrying about disrupting the class or overwhelming new people in the workshop. One instructor, who experimented with a commercial Windows tutorial for part of a workshop, found it hard to get the participants to leave the tutorial for some more traditional instructor-led training. The color and animation were just too engaging! We have developed two computer simulations to introduce new users to services on our academic mainframe, and an additional tutorial is planned to introduce the new mail system that ITD will be implementing in a few months. We also conduct some large-scale lecture/demonstration classes for approximately 100 students, to train large numbers of students within a short period of time. When we initially offered large lecture classes, we found that participants often did not do the recommended hands-on exercises after the lecture and thus reduced the extent of their learning significantly. Our response was to drop this style of instruction. Due to participant feedback, however, we've brought the lectures back, but now include as part of the class a hands-on instructional component. At the conclusion of the lecture, students go with an instructor to four nearby computer classrooms, where they can immediately apply what they've learned--either by working on the "real system" or on a computer-based tutorial/simulation program that provides corrective feedback for mistakes. Working with a student from the Film and Video area, we are producing two short videos for students on how to use Microsoft Word to write a paper. This is our first experiment with video for instruction, and depending upon our experience, we may do more videos for systems being planned for the coming year. The Microsoft Word videos will be available for checkout, shown in campus computing sites, and possibly broadcast on the University's cable television station. As user experience levels grow, we are finding that short, focused presentations can be very effective and are more likely to draw an audience than longer, more comprehensive training sessions. We are also finding that we can reach a sophisticated audience by offering training as part of a larger, campus-wide special event on a particular topic. Last fall we held a day-long event, "Working Smarter with Databases and Spreadsheets." As part of this event, a series of ongoing, very specific half-hour presentations drew a capacity crowd (fifty participants at each half-hour session). This year, a similar event focusing on electronic mail had equal success. It appears that this kind of short lecture/demonstration is just what is needed for an experienced audience. Staffing and preparing new instructors With more than 10,000 registrations each year for our regularly scheduled workshops, this is clearly our primary vehicle for training. The strength of this program is largely the result of a talented and caring staff who take great pride in their teaching. In recruiting new staff, we look for people who not only know their subject, but also have excellent interpersonal skills and the ability to organize their presentations and written materials logically. To prepare new teachers, we developed several strategies: * A buddy system for preparing new instructors. New instructors sit in on classes of an experienced instructor and gradually take on responsibility for solo teaching. * A self-evaluation form for instructors to use when preparing for classes. * Sharing of tips, ideas, and problems at monthly teacher meetings. * An instructor handbook for policies/procedures and a description of our philosophy of "good teaching."[1] * The development, over time, of a very high standard of excellence for both new and old instructors to live up to. That the above strategies work is born out by the consistently high comments our instructional staff earn from the participants in their classes. The most typical set of adjectives that workshop participants use to describe their instructors is "caring, patient, and knowledgeable." Measuring and increasing program effectiveness There are essentially two ways users help us increase our effectiveness in the training program. One is by giving us feedback immediately after classes or in a follow-up telephone survey, and the other is by working with us in partnership. User feedback through evaluations At the end of each workshop, we distribute an evaluation form that collects feedback about the presentation of material, the skills of the presenter, the organization and content of the class, classroom facilities, and anything else participants want to share with us. We also ask about other training that might be desired in the future. Instructors set the stage for a very high return rate by assuring all participants that the evaluations will be read carefully and that they will be the basis for improving the workshop for future participants. There has been much discussion among trainers in the industry about the quality of information that such evaluations yield. Some trainers have dismissed them entirely as being nothing more than "smile sheets" for instructors, which reflect more about the geniality of an instructor than about his or her ability to teach or the appropriateness of the workshop content. While there is, no doubt, some truth to this, the evaluations nonetheless do reveal content and organizational weaknesses of a class and offer a unique opportunity to identify and fix problems in classes before they are offered again. Concerned, however, that relying exclusively on in-class evaluations did not accurately tell us how effectively our classes give people the skills they need, we designed a post-workshop telephone survey conducted about six weeks after a workshop that could tell us if the content of our workshops was meeting actual participant needs, and if they could, in fact, apply the information and skills learned in class on the job. Although telephone surveys reach fewer participants than in-class evaluations, we find them useful in capturing information that the in- class form cannot. Looking at the information we collect from each type of evaluation, we have found the following to be the case: * The majority of participants are actually using the product/information they learned about in training. (There is no way of knowing this in evaluations done at the conclusion of the class.) Those who are not using the product usually indicate that they will be using their new skills soon, when they start a new project. * The workshops cover what people actually need and they do indeed learn the material. (There is no way of knowing this in evaluations done at the conclusion of the class. We can only know about anticipated need or perceived effectiveness of training.) * Some of the people we survey tell us about additional training they need. This gives us the opportunity to describe additional training available or alerts us to needs we are not currently meeting with our curriculum. * The surveys help us understand more about the work in which our participants are engaged and thus enable us to include more relevant exercises and information in our classes. * Participants tell us how they use workshop handouts after class. Although we suspected they were useful as reference materials, for training others in an office, and as a refresher for what was learned, the phone surveys have confirmed this is happening with regularity. * Although we do not specifically ask for feedback about the workshop instructor, participants use this opportunity to give unsolicited feedback on instructors--both positive and negative. Some participants have commented that they were more comfortable bringing up negative aspects of an instructor's presentation in a phone call than they were in a written evaluation form. All participants we have called have been eager to take part in the survey and seemed pleased to hear from us. Although most survey calls take only about five minutes to complete, the phone calls can be a time- consuming activity. Sometimes it is necessary to call back several times to reach someone. Other times it is necessary to make an appointment for a convenient time to call back. Occasionally, there is a respondent who takes this opportunity to share a great deal with the caller and engages in useful dialog for as much as twenty minutes. While we originally thought that all instructors would pair up with other instructors to call participants in each other's classes, in practice it has been a task that instructors rarely do, citing a lack of time as their reason for not doing so. The majority of calls, therefore, have been made by the education manager. Also, now that we've learned that participants may want to talk about specific instructors in the post-workshop telephone call, it seems less desirable to have colleagues calling about each other's classes. We have found that it is important for the person who is calling class participants to have a broad knowledge of ITD services, since the calls are an opportunity to help people find out about other services they may need. We also found that it is helpful, but not critical, for the caller to have some familiarity with the topic of a workshop. While we are pleased with the survey as a means of getting important feedback from our workshop participants, we are not yet convinced we have found the best methodology for conducting the survey. We plan to give more attention to this in the future. Feedback from post-workshop evaluation surveys is shared with individual instructors and topic groups, depending upon the nature of the feedback. Feedback of general interest to all instructors is shared among all instructors at instructor meetings. Partnerships Working in partnership with users to meet their needs is another way in which we offer our services to the community and get useful input from users. Partnerships typically take one of three forms: (1) working with an administrative office to train on a new system they have acquired; (2) working with a faculty member to help his or her students get the skills they need; and (3) working with a departmental administrator or local area network administrator to train his or her staff. In all cases, working in partnership gives us an opportunity to learn more about the larger environment in which we work and the work and needs of those who use our services--the "real world" context in which people use the computer skills we teach. These opportunities often have an effect on the regular classes we teach. As we learn more about the environment of our users, we are better able to make our offerings more reflective of their needs. An example of an administrative office partnership is the training we provided with the University's Development Office to train users on their new mainframe computer system. This involved co-designing and teaching workshops that not only covered the basics of the computer system, but also covered how the system could be put to good use by development officers, and new ways of working in some cases. The computer aspects of the training were the domain of our ITD staff. The more job-related components were the domain of the Development Office. The quality of the training was enhanced by each of us attending to the parts we knew best. An example of partnering with a faculty member is doing a specific workshop at his or her request; we do many of these. Sometimes we make our materials available to teaching assistants and faculty and consult with them on their use. Other times, however, it is best for us to do the training. The workshop might be identical to one we regularly offer, or we might work with a faculty member to tailor the content for the specific needs of the students and a specific class activity. The third type of partnership involves designing training for a departmental group that might be getting new computers, new programs, or a new network installation. Or a departmental administrator or dean might desire to enhance the skills of his or her faculty or staff. In meeting with the key administrators in the department, we learn more about the needs of their group and can modify our standard training as needed. It is important that the administrator who is speaking on behalf of other staff or faculty is, in fact, aware of the actual needs and time constraints for training. We have learned the proverbial hard way that this can not be taken for granted. Sometimes it is necessary to ask an administrator to better identify needs and to assist him in the process. It can be a costly waste of resources if the training does not match what is needed or if the administrator has not paid attention to time and schedule constraints. Some administrators have found that it is better to work out individual training plans with staff and make use of our regular workshops, than to set up a program for their department as a whole. Special training for a group or department works best when a new system or network is being installed, and everyone needs to learn the same thing at the same time. Challenges We face challenges in the coming year related to the economy, the widening gap between experienced and novice computer users, and the changing technology environment, especially distributed computing. Reduced resources One of the issues at the University with which we must contend is reduced resources, coupled with the need to move into new areas of activity. A growth in new activities is unfortunately not always accompanied by a growth in resources. This means that we must find ways to train our existing staff to take on new responsibilities, and we must find a way to let go of ongoing activities. This has meant cutting back some services, to allow more time for staff development. Where we have identified quality commercial training materials, we are beginning to use such materials, so that we can free up staff for new activities. The introduction of computer-based tutorials and audio-cassette training tapes in some cases has allowed us to reduce the number of training classes we must offer in specific areas and in other areas has allowed us to drop classroom training entirely. As services are considered for reduction, it is important for us to engage in an active dialog with users about what the potential service cuts will mean to each of us and what other options there might be. We must be very aware of the real needs of our community, since we literally can not afford to offer services that are not needed, nor can we afford to overlook those that are. Widening gap in skill level Another challenge we face is the widening gap of skills among participants in classes. As more and more of our users become sophisticated computer users, the audience we serve is increasingly more differentiated. "One-size-fits-all" classes will suit increasingly fewer participants. It is important to recognize this trend and to provide learning opportunities that can be customized for different participants in a single class. It will not always be practical to offer several classes for different audiences. The person learning how to use Microsoft Excel as her first spreadsheet program, for example, has different needs and will learn differently from a person switching from another spreadsheet program. One way we are attempting to meet this challenge is by designing workshops that can accommodate a variety of learner needs. This might mean building time for independent work into a workshop, so that an instructor can work with students who need extra help--whether remedial or advanced. This independent time might be spent working on printed exercises, or it might be structured time within a class to work on a computer-based tutorial. Another approach might be to write a variety of exercises that are tailored to the needs and experience levels of different groups. Yet another tactic might include writing exercises in different ways for different groups. For example, steps might be spelled out in detail for some groups, while on another page the exercise might require participants familiar with the subject to apply more of what they know toward the exercise. A related challenge is reaching the most sophisticated users. Workshops might not be best for this audience, but rather short talks, demos, articles, and tips may be the best mechanisms. We've had good success experimenting with short, to-the-point presentations and articles in newsletters for this group. More is yet to be done, and more information from this growing community will be critical. New models for distributed computing support Finally, we are being challenged to develop good models for training in a distributed computing environment. Increasingly, we expect to see some departments and units on campus providing services for their own staff, students, and faculty, and possibly for the campus at-large. What role should a centralized training service play in this new environment? Do we train end users? Do we train other trainers? Do we provide training design advice? Our challenge will be to explore with new service providers the best way to work together for our mutual benefit and for the benefit of the campus. None of us can afford a proliferation of duplicate services; nor can we afford to omit critical services because of faulty communication with each other. Only by working together with new service providers to understand our common and unique audiences, audience needs, and our strengths and skills as service providers, will we develop a good working relationship and define the right models for training in a distributed computing environment. To the extent that distributed systems encourage more collaborative work relationships throughout campus, we look forward to enabling our community to not only work effectively with information technologies, but to work better together because of it. ======================================================================== Footnote: 1 Introduction to Teaching Workshops: Instructor's Handbook (CSD0894) may be ordered from the CAUSE Information Resources Library. It includes the instructor self-evaluation form, the workshop evaluation form, and the post-workshop telephone evaluation survey referenced in this article. For information, call 303-939-0310 or send e-mail to: orders@cause.colorado.edu. ======================================================================== Elaine Cousins is User Services Manager of Education within the Information Technology Division of the University of Michigan. She has been actively involved in end-user support at Michigan since 1984 and as manager of education since 1987. She has an MA in music literature from Eastern Michigan University and a BA in anthropology from the University of California, Berkeley. Prior to joining the staff of the University of Michigan, she worked for the international high school exchange program, Youth for Understanding. ************************************************************************ Developing Effective User Training Programs at the University of Michigan