Developing Effective User Training Programs at The University of Michigan |-------------------------------------| | Paper presented at CAUSE92 | | December 1-4, 1992, Dallas, Texas | |-------------------------------------| DEVELOPING EFFECTIVE USER TRAINING PROGRAMS AT THE UNIVERSITY OF MICHIGAN Elaine Cousins User Services Education Manager, ITD Consulting and Support Services The University of Michigan Ann Arbor, Michigan 48109 Elaine_Cousins@um.cc.umich.edu ABSTRACT Designing effective end-user training at Michigan is a result of user involvement, a responsive and talented training staff, and diversified formats for different subjects and audiences. This session is designed to acquaint training providers and designers with some of the ways in which Michigan actively solicits user feedback on workshops and works together with managers and faculty to design training that is reflective of their needs. This presentation will give an overview of the University of Michigan's computer training program which serves over 10,000 faculty, staff, and students each year. The goals and scope of Michigan's program will be described and the ways in which user feedback is solicited and used will be explored. Additional topics covered include curriculum planning, the role of non-classroom training, measuring program effectiveness, preparation of new instructors, and partnerships with administrative offices, managers, and faculty. INTRODUCTION This paper will introduce the end-user training program at The University of Michigan by focusing on organization, training formats, staff preparation, and how users help shape our program. It is hoped that by telling a little of the Michigan "story," others will have the opportunity to benefit from our experience as well as their own. Overview of the Michigan Program End-user training is designed to enable faculty, staff, and students to effectively use the technology available on campus. This might include information about the campus computing environment as well as training in specific technologies. Training is provided in a variety of formats including: --Regularly-scheduled workshops (approx. 650 hours per term; approx. 100 course titles). --Ad-hoc workshops by special request of faculty or staff groups. --Computer-based tutorial/simulations (commercial products as well as some developed by our own staff). --Special programs for students (Residence Hall instruction and introductory classes in the campus computing sites). --Short presentations as part of special campus-wide events on a particular topic. Information technology services are the responsibility of the Information Technology Division (ITD). The following units comprise ITD: --University Information Systems (administrative system development) --Research Systems (mainframe and supercomputing services for researchers and academic users) --Instructional Technology Systems (campus computing sites, computer classrooms, and instructional uses of technology including courseware development) --Consulting and Support Services (training, consulting, documentation, and planning services; publicity and needs assessment activities) --Resource Administration ( internal personnel and budgets) --Network Systems (voice, data, and video networking services) Instructors for workshops are drawn primarily from the Consulting and Support Services staff, but also from other areas of ITD. In particular, we depend upon staff from our Network Systems group to teach networking classes and upon staff from the Instructional Technology Systems group to teach courses in authoring systems. The ITD Education Manager is a part of the Consulting and Support Services (CSS) management team and functionally manages the instructional staff although they report administratively to other managers. The main advantage of this "functional management" approach is that we're better able to draw upon the expertise of the entire organization and assign the best person to a particular teaching assignment. We're also able to provide staff with a more varied set of responsibilities. The functional Education group of approximately 30 instructors is divided into topic areas with a coordinator for each area. The topic areas are: --Systems Use (Macintosh, DOS, UNIX, and MTS (Michigan Terminal System, our academic mainframe operating system) --Administrative Data --Electronic Mail and Conferencing --Database Management --Desktop Publishing --Graphics --Spreadsheeting --Word Processing --Statistical Computing --Networking and Data Communications --Authoring Systems and Instructional Technology Topic Coordinators meet once per month as a group and are charged with the continuous improvement of the curriculum in their areas. They meet with the other instructors in their area as needed throughout the year. The entire Education group meets monthly to discuss common issues related to teaching, workshop design, and to identify and provide guidance on any problems that arise. Other staffing needs are met by staff from elsewhere in the Consulting and Support Services group -- all reporting functionally to the Education Manager. These include documentation staff for design of a workshop handout style and editing of publications, clerical support for training materials duplication, network administrators for maintenance of the teaching labs and networks. An additional full-time registration clerk, who reports to the Education Manager, handles program registrations, billing, and tallying of program evaluation data. Curriculum Planning Curriculum planning is an ongoing process which culminates each semester in an assessment of what classes should be taught, how frequently, and when in the semester each should be offered. In order to meet deadlines for publication of the course schedule, the curriculum for each semester is finalized approximately two to three months in advance of the start of the semester. In instructor meetings and discussions, care is given to make sure that curriculum changes are thought through with implications on the entire program. Instructors of courses in a sequence pay particular attention to how their workshops complement one another. When changes are introduced, we're careful not to duplicate efforts or assume that someone else is covering an important subject. Modifications to classes might be made on the basis of participant feedback or because of changes in the technology or the campus computing environment. Instructors review summaries of workshop evaluations and other sources of data about participant needs. They also stay abreast of changes in commercial software products, new programs and services being developed on campus for which training will be necessary, and work with other consultants to identify common problems of users that could be addressed in training. Other information about user training needs come from campus-wide forums (both electronic conferences and actual focus group sessions with users) and by working in partnership with faculty, staff, and students in the community. For more about partnership efforts, see page 8. Training Formats While hands-on workshops have been the mainstay of our training program, self-paced computer-based tutorial/simulations are becoming an increasingly attractive alternative or supplement to workshop training. There are a variety of reasons for this, including: --Individual conflicts with workshop schedules may make it inconvenient to attend a workshop. --With personal tutorials, training is always available "on demand." There is no need to wait for a workshop. --Users of a computer-based tutorial/simulation can work at their own pace. --Unlike printed tutorials, computer tutorials provide corrective feedback when a mistake is made. We have developed two computer simulations to introduce new users to services on our academic mainframe . The first tutorial to train users of an administrative system is planned for the spring/summer term in 1993. This semester, we introduced commercially available tutorials for which we acquired campus wide site licenses. These tutorials cover DOS, Windows, dBASE, Lotus, and WordPerfect. All tutorials are available for use in campus computing sites andavailable for staff and faculty use in their offices. We are additionally requesting that our campus computing sites routinely install the vendor tutorials that accompany the software at the sites. We are experimenting with using computer-based tutorials in a workshop setting. In training new users of our academic mainframe system, the computer-based tutorial received very high praise from users in a classroom setting. They liked working at their own pace, always getting feedback or help when needed, and having all their individual questions answered without worrying about disrupting the class or overwhelming new people in the workshop. While we were initially concerned that users might not like the computer-based tutorials, this concern has turned out to be seldom encountered. One instructor who experimented with a commercial Windows tutorial for part of a workshop found it hard to get the participants to leave the tutorial for some more traditional instructor-led instruction. The color and animations were just too engaging! In the past, we also experimented with lecture/demonstration classes to train a large number of new mainframe users within a short period of time. The effectiveness of these classes, however, was severely reduced if the participants didn't follow up with a hands-on experience soon after the lecture. Our experience was that participants often did not do the hands-on exercises after the lecture and thus reduced the extent of their learning significantly. As user experience levels grow, we're also finding that short, very- focused presentations can be very effective and more likely to draw an audience than longer, more comprehensive training sessions. We're finding that we can reach a sophisticated audience by offering training as part of a larger campus-wide special event on a particular topic. This fall, we held a day-long event "Working Smarter with Databases and Spreadsheets." As part of this event, a series of ongoing very speciifc half-hour presentations drew a capacity crowd (50 participants at each half-hour session). It appears that this kind of short lecture/demonstration is just what is needed for an experienced audience and we'll be exploring scheduling more of these in the winter term. Staffing and Preparing New Instructors With 10,000+ registrations for workshops, this is clearly our primary vehicle for training. The strength of this program is is largely the result of a talented and caring staff who take great pride in their teaching. In recruiting new staff, we look for people who not only know their subject, but also have excellent interpersonal skills and the ability to organize their presentations and written materials logically. To prepare new teachers, we developed several strategies: --A buddy system for preparing new instructors. New instructors sit in on classes of an experienced instructor and gradually take on responsbility for solo teaching. --A self-evaluation form for instructors to use when preparing for classes. (See copy of form below.) --Sharing of tips, ideas, and problems at monthly teacher meetings. --An instructor handbook for policies/procedures and a description of our philosophy of "good teaching." Copies of this handbook are available upon request. --Gradually, over time, created a very high standard of excellence or new instructors (as well as old) to live up to. That the above strategies work is borne out by the consistently high comments our instructional staff earn from the participants in their classes. Caring, patient, and knowledgeable are the most typical set of adjectives that workshop participants use to describe their instructors. Instructor Self-Evaluation Form (reprinted from the Instructor's Handbook): --Monitoring Workshop Quality: Instructor Self-Evaluation Form The following self-evaluation form was developed by ITD-CSS workshop instructors and reflects the qualities we think are important for successful workshops. Please use this form as a yardstick to measure your own performance and we invite your feedback on the form. --Classroom Environment Ö Before the students arrive, is the classroom neat and the temperature comfortable? Ö Have you checked to make sure that all equipment is working? --Clarity of Presentation -Voice projection: Ö Can people hear you at all times throughout the classroom? Clear directions and instructions: Ö Are your directions to students clear? Ö Do they know what they are supposed to be doing as you progress through the workshop? -Clear explanation of concepts: Ö Do you explain new material clearly? Ö Do you define new terms when you use them? Ö Do you choose analogies that can make concepts easier to understand? --Rapport with Students Ö Are your interactions with students, warm, hospitable, personable, friendly as opposed to rude and abrupt? Ö Are you patient with your students? Ö Are you supportive and helpful as opposed to condescending and arrogant? Ö Do you convey enthusiasm about the subject matter? --Workshop Organization -Introduction: Ö Does your workshop introduction set the appropriate foundation for the workshop? (Goals, objectives, prerequisites, format, seating arrangements) -Content Presentation: Ö Does your workshop progress logically? Ö Does the structure allow for an appropriate balance between lecture and participant involvement? Ö Are the exercises well thought out and able to be completed in the time allowed? Ö Is the timing of each section appropriate to its importance and complexity? Ö Does the pace reflect the experience of the target audience? Are you able to accommodate faster and slower learners? -Conclusion: Ö How effective is your closing statement(s) in tying together all elements of the workshop? Ö Are additional resources mentioned? --Handling Problem Situations Ö How effective are you at handling questions beyond the scope of the class? Ö How are you handling questions that you do not have the expertise to answer? Ö How effectively are you handling students who can't keep up with the pace of the class? Ö How well are you handling latecomers? Ö How well are you handling participants without necessary prerequisites? --Outstanding Qualities of Presentation Ö What are you doing that's working exceptionally well for you? analogies? making people comfortable? exercises that work well for slower and faster learners? a strategy for involving participants in lecture/demonstrations? making people laugh and enjoy themselves? Please share these at group meetings or in the staff conference. In the next two sections, we will discuss the ways in which working with our users and listening to their needs and comments help us design and improve classes. There are essentially two ways in which our users help us increase our effectiveness. One is by giving us feedback immediately after classes or in a follow-up telephone survey and the other is by working with us in partnership. Measuring Program Effectiveness Our workshop evaluation form, which is distributed to each participant after each class, asks about the presentation of material, the skills of the presenter, the organization and content of the class, classroom facilities, and anything else participants want to share with us. We also ask about other training that might be desired in the future. Instructors set the stage for a very high return rate by assuring all participants that the evaluations will be read carefully and that they will be the basis for improving the workshop for future participants. There has been much discussion among trainers about the quality of information that such evaluations yield. Some trainers in the industry have dismissed them entirely as being nothing more than "smile sheets" for instructors which reflect more about the geniality of an instructor than about his or her ability to teach or the appropriateness of the workshop content. While there is some truth to this, no doubt, the evaluations do reveal content and organizational weaknesses of a class and offer a unique opportunity to identify and fix problems in classes before they are offered again. Concerned, however, that relying exclusively on workshop evaluations did not accurately tell us how effectively our classes give people the skills they need, we decided to to do a follow-up survey with participants. In the past few months, we designed and piloted a post- workshop telephone survey that could tell us if the content of our workshops was meeting actual participant needs and if they could, in fact, use the information and skills they learned in class in their own work. Copies of both the workshop evaluation form and the follow-up telephone survey are available upon request. While we are just beginning to survey participants, the results of the pilot survey give a glimpse of what we might learn from the surveys. The pilot was actually designed to test the form and learn more about the skills needed to make the survey calls. Nevertheless, comparing the information we received from each type of evaluation for two word processing classes in the pilot, we see the following new information emerging: --The telephone survey reaches fewer participants than in-class evaluations. --The majority of participants are actually using the product/information they learned about in training. (There is no way of knowing this in evaluations after class). Those who were not indicated that they would be using their new skills soon when they started a new project. One person called said that they would be using it at home if not on the job. --The workshop covered what people actually needed and they did indeed learn the material. (There is no way of knowing this in evaluations after class. We can only know about anticipated need or perceived effectiveness of training.) --Some of the people surveyed told us about additional training they wanted. Although we might not always add the suggested content to the class they took, this gave us an opportunity to describe additional training available or alerted us to needs not being already met by the curriculum. --Participants told us how they use workshop handouts after class. Although we suspected they were useful as reference or for training others in an office, the survey confirmed that this is happening with great regularity. These results are reported here only as a preview of the kind of information that we will learn from the survey. Because only a few classes were surveyed in the pilot, the actual information is, at best, tentative. It is interesting to note that in both the telephone survey and the workshop evaluations, suggestions for improvement or on what to keep doing are very similar. There is not a lot of difference in responses, although the telephone survey comments do appear to be a little more general and reflective. For example, one person volunteered that they have difficulty attending a class when it is only offered once in a term. This kind of information does not normally come out in a written workshop evaluation form. Similarly, another respondent in the telephone survey revealed that they have difficulty getting enough hands-on experience when they work with an experienced partner in a workshop. All respondents we reached were eager to take part in the survey and pleased to hear from us. It was sometimes necessary to call back several times to reach someone. Other times we may have had to make an appointment for a convenient time to call back. Each survey call took about 5 minutes to complete. Callers felt that it was generally not necessary to be very familiar with the topic of the class. They did feel though that knowledge of our ITD services was important since the calls were also an opportunity to help people find out about other services they needed. Because it is a time-consuming task, not all instructors are enthusiastic about making these calls. All are looking forward to the information they will yield, however. This semester we are having calls made by instructors on a voluntary basis and by our full-time registration clerk. Instructors making calls are working in partnership with a colleague so that each is calling about a workshop of the other. Results of the surveys will be tallied and shared with the individual instructor of each class surveyed. Instructors will, in turn, share the results with their colleagues in curriculum meetings. Results of general interest to all instructors will be shared among the larger group of instructors. Partnerships Working in partnership with users to meet their needs is another way in which we offer our services to the community. Partnerships typically take one of three different forms: --Working with an administrative office to train on a new system they have initiated. --Working with a faculty member to help his or her students get the skills they need. --Working with a departmental administrator or local area networking administrator to train his or her staff. In all cases, working in partnership gives us an opportunity to learn more about the larger environment in which we work and the work and needs of those who use our services -- the "real world" context in which people use the computer skills we teach. These opportunities often have an effect on the regular classes we teach. As we learn more about the environment of our users, we're better able to make our offerings more reflective of their needs. An example of an administrative office partnership is the training we provided with the University's Development Office to train users of their new mainframe computer system. This involved co-designing and teaching workshops that not only covered the basics of the computer system, but also covered how the system could be put to good use by development officers and new ways of working in some cases. The computer aspects of the training were the domain of our ITD staff. The more job- related components were the domain of the Development Office. The quality of this training was enhanced by each of us attending to the parts we knew best. An example of partnering with a faculty member is the many workshops we do for classes at the request of faculty. Sometimes we make our materials available to TA's and faculty and consult with them on their use. Other times, however, it is best for us to do the training. The workshop might be identical to what we regularly offer, or we might work with a faculty member to tailor the content for the specific needs of the students and a specific class activity. The third type of partnership involves designing training for a departmental group which might be getting new computers, new programs, or a new network installation. Or, an administrator might desire to enhance the skills of his or her faculty or staff. In meeting with the key administrators in the department we learn more about the needs of his or her group and can modify our standard training as needed. It is important to know that the administrator who is speaking on behalf of other staff or faculty is, in fact, aware of the actual needs and constraints on time for training. We have learned the proverbial hard way that this can not be taken for granted. Sometimes, it is necessary to ask an administrator to better identify needs and to assist them in the process. It can be a costly waste of resources if the training does not match what is needed or if the administrator has not paid attention to time and schedule constraints. Some administrators have foundthat it is better to work out individual training plans with staff and makes use of our regular workshops than to set up a program for their department as a while. Special training for a group or department worked best when a new system or network is being installed and everyone needs to learn the same thing at the same time. Challenges We face challenges in the coming year that are partly related to changing technologies, partly to the economy, and partly to the widening gap between experienced and novice computer users. At The University of Michigan, one of the issues with which we must contend is reduced resources coupled with the need to move into new areas of activity. A growth in new activities is unfortunately not always accompanied by a growth in resources. This means that we must find ways to train our existing staff to take on new responsibilities and we must find a way to let go of ongoing activities. In education, this has meant cutting back some services to allow more time for staff development. We are substituting non staff-intensive activities such as commercial computer tutorials to free up staff for new activities. It will be important for us to monitor the success of these substitutions and to actively solicit user input and actively listen to what our users have to say about them. As services are considered for reduction, it is important for us to engage in an active dialog with users about what the potential service cuts will mean to each of us and what other options there might be. We must be very aware of the real needs of our community since we literally can not afford to offer services that are not needed; nor can we afford to overlook those that are. Another challenge we face is the widening gap of skills in classes. As more and more of our users become sophisticated computer users, the audience we serve is increasingly more differentiated. "One size fits all classes" will fit increasingly fewer students. It's important to recognize this trend and provide learning opportunities that can be customized for different participants in a single class. It will not always be practical to offer several classes for different audiences. The person learning how to use Microsoft Excel as their first spreadsheet program, for example, has different needs and will learn differently than a person switching from another spreadsheet program. Finally, the last challenge addressed here is the need to explore the best ways to reach the most sophisticated users. Workshops might not be best for this audience, but rather short talks, demos, articles, and tips. We've had good success experimenting with short, to the point presentations and articles in newsletters for this group. More is yet to be done and more information from this growing community will be critical. We need to teach people not only how to work the new technology, but how to work smarter with it.