Organizing Support in a Distributed Computing Environment |-------------------------------------| | Paper presented at CAUSE92 | | December 1-4, 1992, Dallas, Texas | |-------------------------------------| Organizing Support in a Distributed Computing Environment Brad Stone Brigham Young University Provo, Utah ABSTRACT Brigham Young University, like most institutions, has experienced rapid growth in the use of distributed computing systems. However, the maintenance and support of distributed computing has not kept pace with its growth. BYU has researched several methods of support, ranging from departmental autonomy to a central hot-line for total campus support. Each method has met varying degrees of success as well as some major problems. In light of their research, BYU implemented a method of computing support known as the "Computing Support Representative" (CSR) Program. This has provided the coordination and economy of scale of centralized support, while retaining the autonomy and responsiveness of decentralized support. The CSR Program is patterned after a national business franchise that provides the standards, support, and training of a centralized administration, while allowing each unit to operate much like an independent small business. INTRODUCTION In the 1970's, computing at Brigham Young University, like most institutions, consisted of a few large mainframes. In the 1980's, personal computers became more powerful and we started seeing more and more of the University's computing done on the desktop. Computing was becoming less centralized and more distributed. Early in the 1990's, another computing shift occurred. By networking these distributed computers we found that productivity would increase. Resources could be shared, so BYU installed a new fiber-optic based campus-wide ethernet network. As computing changed, so did computing support. The support structure seemed to mirror the technology it was supporting. In the 1970's, support was centralized in one area, just as computing was. When employees had problems with the mainframe, there was only one place to call, mainframe support. In the 1980's as employees started having problems with their personal computers, mainframe support could no longer provide the answers they needed. We started to see computing support arranged at an organizational level, to help employees with needs specific to their organization. These internal support groups did not interact with support groups in other organizations unless absolutely necessary. Realizing that computing support mirrored the technology it was supporting, it was decided that "networked support" was the model for the 1990's. By linking all these distributed support groups, solutions are found much faster, hence "networked support" is the basis of our computing support program. No industry changes faster than computing. As the fundamental principles (paradigms) behind computing change, so must other aspects, such as computing support. Joel Barker defines a paradigm as follows. "A paradigm is a set of rules and regulations (written or unwritten) that does two things: (1) it establishes or defines boundaries; and (2) it tells you how to behave inside the boundaries in order to be successful."[1] He goes further to compare a paradigm to a game such as tennis. "Is the game of tennis a paradigm? If you think about it for a minute, you'll discover that it is. Does the game of tennis have boundaries? Of course, that's the easy part. The tricky part has to do with success and problem solving. What is the problem in tennis? It's the ball coming over the net. And you must solve that problem according to the rules of tennis." In the game of tennis, the rules haven't changed for many years, not so with computing, and the key is to learn the new rules to insure success. The paradigms governing computing are changing. Bigger is not necessarily better. Isolated personal computers are not the most efficient. Despite what their sales people will tell you, IBM personal computers are not necessarily better than the "clones". The paradigms governing computing support are also changing. What worked in the past will not necessarily work in the future. The rules of the game are changing. We offer the following solution to the changing paradigm of computing support and label it the Computing Support Representative program. COMPONENTS OF THE SUPPORT PROGRAM The five principles There are five principles of computing support. With any one of these missing, your support program will not be effective. With all in place, the program will work efficiently and smoothly. These principles and the premises they are built upon follow. --1. Channel for resolving problems Premise: The employees have more important things to do than fix problems with their computer. The most obvious aspect of computing support is problem resolution as it is the most immediate. Not all your employees are involved in a computer related field, although a great number of them use computers daily. By keeping their computers running smoothly, and teaching them how to use their computers efficiently, your employees become more valuable. They not only increase productivity, but also enjoy their work more. Whenever an employee has a computing problem, they should be able to report it and feel confident that their computing support personnel will take responsibility for solving the problem quickly, without losing track of it. The employee also expects to be informed continually on the status of his or her problem. We have found that employees are not so much concerned how quickly the problem is resolved as being kept informed of its progress. Most employees understand that some problems are very difficult, and will require some time to solve. They need to be kept up to date and informed when their problem will be solved. --2. Channel for information to and from employees Premise: The employees needs to know about their computing environment, and have concerns that need to be addressed by administration. Resolving problems is very important. However, by establishing an information channel between your administration and employees, you can keep problems from happening. Your employees need information about their computing environment. They need to know about computing standards, changes to the network, when computing events are happening, what computing services are available to them, and more. A lack of communication leads to the employees distrust of their computing environment. Many fear what they don't understand. This distrust may appear as a reluctance to use the network, frustration in using "non- standard" or "non-supported" products, and even aggravation with those setting computing guidelines. Your employees have ideas and concerns that the administration needs to address. The employee reports these issues to the CSR and the CSR polls other CSRs to find if there is an institution-wide need. The administration is then involved to resolve concerns and act on new ideas. With proper communication in place, your employees won't be buying hardware or software that is incompatible with others in the organization. They can plan for scheduled network outages and changes, and they feel like their concerns are being addressed. --3. One phone number for employees to call Premise: Employees will either experience "the run around" or won't know who to call for help. Telephone support lists (stating who to call for which types of problems) do not work. We have tried them many times, and they do not promote quality support. Even if the employee knows who to call for what type of problem, they may not know what their problem is. They only know that they want to do something with their computer and it doesn't work right. The problem may be in the hardware, software, or even somewhere else on the network. Ralph Wilson, manager of training at Computer Hand Holding, Inc., sums it up this way. "In many organizations there are several help desks, one still devoted to mainframe access, one for PCs, and perhaps one for network issues and telecommunications. Throw in other phone numbers for hardware installation, maintenance, and telecommunication issues, and users are often confused when they have to decide who to call. Even support staff themselves may have trouble figuring out who's supposed to handle what."[2] Every employee should have only one telephone number to call to report all computing problems, whither it deals with hardware, software, or operating systems on any platform used within the organization. The support person on the other end of the telephone should take responsibility for the problem, do some initial diagnoses, and call whoever is necessary to help get the problem resolved if he can't resolve it on his own. A single support telephone number eliminates the need to know who to call for help, is easier to remember that an entire telephone list, and eliminates run-around. --4. Problem coverage during normal working hours (8-5, M-F) Premise: In emergency situations, employees need computing help immediately, and expect it during normal working hours. Perhaps there is nothing more frustrating to an employee than to have a computing emergency, and have no one around to help them. Most of us have had the frustrating experience of needing a report printed in fifteen minutes, but for some reason the printer is not working. While other aspects of computing support are not as immediate, problem resolution needs to be available during normal working hours, even during lunch time. --5. Support person has allegiance to the organization Premise: A support person within an organization knows more about the computing needs of their employees than any support personnel at an institution level. For years we have been trying to implement some sort of campus level computing hot-line with varying degrees of success. No matter how many resources we poured into this campus level support, we could never hope to support anything more than the mainstream applications and common configurations. Those organizations that used applications or hardware specific to their discipline were always left on their own. The employee was required to support those items by themselves. With a support person within the organization, we found that it was much easier to offer support. The support is tailor-made for the organization. The support person knows the software programs their employees are using, and knows what hardware platforms they are using them on. Even more important, the support person knows the professional background of the employees and can offer them solutions to computing projects that an institution support person could never offer. Employees view this support person as an insider and are less likely to become intimidated. TYING IT ALL TOGETHER Instead of the employees calling whomever they can, they contacts their organization support person. This support person already knows about the communications software and network connections that the employees are using. He also knows how the network is configured within the organization, and can most likely diagnose the problem area (mainframe, network, or workstation) in a matter of minutes. Once the problem is isolated, this support person can contact the appropriate service group and speak intelligently with them, resolving the problem much quicker, without a lot of anxiety on the part of the employee. This support person is what we call a Computing Support Representative or CSR. The Computing Support Representative (1st Level) The CSR is responsible for resolving problems and distributing information during normal working hours. From the employee's point of view, problems can be resolved by dialing a single support phone number. They know that their CSR is someone that knows them and their computing practices (see Figure 1). With the CSR program is place, there should never be a computing problem that can't be resolved to the employee's satisfaction. The Computing Support Network We have now identified a means for quality support by appointing a CSR to organize problem resolution and distributing information. This works as long as the CSR does not come across any problems that are out of his sphere of expertise. The key is to choose a CSR that is as familiar with the organization's needs as possible. We then train the CSR in the areas he is weak, and back the CSR with the experience of others. In the same way as a number of personal computers on a network become more powerful than single computers by themselves, a network of CSRs can become more powerful than a group of single CSRs. This linking of collective knowledge of all the CSRs becomes the Computing Support Network. Let me emphasize that this is not a computer network made up of wires, but a people network where information and ideas can be shared. All of our CSRs have talents in their own specific areas, both from background and because of the special needs of their organization. One CSR may be very adept at solving Novell Netware problems, yet be weak in Unix. Another may understand Unix, yet not understand the intricacies of setting up MS Windows. When one CSR has a problem or question that he cannot resolve, he can feed it into the Computing Support Network and other CSRs can help him (seeÊFigureÊ2). Exactly how this happens is covered later in the section on CSR communication. The Computing Support Team (2nd Level) The Computing Support Network works very well for resolving problems a CSR may not be familiar with, however CSR to CSR communication is not enough. There are times when a CSR will be faced with a nasty sort of problem that may spread across many different areas of expertise. For example, the problem may involve a conflict between a software network driver, the network card, and a file server. To resolve these really tough problems, they need to be escalated above the CSR to a second level of support. This second level of support consists of institution level experts that work together to resolve problems. These are some of the most experienced people in diagnosing hardware problems, application problems, database design problems, network problems, mainframe problems, operating system problems, etcetera (see Figure 3). If a CSR escalates a problem to the second level, then there is a good chance that it will require multiple experts working together to find a solution. The CSR is still responsible for resolving the problem, however the second level will work with the CSR to diagnose the problem and offer possible solutions. With this second level in place, the CSR is never put in the awkward position of being responsible for resolving a problem, but not given a way to do so. The Vendor Connection (3rd Level) Computing vendors play an important role in the Computing Support Network. It is every vendors dream to be able to contact all of the employees in an institution that are interested in their products without bothering those who aren't. By using the Computing Support Network, a vendor can keep the CSRs up to date on bug fixes and new releases of their products. In turn the CSRs can pass this information on to those within their organization that they know are interested in the vendor's products. For example, Microsoft may have just released version 2.0b of it's Word for Windows product. The upgrade is free and fixes a few bugs in the previous release. The Microsoft representative can give this information to the CSRs who in turn can pass it along to all their MS Windows users while not bothering their Macintosh users. Using the Computing Support Network, vendors can also announce presentations and seminars that employees may be interested in (see Figure 3). The Computing Support Network works both ways. Not only can vendors keep the CSRs informed, but the CSRs can inform the vendor of bugs, complaints, and possible enhancements the employees in their organization need. The CSR sometimes speak for hundreds of employees, and when the vendors realize this, they will be motivated to act on the CSRs recommendations. Sometimes the vendor will need to be involved in the problem resolution process, and by linking them into the Computing Support Network, the CSRs know who to contact and generally receive better attention than an unknown user. This leads to a very good relationship between an institution and the vendors that supply them products. At the present time, we have good communication with Cicso, Microsoft, Apple Computer, Novell, and others. We also treat our campus services (computer repair and network installation) as vendors. The Computing Support Coordinator We have now described a support structure that is complete except for one part that makes everything work. This part is the Computing Support Coordinator who is a person who ensures that effective communication and problem management occurs between all three support levels. The Computing Support Coordinator is also an evangelist for the support program who can transfer enthusiasm, along with concepts and principles of the support program, to administrators and CSRs. The duties of the Computing Support Coordinator are as follows: 1. Act as problem manager. Track problems that get escalated from the first level to the second level. This ensures that no problems get lost and that solutions are properly cataloged for future reference by CSRs. 2. Coordinate Communication among the CSRs and technical support staff. The Computing Support Coordinator acts as a "directory" in the network of CSRs. When the CSRs do not know who to call for a given problem they can contact the Computing Support Coordinator who will direct them to a person who can help them. 3. Train CSRs. The Computing Support Coordinator accomplishes training through specialized training classes which focus on specific topics that the CSRs need to know. These classes can be taught by second level support personnel, or by other CSRs. Classes not only help train the CSRs but also bring them together so they can learn more about the skills and interests of one another. This helps foster a networked support environment. 4. Publish a periodic newsletter. Newsletters help keep the CSRs informed of new methods of support and the appropriate use of computing technology for their employees. The newsletter also provides a forum so that CSRs with particular specialties can contribute to the Computing Support Network by publishing articles about their own expertise. 5. Audit problem resolution. The Computing Support Coordinator contacts CSRs, as well as employees, to get feedback on how computing support is being perceived within the institution. This follow-up acts as a quality control indicator to keep the computing support structure service-oriented. 6. Focus support personnel on solutions to problems. Individuals with in-depth knowledge of particular computing technology sometimes become so focused on the activity of investigating solutions that they forget to solve the problem. The Computing Support Coordinator helps to channel their energy in productive ways. HOW TO SET UP THE SUPPORT PROGRAM Initial Setup This next section describes the preliminary steps to organizing this computing support structure at your institution. It should give you a good idea of the resources and time involved. --Who is responsible for resolving computing problems? One of the first questions you must ask yourself is "Who is responsible for resolving computing problems?" If you are wise, you won't ask this question too loudly until you have decided the answer for yourself. At Brigham Young University, we found that it would be best if each college were responsible for providing support for their employees, and structured our support program around that premise. While this is probably the best approach for most large institutions, it may not be right for all. For example, our Hawaii campus is about 1/15th the size of our main campus, serving approximately 2000 students with 200 employees in 18 major divisions. We found it impractical to try and appoint support personnel in every division, and decided to assign Computer Support Services (campus level support) the responsibility of resolving computing problems. A CSR was then appointed in each division to act as an information channel and consultant for computing issues. --Hire a Computing Support Coordinator Look around your institution and see who is responsible for insuring the employees are receiving quality support. If you don't see anyone, or see multiple people in various divisions, then chances are you won't see quality support. It is essential that a Computing Support Coordinator be hired to get your support program off the ground. --Assess Size of job and Resources available You have a big task ahead of you, and you need to formulate a game plan. You will want to make sure you know what resources you have currently available and where more resources are likely to be needed. Start by sitting down with an organizational chart of your institution. Highlight the major organizations, organizations that work closely together and those that don't due to geographic or political separation. Record the number of employees in each organization, and any support resources that may exist on campus, such as help desks, computer labs, known computing specialists, hardware support contracts, campus level services, etc. It may also be helpful to take a walking tour of various organizations to find out how they use their computers, and how they presently resolve problems. How many support people should be employed? PC Week surveyed over 500 corporations making up their Corporate Advisory Panel and found that each support employee was supporting 77.5 computers in 1988 (up from 73.4 the year before)3. With the CSR program properly in place, we have seen a single full time support person supporting over 100 employees, and the quality of support is greater than when our support people were closer to the 77.5 ratio and didn't have networked support. Here is a rare case when you can have the best of both worlds; better support for less expense. --Obtain Support from Administration Organizing a new support program that encompasses your entire institution will involve change, which is often resisted and sometimes painful. At some point you will be asked by what authority you can dictate this change. You will need support from the highest levels of administration if your new support program is to succeed. If your administration is in touch with the employees, they will know that there is a need for the five principles of the computing support program, and will welcome a way to provide efficient support for the institution. --Meet with organizational leaders Once you have support from the administration, it is time to work with organizational leaders to appoint someone as the Computing Support Representative within the organization. You will need to arrange a meeting with them, or at the least send a memo to them explaining the role of a CSR and the need of having this person affiliated with their organization. Any organization that uses computers has some way of supporting them, even if it is undocumented and confusing. It may be some internal support organization, file server administrator, or enthusiastic secretary. The organizational leader will help you by specifying the person she feels is most qualified to coordinate support. This will give you a place to start. --Opening CSR Meeting Now that you have a number of people appointed to be CSRs, you will want to set up a meeting with them to explain the CSR program in detail, and notify them when future CSR meetings will be held. Establish CSR Communication The need for effective communication can not be stressed enough. There are three types of communication necessary for CSRs; time critical, general information, and reference. --Email List (Listserv) and Newsgroup (Usenet) For time critical information, nothing beats electronic mail (e-mail). No matter how prevalent e-mail is in your institution, the CSR must have an e-mail account to link him to the Computing Support Coordinator. Hopefully, every employee will also be using e-mail, making it easier for the CSR to distribute information to the employees within the organization quickly and easily. At BYU, we use two different ways to distribute this time critical information. First, we use a Listserv4 e-mail list. All of the CSRs (and even some interested employees) subscribe to this e-mail list. Information such as virus alerts, scheduled network outages, vendor announcements, and surplus equipment lists are sent to the CSRs, who in turn pass along this information to interested employees. CSRs use this e-mail list to post questions to other CSRs. This e-mail list is also mirrored on a Usenet Newsgroup for those CSRs that prefer to read and post information in that manner. --Monthly Meetings Monthly CSR meetings are held to disseminate important information that is not time critical. These meetings are held once a month at the same time and place, and are never longer than one hour. We have found that most of our CSRs are too busy to devote a large amount of time to long or frequent meetings, and that all the information they need can be distributed in an hour or less. In monthly meetings we cover such things as campus computing standards that have changed, proposed new standards, problems that keep cropping up in different organizations along with their solution, and anything that would be too complicated to convey in writing. These meetings also foster CSR to CSR contact that enhances networked support. --Training Meetings The key to quick problem resolution is to have well trained CSRs. Most of the CSRs will be clever in certain aspects of computing, but may be weak in others. It is your goal to get all your CSRs trained in the basics of computing support. Our training classes are taught by both second level support personnel and CSRs in areas such as the Internet, operating system software, diagnosing hardware problems, database design issues, Novell Netware, and others. --Newsletter A monthly newsletter is published with any information that the CSRs may need to refer to in the future. There are articles on common problems and their step-by-step solutions, articles that explain topics such as networking, and articles on campus standards. These articles are written by the Computing Support Coordinator, second level support personnel, and even CSRs. CSRs are encouraged to distribute a newsletter within their own organization with information that is specific to their computing. To facilitate this, we make all of our campus newsletter articles available to our CSRs in electronic format so they can include any of the campus articles in their own newsletters. --CSR Phone List Your goal is to define a CSR for every employee within your institution. The easiest place to start is to assign CSRs along political or geographic boundaries within your institution. You also need an easy way for employees to determine who their CSR is. We started by making a list of all the organizations within our institution with information about the director and the CSR (see Figure 4). Send information to end users With all of the above in place, you are now ready to advertise the CSR program to your entire institution. Don't let it concern you if it takes you a while to get to this point. There are a lot of changes involved in organizing such a program, and changes take time, but it is worth the effort and the wait. It took us almost two years to implement this program at BYU. Hopefully this document will help you to avoid the pitfalls we fell into, and you will be able to set up the program much more quickly. --Advertise the support program If you have an institution-wide newsletter, this may be a good place to start. The main points to cover are that all employees have one phone number to call for any computing problem, and that they will be receiving information within their organization on what that number is and how to use it. --Personalized information You will need to send out information to all the employees individually. This information should be personalized to explain how they use the CSR program. Although all of our organizations follow the five CSR principles, none of them are set up exactly the same. We printed up a brochure that details the duties and responsibilities of a CSR, and left a space where personalized information could be printed, such as the phone number that employees should use to report computing problems. This brochure was then sent to every employee with a memo from the organization's CSR describing who he was, and how he would help them get answers to computing problems. Also included is a sticker identifying the CSR and their support phone number which can be adhered to the front of the employees computer, the employees phone, or a Rolodex card. --Explanation letter to organization leader In addition to the personalized information being sent to every employee, the organizational leader receives a special memo describing how the CSR program is operating within the organization, and how the Computing Support Network will back up the CSR when he has really tough problems. This memo can be sent from the Computing Support Coordinator. Build CSR Resources Once you have the CSR program in place, you need to concentrate on providing resources for your CSRs. You're goal is to make it as easy as possible for them to solve problems. You will want to implement a problem tracking system so your CSRs and second level support personnel can work together on a problem without losing track of it. Give each CSR a three ring binder to store handouts and newsletters for future reference. Create quick reference guides on everything from electronic mail to network installations for both CSRs and employees. Set up an on- line information source such as Gopher5 to archive support information that is easily retrievable, and create a CSR library that contains manuals, reference books, and software that CSRs find useful. NOTES [1] Joel A. Barker, Future Edge: discovering the new paradigms of success (New York : William Morrow, 1992) p.32. [2] Ralph Wilson, Help! The art of computer technical support (Berkeley: Peachpit, 1991) p.40. [3] Ralph Wilson, Help! The art of computer technical support (Berkeley: Peachpit, 1991) p.49. [4] Eric Thomas, Revised LISTSERV; BITNET-Oriented Presentation email "Info Pr" to LISTSERV@RICEVM1.RICE.EDU. February 1987 [5] Gopher clients, servers, and documentation are available via anonymous FTP to boombox.micro.umn.edu.