Implementing A New System On Time In Bad Times Copyright CAUSE 1994. This paper was presented at the 1993 CAUSE Annual Conference held in San Diego, California, December 7-10, and is part of the conference proceedings published by CAUSE. Permission to copy or disseminate all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, that the CAUSE copyright notice and the title and authors of the publication and its date appear, and that notice is given that copying is by permission of CAUSE, the association for managing and using information technology in higher education. To copy or disseminate otherwise, or to republish in any form, requires written permission from CAUSE. For further information: CAUSE, 4840 Pearl East Circle, Suite 302E, Boulder, CO 80301; 303449-4430; e-mail info@cause.colorado.edu IMPLEMENTING A NEW SYSTEM ON TIME IN BAD TIMES Elaine David The University of Connecticut Storrs, CT 06269 Abstract In 1991, the Student Information area of the University of Connecticut Computer Center faced many problems. There was very little documentation, many jobs were not in production and being run as 'test' jobs from programmers' machines, and the staff had no overall knowledge of the projects under development. In addition, bad economic times had resulted in the loss of many knowledgeable personnel. In the midst of these difficulties, the student information group was assigned the task to implement a university-wide touch- tone registration system. In order to cope, we restructured our group to insure rapid development and first-time perfect operation of the new system. This paper will discuss our new standards and procedures, the problems we have encountered, and the progress we have made toward achieving the goal of installing a touch-tone registration system which would work perfectly the first time. IMPLEMENTING A NEW SYSTEM ON TIME IN BAD TIMES INTRODUCTION In the Fall of 1991, in an effort to cut down on payroll expenses to the State of Connecticut, University of Connecticut employees were encouraged to take early retirements and voluntary layoffs rather than face mass firings. The loss of staff within the University community resulted in greater demands on the computer center for additional computerization of University office functions to increase University efficiency; the loss of highly knowledgeable staff within the computer center made meeting these demands more difficult. Since there was no possibility of replacing lost positions for the foreseeable future, Administrative Services decided to consider the possibility of restructuring its staff in the hope of becoming more efficient. In December, 1991 Administrative Services was restructured to consist of 3 teams of programmers each headed by a Team Leader. Each team would be responsible for multiple projects and team members would move from project to project within the team, depending on need. Team 1 was assigned student and academic applications, including the student information system, the auditing/advising system, and the new (yet to be programmed) touch-tone registration system. The team consisted of 2 senior programmer analysts (one of which was made team leader), 3 programmer analysts, and 1 programmer (transferred from production support). Although the main impetus for using the team approach was the need to restructure due to the loss of personnel, Team 1 viewed this change as an opportunity to improve the overall student information system. Over the years, many of the team members had voiced concern about some of the ways we operated. With the formation of this team we decided to review the concerns we had, rank them and develop a plan for improving the way we worked. CURRENT SYSTEM PROBLEMS In meeting with the team, three main areas of concern were identified: personnel concerns, current system concerns and new system concerns. The loss of 3 key members involved with student information systems in November, 1991, meant a loss of 60 years of combined experience. The manager who left had written many of the original student record systems programs which were still part of the newer system. His loss meant that any problems with or changes to these programs would create a problem for the computer center staff. The project leader who left was a trusted member of the University community. She served as a primary interface with the various departments, and was very knowledgeable in their needs. The primary analyst who left was the person involved with maintenance of the files, and who oversaw grade processing. Also, he was the person who had investigated the purchase of a voice response system for the new registration application. By November, 1991, the morale of Team 1 was at an all time low. Not only did they have to deal with the added stress of increased work loads, frozen salaries, and lack of certainty about the future, but they also had to deal with the fear of failure due to lack of knowledge (regarding both specific tasks, and a general overview of the entire system). Because of the prior stability of staff and the number of staff members involved in the student/academic systems, the computer center had allowed itself the luxury of permitting specialization. The staff member who was initially involved in a particular programming task was later the person to be involved with any modifications or problems dealing with that program(s). In short, we had permitted 'ownership' of information. This practice was beneficial in enabling us to do tasks quickly, but the lack of crosstraining backfired when we lost programmers involved in some of the major areas. Given that there was no hope for new staff, and that no current staff member was familiar with the overall student record system, it was time for us to require a broader knowledge base of the staff, and to begin a program of crosstraining. In our team meeting discussions several major problems emerged. The first problem that we noted was that not all jobs were in production. (Only production jobs are scheduled by the user through the scheduling office.) Some jobs were still in the 'test' library and were being scheduled by a programmer at the request of a user. Other jobs were being run by programmers from a programmer's machine at a user's request. Also, many "errors" were being corrected "on the fly" without being logged in via a service request. This practice permitted undocumented modifications on user demand without factoring in other requests for programmers' time. The second problem we encountered was the lack of documentation (or minimal documentation) of the system (jobs/programs/interfaces). This meant that anyone other than the programmer who was initially involved with the job/program/interface would have difficulty determining the nature of any problem and method for proceeding when a problem developed. The third concern with the current system involved grade processing. This had always been a major effort by the computer center. It had been handled by two of the members of the staff who had recently left, and required all night overseeing by them. It was a process which rarely (if ever) ran smoothly, although the specifics of what went wrong were not known, as the procedures involved had not been documented. Grade processing was next scheduled for December 30, 1991 (1 month away from the time of reorganization), and would require the team's immediate attention. Despite the financial problems at the University, the administration continued to maintain its strong commitment to the need for a touch-tone registration system. The then current system of processing preregistration requests using a batch system and handling over 7000 students at add/drop using punched cards was no longer considered acceptable. Although some online capability already existed for the regional campuses and the continuing education office, this capability was not available at the Storrs campus. In addition, a 'promise' had been made by staff members who had since left that it was feasible to have a new registration system up and running by August, 1993. There was not a single computer center staff member remaining who had been involved with the touch-tone project. Although a general plan existed, no detailed analysis of any of the 'subfunctions' of the system was available. To have any chance of meeting this new challenge, it was necessary to move immediately to ascertain what needed to be done, what could be done, and the resources required to get it done. IMPLEMENTING THE TEAM CONCEPT (ASSESSMENT) In December, 1991, a detailed analysis and plan were prepared for submission to the Touch-Tone Steering committee (consisting of the Associate Provost, members of the Registrar's office, associate deans from several colleges, and computer center staff). The analysis showed all of the tasks required to meet the goal of the original project plan, and a time estimate (in hours) for each task. By reviewing the amount of time team members had spent on previous projects, it was estimated that the team could (at best) devote 70% of its productive time to this new project, and still manage its other functions. Using this information, it was clear that we could not provide all of the functions requested by August, 1993. However, a plan was proposed which called for a phasedin approach to introducing touch-tone registration. The plan called for an online method for handling add/drop for August, 1993, which would eliminate the need for punched cards and reduce the lines of students waiting to change courses. Also, the plan called for a touch-tone tone registration system to be available for add/drop with limited functionality for January, 1994, and a fully functional system available for January, 1995 which could handle not only the add and drop period but also the preregistration period. It was imperative that the touch-tone registration system be operational within this new time frame; because of the high visibility of the project, it would have to work perfectly the first time. To address the concerns of the team involving personnel issues, current system issues, and new system issues, and insure the success of the new registration system, the team decided that it would be beneficial to hold frequent working sessions to determine how we would proceed and to keep everyone informed of what was happening. IMPLEMENTING THE TEAM CONCEPT (GRADE PROCESSING) We began by reviewing the current schedule submitted by the system administrator from the Registrar's office and the schedule from the computer center scheduling office which showed dependencies and run times for each job. We decided that since we were sufficiently unfamiliar with the process, we would carefully monitor grade processing in December to insure that any problems would be detected as early as possible (hopefully prior to the printing of grade mailers and transcripts). We determined potential places for failure within the process and decided to back up our files before the running of these jobs as a safety measure. We also discussed how we could know that a particular job was producing the correct results when the job ran successfully. For many of our jobs, summary reports were produced. However no one was looking at the reports until the following day, when the entire grade processing had been completed. These jobs would now be flagged to indicate that the process was not to continue until the reports were read and approved. Jobs which were not producing 'readable' reports were modified to provide better information. In reviewing the current grade processing schedule we noticed that processing jobs and printing jobs were interspersed, so that jobs which required checking by the user might occur at 2:00 a.m. The schedule was revised to do the processing first and the printing of transcripts, mailers and letters later in the evening, with the expectation that if all the processing was correct then the outputs would also be correct. Before running the grades, the team did a walkthrough of the process and discussed how we would recover if a problem occurred at any stage of grade processing. These recovery procedures and the additional jobs needed for recovery were then documented. Although we felt we had done a good job in improving grade processing the team decided to be available during our first trial. With pizza donated by our director to fortify us, we watched as job after job ran successfully. We checked all the output summaries, verifying the information reported and all outputs, giving special attention to grade mailers, probation and dismissal letters and transcripts. The December running of grades was the fastest and best grade processing the University ever experienced. Our future goal was to have grade processing run smoothly without the need for the computer center programming staff overseeing the process. This goal was accomplished the next time we ran the grade schedule, in May, 1992. Once grade processing was no longer a major concern to the team, we decided to tackle the problem of the current Student Records system. We realized that we could not undertake a major new project if we were going to be constantly pulled away to handle 'problems'. Therefore we needed to put together a plan for minimizing 'problems' so that we could focus on new tasks. It was important to insure that we would in fact devote 70% of our productive time to the new registration system, if we were to meet the deadline that was set. IMPLEMENTING THE TEAM CONCEPT (OTHER PROBLEMS) The first item in our plan was to continue our group meetings to discuss problems, issues and overall design objectives. We decided to schedule regular weekly two hour meetings to discuss general issues and to schedule other meetings as needed. The team set the agenda for each meeting, including any questions or concerns they had, and the agenda was distributed prior to the meeting. In addition, a running task list was maintained by the team leader and at the beginning of each meeting outstanding tasks were reviewed to determine their status. New tasks were added to this list as they were assigned to the team members. We decided that our next priority would be to review every job that was part of the current system, and put all nonproduction jobs to production status. This review included jobs which were on programmers' machines, test jobs in the test library and preproduction jobs in the test library. During this review we found that in some cases we had several versions of a job. In the past this had created problems when we went to change the wrong version of a job. As a group we determined which was the correct version and deleted all other versions from either a programmer's machine or from one of the libraries. By May, 1993, we had cleaned up the test library and put all our jobs to production status. As part of the process of putting jobs to production status the team reinstated a policy of creating programmer and user documentation to accompany all production jobs. Also, a policy was established that all team members were required to spend 10% of their time creating documentation for 'old' jobs. We decided that this documentation would reside on a special machine to which we all had access, and that all job and program documentation would follow a specified format that we created. One of the team members was assigned the job of insuring that new documentation adhered to the standards and creating an index to the documentation. By July, 1993, we had created 400 pages of new documentation. Several programmers had noted that they had created 'special' jobs/procedures for handling problems that they had to deal with. These jobs were located on their own machine. To improve the technical competence level of the team we decided that the procedures would be documented and the jobs would be put in the 'test' library. We came up with a naming convention for identifying these jobs and distinguishing them from test jobs which would eventually go to production. Ultimately, this permitted flexibility in assigning 'problem' tasks to programmers. If the documentation was well written and the job/program was available then anyone could solve the problem without the need to 'reinvent the wheel'. Each time documentation needed to be used team members were provided an opportunity to reassess the usefulness and accuracy of the documentation. A programmer who did not feel the documentation was sufficient for his/her needs went back to the programmer who initially wrote the documentation and asked for improvement. In several instances programmers would ask another programmer to review their documentation before it was finalized, rather than have to redo it later. Although the team members initially balked at the tedium of having to create documentation, they have been relieved at knowing that they are now no longer the only people who can handle a given problem. Another technique used to increase the versatility of the team members was to have one programmer work with another, more knowledgeable programmer on a particular problem. Team members were more willing to take criticism from their fellow team members than from the team leader who would be responsible for evaluating them. This process also promoted the team concept. IMPLEMENTING THE NEW SYSTEM In March, 1992, the team began tackling the new tasks for the registration system. Each month we monitored our progress, by using an inhouse reporting system called Time Track. Much to our dismay, we discovered that we were not spending 70% of our time on the registration system as we had planned. In analyzing the data we learned two things. First, programmers were spending considerable time responding to ad hoc (telephone) requests from users. We also learned that the written service requests from the Registrar's office that were being submitted were not being ranked in priority order. Not only was the time we spent on other tasks affecting our ability to work on the new system, but the constant telephone interruptions from users were causing the programmers to lose concentration when they were working on the new system. Although we had improved our efficiency through proper documentation, crosstraining and improved procedures, we now needed to improve the work habits of both the programming staff and users to meet our ultimate goal. In meeting with the team, several problem areas were identified. 1. Users were interrupting the programming staff with telephone calls which were in fact service requests. 2. Programmers found it difficult to say 'no' to ad hoc requests which only took a 'couple of hours' of their time. 3. Because there was no paper trail for many of these ad hoc requests, specifications were not finalized prior to programming and therefore what a programmer initially thought would be a two hour task could actually take several days. 4. Usually service requests were not being properly prioritized by the users, so that a 'nicetohave' enhancement would be sent in with a vital 'musthave', without being distinguished. Team members felt that while the current system was negatively impacting their productivity, they did not feel comfortable with denying users ad hoc requests, even though it was the policy of the computer center to require written requests. The programming staff had worked closely with many of these users, and a good rapport had been established. They felt that it was important to preserve these relationships, and they felt that by denying any request they would jeopardize the good relationships. A suggestion of having all telephone calls go through a central number so that they could be screened was overwhelming rejected by the team. Since many of the team members had young children, they felt it was important to have direct outside phone lines. It was decided that we first needed to educate the users about our policies and how they would be implemented, before we could expect the programming staff to adhere to the policies. The following policies were restated and agreed to by the administration and Registrar's office staff. 1. Telephone calls to programmers would be limited to those required for the implementation of an already assigned service request. The users were informed that programmers would no longer be permitted to spend their time servicing undocumented requests. 2. All requests for service would be handled via a written service request. Any emergencies, which required immediate attention and could not wait for submission of a service request would go though the team leader. (Even emergencies were required to ultimately have a service request submitted and a number assigned to the task for auditing purposes). 3. Until we felt comfortable that we could meet the time line developed, we would only handle vital (emergencies and mandates) requests. Any other service request demanding our attention would require the signature of the Associate Provost before it would be done. This practice would insure evenhandedness for the users. At first the users were unhappy with the rigor that was being imposed, but as time went on, and the quality of our service improved, more and more users directed their call to the team leader and submitted proper written service requests. ACHIEVING OUR GOALS A PROGRESS REPORT As we continued to develop the new registration system we implemented several new techniques which in retrospect were crucial to our initial success. First, we kept a paper trail of all communication concerning the new system. This was a carryover from requiring a written service request. For the touch-tone project we decided to request sign-off of written design specifications for each subtask, be it a new directory of classes, extended security, development of a scheme for creating access time blocks for students to call, or developing an administrative online add/drop program. Programming did not begin until all of the specifications were determined for that subtask and we had a written sign-off. We wanted our programs to be of first quality and the only way we knew to achieve that goal was to avoid modifications to the original programs once implementation had started. What we offered to the users was our analytic skills in developing good specifications, clear documentation on the specifications, and a walkthrough to insure that the programs (once developed) would serve their needs. In return, we demanded from the users full attention to the analysis and walk-through, as well as written acceptance of the specifications. We never rushed the users to accept specifications before they were ready to do so, but the users knew that programming would not begin until they gave the 'go ahead'. In addition, the users were responsible for final testing and acceptance of the program as meeting the specification agreed upon. We agreed that unless the program did not meet the specifications, or there were policy changes which affected the program, we would not alter a program. Of course we would always make modifications due to programming errors, but we believed that these situations would be minimal once we received written acceptance. Because our goal was to have a product which would be perfect the first time it was used in production, the team decided that it was important to provide for a large safety window for testing. This meant that at times we needed to 'perfect' the basic system before adding enhancements. Our programs were written with sufficient flexibility to allow other functions to be added in the future. Not only did we require behavior modification from our users but the team underwent its own kind of behavior modification. The old attitude that it was all right to make mistakes as long as you were available to correct the problem when it was discovered was no longer acceptable. Not only could we not afford the time to correct programs, but too many mistakes had hurt the respect and trust from the user community. We began to improve our own testing techniques. For batch programs this meant running the production JCL with the minimum number of changes possible, and including the output from the test with the production run book (which contained the JCL, message library and programmer and user documentation and was available to the scheduling and operations areas). The team leader would not sign off on any production run book without the inclusion of the output from a test. For online programs, programmers, after doing their own testing, might enlist help from fellow team members. The same documentation that would be turned over to the user was given to a team member and the team member was requested to try out the software. Feedback from the testing was discussed at our weekly meetings, and any necessary changes were made prior to turning over the software to the user for testing. One of our main concerns in using an online add/drop program, in which over 60 terminals would be available for add/drop and a larger number of terminals would be available to access the Master Schedule to provide student with information on the availability of classes, was that our system would not be able to handle the additional number of I/Os expected without severe degradation to the entire CICS system. To deal with this potential problem the team included Systems personnel in the early stages of planning performance testing. Not only did they take part in file design analysis and assist us in developing a test plan, but they also carefully monitored the CICS system during testing and after the programs had been put to production. The reports that they produced for us provided us with the information needed to define our VSAM files in the most efficient way possible and decide where to put files to minimize contention. One of the contributing factors to the success of both the online administrative add/drop application and later the touch- tone registration application could be that in designing the requested applications the team also took into account user procedures. Often it was the team that first recognized that current procedures were no longer compatible with a new computer system. This was exemplified when we went from a punched card system for add/drop to a computerized system and the team recognized that a mechanism for advising the student of cancelled courses and courses which no longer had seats left would need to be developed, as the presence or absence of cards in a box would no longer provide this information. As a result the team developed a public access program to the Master Schedule which students could use. Since the Registrar's office and other administrative personnel already had online access to the Master Schedule, the importance of this addition to the project was not fully appreciated by the administration until add/drop was underway. By March, 1993, we had completed the programming for the administrative online add/drop system. The documentation for the system was complete; key administrative personnel had been trained; testing had been done by both the team and the Registrar's office; and we had received a sign-off by the Registrar's office indicating that the system we had met all specifications to their satisfaction. We were now ready to devote more of our attention to the online application which would support the new telephone registration system. The new touch-tone registration system (TTR) project provided the team with a wonderful opportunity to work in a new and better way. First, we decided that since TTR would be a separate system, we would abandon the standards developed by Sigma Corporation and used in all of our online student record system programs so far. These programs contained many lines of code which was not needed for our installation, and which many of our programmers did not understand. We decided that the programs would be written in COBOL 2 and make extensive use of the temporary storage queue capability. By making this decision we committed ourselves to being pioneers, since we were the only programmers within administrative services to use COBOL 2. Since we would be treading into some new territory, we decided to make all technical decisions as a team. The team leader, acting as lead analyst for the project, worked with the Touch-Tone Steering committee to develop clear specifications. Most of these specifications (in broad terms) had already been determined when we put together the project plan. However, details were now needed. Once we had sufficient detail from the committee, the team reviewed the information to determine if additional information was needed which might affect the way we designed the system. These questions were documented and sent to the Registrar's office personnel for response. With the information we had, we began to develop an overall design of the system. This design, expressed both in words and as a flow diagram, was then submitted to the committee for review. The committee met, discussed the overall plan, and gave their permission for us to proceed. The team then proceeded to expand the overall plan into more detailed specifications. Since the voice response application would be written by an outside vendor, it was important to have clear documentation of the system, not only for ourselves but also for the vendor. The textual documentation was expanded first, and reviewed by several members of the team. This plan was then translated into a flow diagram by another team member. Meanwhile, other team members used the plan to develop screens and to begin programming the mainframe application. It was always necessary to keep consistency between the textual plan, the flow diagram and the programs. As the textual documentation became more and more detailed, so did the flow diagram and the programs. During this detailed design phase the Touch-Tone Steering committee was unavailable for meetings because of scheduling problems. Before the beginning of the programming phase, the team made several design decisions, to insure consistency from one program to another. Several different members of the team were assigned different parts of the programming effort. While this meant that we needed to meet more frequently (now twice per week), the team felt that the time spent during these meetings was beneficial. Once the programming was complete, it was time to begin testing not only that the programs worked, but that they worked based on the documentation and flow diagrams that were developed. Each member of the team was given a part of the system to test which was different from that part of the system that he/she had programmed. Since we were sure that errors had to exist, the goal was to uncover as many of them as possible before we turned the system over to the user and the vendor. Rather than be embarrassed by errors which were uncovered, team members would thank each other for discovering an error. For three months the team tested and simultaneously modified documentation, flow diagrams and programs. Although errors were uncovered, none of the corrections required a change to the basic design of the system. On September 21, 1993, the Touch-Tone Steering committee was given a demonstration of the mainframe application with simulated voice response, which would be used in the TTR system. It would have been nice to be able to report that the committee was completely satisfied with the new system. Unfortunately, although the system performed to specifications, the committee demanded changes that would affect the design of the system. This failing is clearly a result of the different time scales which the user community and computer center community operate. Users are embedded in their day to day concerns while giving only partial attention to specific fragmentary questions of design while the programmers and analysts are completely stymied by questions which are either not answered or answered insufficiently. The users on the Touch-Tone Steering committee who were expected to act as consultants to the project were not released from any of their normal obligations; therefore they had very limited time to devote to the project. Critically reading detailed documentation and finding time to meet regularly as a group proved impossible. In our case when the users finally gave their full attention to the Touch-Tone project they discovered that assumptions made by the programmers and analysts although workable were not to their liking and they demanded changes. This particular problem is one that deserves consideration by the administration for future large scale projects. CONCLUSION At the time of finalizing this manuscript, we have not yet implemented the touch-tone registration system, which is scheduled to take place in January, 1994. But whether or not we make the deadline the team feels that it has learned several important lessons, which have permanently changed the way we work, the way we interact with each other and the way we deal with the user community. We recognize and have gotten the computer center management to understand the importance morale plays in our performance. Whenever possible we encourage users to write notes of appreciation. As a team we celebrate our successes, and we have come to realize that our individual successes are in fact team successes. Although the monetary resources of a state institution are limited, management has tried to implement reclassification in a more timely manner to insure that the staff is working at its potential. In reflecting over the past two years, our team has come to appreciate how far we have come in improving our own work habits to provide better products and better service to our users. We turned a difficult situation into a window of opportunity to review our past practice, and explore innovative changes in methodology. Although meeting as a team is time consuming, we find that the time is well spent. The ability to discuss problems, issues and overall design objectives has cut down on possible errors, and has caused each of us to feel part of every project. Having clear documentation centrally located has enabled us to support one another and ultimately give better service to the users, who no longer have to wait for the availability of a specific programmer. Each of us, in one way or another has increased our technical competence, either through formal training, through workshops given by our colleagues, or through selfteaching. Each member of the team has a clearer understanding of the overall student information system. We have made strides in improving our communication with the users by maintaining a paper trail of requests, specifications and desired enhancements. By requiring a signoff of written specifications before programming and after job acceptance, we have increased the likelihood that we understand what the users wants and the users understand what they have been given. As a team we have worked to install a new system, beginning with a needs analysis and progressing to a description of the system, flow charts, topdown design, programming and thorough testing of the system. We feel that as a result of the rigid standards which we agreed to adhere to, we were able to create a firstclass product. Team 1 wishes to acknowledge that the success they have achieved could not have come about without the support and backing of its management, for which we are grateful.