The Process of Re-Engineering from Mainframe Systems to a Distributed/Client-Server Environment Copyright CAUSE 1994. This paper was presented at the 1993 CAUSE Annual Conference held in San Diego, California, December 7-10, and is part of the conference proceedings published by CAUSE. Permission to copy or disseminate all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, that the CAUSE copyright notice and the title and authors of the publication and its date appear, and that notice is given that copying is by permission of CAUSE, the association for managing and using information technology in higher education. To copy or disseminate otherwise, or to republish in any form, requires written permission from CAUSE. For further information: CAUSE, 4840 Pearl East Circle, Suite 302E, Boulder, CO 80301; 303449-4430; e-mail info@cause.colorado.edu The Process of Re-Engineering from Mainframe Systems to a Distributed/Client-Server Environment Ardoth A. Hassler Executive Director, Computer Center Leonard J. Mignerey Director, Management Information Systems The Catholic University of America Washington, D. C. ABSTRACT The Catholic University of America has recently completed converting 10- to 20-year-old legacy systems from a mainframe to a distributed/client-server environment using a common, integrated database. The solutions utilize a mix of institutionally developed software and purchased packages, including an ad hoc reporter/distributor. The paper describes what went right and what went wrong during the re-engineering process, including: * establishing that the re-engineering is an institutional issue and not a technical problem * building vs. buying issues * choosing the right tools * securing critical data resources * the necessity of user involvement * the need for standardization * issues of staff training and turnover * the applicability of Murphy's Laws The objectives of the paper are to share CUA's re-engineering experience with other schools that are facing major system conversions to integrated systems, a common database and/or a distributed/client-server environment. I. INTRODUCTION AND OVERVIEW OF CUA The Catholic University of America (CUA) is a private, religious affiliated, "Doctorate I" institution located in Northeast Washington, D. C. CUA has approximately 6,500 students, 60% of whom are graduate students; 400 full-time faculty and 1,100 staff. The computer center supports academic and administrative computing; voice and data communications units also report to the executive director. Beginning in 1969, CUA used mainframe computers from Digital Equipment Corporation's DECsystem10 family. In 1972, CUA was believed to be one of the first universities to run its administrative computing functions on a PDP-10. Support to academic and administrative users grew and improved through the years. In 1983, Digital Equipment Corporation announced that there would be no more computers produced in the DECsystem10/20 family, but that technical support would continue through December 1993. CUA formed a committee to make a proposal for replacement systems, which resulted in the purchase of Digital VAXes. The first VAX arrived in early 1986. All academic users were successfully moved off of the DECsystem10 to VAXes and microcomputers in calendar year 1986. The administrative conversion/re-engineering and the move to an integrated database of student and financial information was much more complex. Decisions were made early in the process that existing systems would be converted "as is" into a common, relational database, with only minor system modifications. By 1989, after nearly four years of effort, it was evident that the conversion was stalled as only one administrative system had been transferred to the VAXes. Changes in personnel at both the executive director and the director of management information systems level got the process back on track and led to the completion of the re-engineering process and the decommissioning of the DECsystem10 in July 1993. II. INSTITUTIONAL ISSUES VS. TECHNICAL ISSUES Support from the highest levels of the institution is probably the single most important factor in any re-engineering effort; it is critical for setting priorities and resolving issues between different offices. Having adequate resources is crucial to accomplishing the task. The process of integrating data from many sources required CUA to answer some questions that it probably would have preferred were never asked! For example, before the move to a common database, a student who was never admitted could register; a student who had never registered could pay a bill. Offices responsible for billing students were often also generating the charges as well -- a conflict of interest, at best. One of the best allies during the conversion process became the Internal Auditor. During the re-engineering, the business of the university did not stop -- students still needed to register; employees needed to be paid. It was necessary to maintain continuous operations in each administrative office and in the computer center. This placed a larger burden on the user offices than it did on the computer center since operations are done in production mode, with programmer intervention only required when a problem occurred. In only one or two offices was it possible to justify additional staff during the parallel operations phase of the effort. Major problems occurred when the offices involved viewed the re-engineering as merely a technical problem. Too often, the users regarded the conversion effort as an intrusion upon their work. In a few departments it was difficult to get users to provide such basic systems fundamentals as a requirements definition. It was often necessary for computer center management to elevate issues of non-cooperation to the executive vice president, who would then address those issues with the cognizant vice president for the area. The net result was delays and a few missed deadlines. Often, MIS staff were in the driver's seat for what should have been user decisions. One of the successes of the academic conversion had been the moratorium placed on new development on the DECsystem10. This was not done early in the administrative conversion and significant effort was expended modifying dead-end DECsystem-10 systems. In 1990, agreement was reached with the support of the executive vice president that only mission-critical changes and/or problem corrections would be done to DECsystem10 software. This allowed the MIS staff to focus on the new systems for the first time in four years. Issues of interoperability between some offices, such as sharing data and creating common procedures, are still being resolved. With the creation of a common database, responsibilities within some offices shifted. Because three vice-presidential areas are involved, some of those staffing issues still need to be addressed. III. BUILDING VS. BUYING ISSUES: A COMPROMISE The building vs. buying decision is one that often generates a tremendous amount of controversy. There is consensus among people who are willing to take an unbiased view that build vs. buy is not an "either/or" decision. At CUA, a split approach was taken. Software was developed in-house to enter the information into a data warehouse, which is the accumulation process. Software to extract information from the data warehouse, known as the usage process, was purchased. The HRS payroll/personnel package, purchased from Information Associates (now SCT), was the only exception to this rule. As was discussed earlier, the primary constraint on the conversion effort was the lack of time remaining before vendor support was dropped for DECsystem10 computers. Thus, when issues affecting the conversion were considered, parameters impacting the time line received greater weight than issues that did not. One of the primary factors of a build/buy decision is the proportion of resources that must be dedicated by the development staff vs. the proportion of resources that must be dedicated by the user staff. With a build decision, the heavier load falls on the development staff; with the buy decision, the heavier load falls on the user staff. At CUA, it was often the case that there was more institutional knowledge about departmental processes within the computer center than there was within a department. This was particularly true concerning the issues of integrating the functions of the individual departments into a process that would operate on a common database. For example, implementing the payroll/personnel package was tremendously difficult. Through no fault of its inherent capabilities, this package took over five years to implement and the effect caused an extreme amount of stress in the user community. All of its major features are still not being used. Thus, although it would seem counter-intuitive, it was determined that systems could be custom built faster and maintained with less effort than if they were bought. The general philosophy adopted by CUA was to build the core portions of a system and then to find microcomputer-based packages to handle specialized functionality within offices. As client- server technology evolves, a decline in the monolithic "do- everything" packages that are priced for large centralized systems is anticipated. Instead, there will be discrete modules that can exist in a distributed environment, and are priced in a more appropriate manner. For example, it does not make sense to pay for a university-wide license on a centralized financial system when there are only two or three people accessing various functions within that system. An example of the microcomputer approach is American Fundware's FINALYZER package. It is a NACUBO-approved package that creates specialized month-end reports, performs financial analysis, and creates standardized year-end audit reports. Similar plans are in place for the needs-analysis portion of the financial aid system. In addition, as previously mentioned, the ad hoc retriever/reporter already performs data extractions that feed into standard microcomputer packages such as WordPerfect and Quattro Pro from the database. A decision will probably be made to purchase a system to support the Offices of Development and Alumni Relations. Currently they are operating with a rudimentary, home-grown, microcomputer- based system. In these offices, there is strong user support for implementing and learning a purchased package. Also, this is such a specialized area that it would make little sense to reinvent the wheel. Because of the strong user commitment, the buy decision heavily outweighs the build decision. IV. TOOLS BEING USED One of the reasons for the success of the academic conversion was the willingness to learn from the successes and mistakes of other institutions that were further ahead in their conversions. This lesson, plus keeping abreast of the availability of software development tools and other changing technologies, became part of the administrative effort. At the start of the re-engineering process, all of the programming expertise within MIS was in COBOL. As a result, the first two systems that were converted -- facilities management and registration -- were written exclusively in COBOL, both the on- line applications and reports. This approach to writing interactive applications was very labor intensive and the results were sub-optimal. In the on-line applications, much of the screen navigation, scrolling in particular, required tremendous amounts of code and the resulting screen performance looked primitive. For the reports, COBOL programs were being developed to perform the function of commercially available ad hoc query tools.[1] When the current MIS director was hired, the facilities management system was complete and the registration system was about 50 percent complete. Graduate admissions was also operational, but was not in the relational database. The new director decided that stopping to retool the development process for the registration system could not be justified from the time- to-completion perspective. However, before the next system (admissions) was started, the approach was changed. First, the development approach to on-line applications was altered. The basic on-line application was split into four conceptual parts: * The user interface (screen handling) * The manipulative engine for performing calculation and enforcing the business rules * I/O modules for data transfer to and from storage * The database DECforms, Digital's forms package, was chosen as the tool for developing the user interface modules. Because it was the 1.0 version of the product, this choice was not without an element of risk. However, Digital indicated a firm, long-term commitment to DECforms. It was decided that coping with an early product release of this powerful product was a better use of time than to continue writing user interfaces in COBOL. One of the missing features of DECforms (since corrected) was that it could not do any mathematical functions. And, although at that time the product was written for character-cell terminals only, part of Digital's long- term commitment was future incorporation of graphical user interface (GUI) capabilities. As evidence of this commitment, there was a significant amount of GUI "look and feel" to the character-cell implementation. It should be noted that three years ago client-server was a new buzzword, so the GUI issue was nowhere near as large as it is today. COBOL was retained as the tool of choice for coding the business rules and performing calculations. In addition, the early DECforms applications required more coding in the COBOL portion than was desired. Though other languages were briefly considered, it made little sense to abandon the in-house expertise of COBOL. Plus, COBOL still fit this space well. The learning curve on DECforms has proved large and made management leery of too many additional changes. The I/O portions of the applications were moved to independent SQL modules. The database of choice remained Digital's Rdb. The second change was to find and implement a commercial ad hoc query tool. The search for an appropriate tool yielded a product by Software Interfaces, Inc. called SQLASSIST. One of the defining features of this product was its tight coupling with Rdb. It was the only ad hoc reporting tool that did not require a separate, proprietary data dictionary. In general, it has proven to be a successful choice. The admissions system, the first system to be undertaken with the new design approach, was completed without converting or writing a single COBOL report. This pattern, for the most part, held true for the remaining systems. A few reports were written in COBOL but this was due to limitations of SQL itself and not SQLASSIST. An added benefit to this approach was that selected members of the user community could also be trained to use this end-user tool. In certain areas this has resulted in a significant reduction in task requests to MIS for report development. In addition, the product has the capability to do data extractions, thus giving users the ability to produce extraction files to feed microcomputer-based products like WordPerfect, Lotus, Quattro Pro, dBASE, etc. As time progressed, staff turnover brought individuals onto the MIS staff who were not steeped in the COBOL tradition. This provided an opportunity to introduce a fourth generation (4GL) product, Digital's RALLY, into the development cycle. The financial aid system was targeted as a pilot project for this technology and a three-member team was formed to develop the system. Development on the other systems continued to follow the DECforms/COBOL/SQL-module/database hybrid, or 3-GL, approach. The jury is still out on the RALLY decision. On the positive side, a relatively inexperienced team was able to create and install a system at least as fast as experienced developers were doing with the hybrid approach. In addition, the prototyping capabilities of RALLY allowed for significant user participation in the initial stages of development. Very soon after the initial design meetings had been completed the users had a product that they could touch and feel. This created among the users a very strong feeling of project ownership, which proved to be invaluable during the inevitable stresses of systems development and deployment. On the negative side, RALLY performance is slower than the performance of comparable systems written in the hybrid style. In addition, because RALLY generates its own code, it is difficult to get an exact grasp of what the application is doing; it is difficult to optimize performance; and it is difficult for a developer who is not familiar with a particular application to maintain it. On small, simple projects, RALLY is a clear winner. However, when a project reaches a certain size and complexity, it is better to develop it with the hybrid approach. After a certain point, the development time needed to make RALLY do complex processing is greater then thedevelopment time using the hybrid approach, which is also easier to maintain. Determining the switch-over point is a matter of judgement and the process for making this type of decision is still being refined. When a clear decision cannot be made initially, one approach being considered is to start the project in RALLY. If the project moves easily to completion, it is kept in RALLY. If the project bogs down due to complexity, then it should be switched to the hybrid method. If this decision is made before an excessive amount of time is spent wrestling with RALLY, the prototyping and user participation advantage of a 4GL is retained while gaining the performance advantage of a 3GL. This positions RALLY more as a prototyping tool then a production tool. However, experience has shown that quite often the prototype and the production product can be one and the same. This "prototype as production model" has proven very useful in developing screen-based inquiry applications. An example of such an application is a budget inquiry application that provides the user with line-item budget information in both summary and detail form. It has proven to be one of the most popular applications and had a development time of about two weeks. Looking into the future, both of these approaches have positioned CUA for the client-server architecture. RALLY can run in client-server mode today, though CUA has not made use of this functionality. One of the powerful features of the product is that one can develop a RALLY application on a non-client-server platform and then convert it into a client-server application simply by changing a few setup parameters. The DECforms/COBOL/SQL module/database model also is well positioned for the client-server world. Each of the four components could potentially be made to run on a separate platform with very little modification to the existing application code. The most likely approach to switching to a client-server environment would be to move the DECforms portion to the desktop and leave the remaining three components on a central processor. A (true) pilot of client-server projects will be attempted within the next 24 months. V. SECURITY The security issue is divided into two components. The first is determining who has the authority to grant or deny access to the applications that access the common database. The second is determining how to implement these access restrictions in a secure manner. CUA has used the "data trustee" concept for at least 15 years -- user departments could decide who had access to what information. However, the computer center had to perform the functions that granted user access to data. It was felt that this was no longer an appropriate situation and that there should be a least one person within each major office (for example, the registrar or dean of admissions) who was responsible for the data that related to that office. In 1986, a "one-user-one-password" philosophy was adopted. In keeping with this philosophy, it was felt that after a user logged in to his/her personal area on the computer, the system should be smart enough to know what that person could or could not do relative to the administrative systems. To meet these parameters, a security package called ADMENU was developed. It is a menu-driven package that presents each user with a customized view of the administrative systems and the applications that are available on each system. For example, when a user types the command ADMENU, s/he might see the registration and admissions systems listed. Other systems would not be listed because this user had not been granted any access to them. The user would then select a system, such as registration. A new menu appears, showing only those registration applications to which the user has been granted access. The user then selects the desired application. The user's process is now put into a captive state: the default directory is changed to an appropriate location; access identifiers to the executables and the database are dynamically constructed; process quotas are dynamically increased; and appropriate logical name tables are assigned to the process. The user's process then runs the requested application. When the user exits this application the setup is reversed and the user is returned to the ADMENU screens. Three special applications have been created within ADMENU to administer this system. System trustees have the ability to create new systems in the administrative suite of applications; the computer center executive director and the director of MIS are system trustees. Application trustees have the ability to create new applications within an existing system; systems analysts in charge of each of the systems are the application trustees. Data trustees have the ability to change the list of who can access existing applications within a system. Data trustees are usually the highestlevel administrator within a functional area and include the registrar, dean of admissions, treasurer, controller, director of financial aid, etc. In addition, all the trustees have the ability to create a peer trustee. This allows an upper-level administrator to share the responsibility with a trusted colleague. As a final level of security, should there be a security problem that needs immediate attention, the system trustees have the ability to remove a user from anywhere in the system with a single command. VI. THE NECESSITY OF USER INVOLVEMENT The importance of user involvement in a development project cannot be overemphasized and conventional wisdom states that one should not start a major project without it. This holds true when the cost of proceeding with weak user involvement is potentially greater than the cost of delaying the project until user involvement can be assured. CUA did not have the option of delaying its re-engineering project. Support for the DECsystem10 was going to be dropped whether systems were converted on time or not. It was only an issue of whether or not CUA would still be able to do business after December 1993. Many of the users viewed the re-engineering process as a computer problem, not their problem. Also, because each department had been working in its own stand-alone system for so long, there was a low level of interest in the issues of other offices. Oftentimes, issues that did not pertain to a specific office were viewed as someone else's problem. Systems analysts often heard the phrase, "Well, I don't know what they do with it [the data] over there but...." In a number of offices this attitude was coupled with resistance to computing as well. For example, the admissions office only entered data into their DECsystem10 applications to avoid being reprimanded by upper-level administrators. Their real system was on intricately designed 3X5 cards. In spite of this environment, MIS was able to proceed because of the cumulative institutional knowledge of the MIS staff and because of receptive employees who were targeted as champions within each of the departments. In the build vs. buy section, it was mentioned that MIS could both develop and deploy systems in a shorter time frame than would be possible with purchased software. For any major development project to succeed there has to be strong expertise in either the development staff or the user staff. In CUA's case the user environment was weak and fragmented. However, within the twelve (now eleven) member MIS staff, six members of the staff had an excess of ten years experience at CUA and had also developed many of the existing DECsystem10 administrative systems. Hence, much of the necessary knowledge to integrate the disparate systems existed in-house. Within the user environment there were a few areas that advocated new technology. The financial aid office was a good example of this. They consistently supported MIS within the user community. Having a least one large department as an ally was extremely important and MIS, of course, made every effort to nurture these relationships. In addition, receptive individuals were found and supported in the other areas. Within the registrar's office, that person was the assistant registrar. In the financial area the vice president was unsupportive, but the controller was very supportive. Within the admissions office the receptive person was a newly-hired employee who was responsible for doing enrollment analysis. This individual's obvious need for accurate data made him a natural ally and he was the first user to make extensive use of the ad hoc query tool SQLASSIST. MIS supplied extensive support and encouragement. In return, he proved to be an invaluable resource within the user community. As the other departments started to notice the information and resources that he was able to access and use, they developed a natural desire to have the same capabilities. VII. STANDARDIZATION CUA seized the re-engineering effort as an opportunity to develop universal codes. The legacy systems had often been developed without the consistent use of codes between systems. A Universal Codes Committee held numerous cross-campus conversations on standardizing the codes. The result was not only better codes, but improved communication. The move to a relational database also afforded the opportunity to store address records for students, for example, in one common place with definitions of who had authority to make changes. Previously, a student's address had to be changed individually in each system. Because CUA is moving to a client-server environment connected via a campus-wide network, the tools to support this environment are very important. CUA now has a policy in place that the computer center will review all administrative hardware purchases. (The academic review is the next step and that goal appears closer on the horizon.) Site licenses and quantity purchases have also encouraged users to stay with software that is commonly used and more easily supported. VIII. ISSUES OF STAFF TRAINING AND TURNOVER One of the biggest, single factors in the entire process was the amount of staff turnover inside the computer center, within the administration, and in the user community. For example, during the conversion/re-engineering, CUA had two executive vice presidents, two computer center executive directors, three MIS directors, three vice presidents for finance and treasurer, two registrars, three controllers, and many, many programmers. The computer center was fortunate that its senior-level technical staff (systems analysts) was stable during the process. Many of the analysts as well as the MIS director and executive director had worked for CUA for 10 or more years and had a wealth of institutional knowledge. This was a double-edged sword however, as it required changing numerous "but we've always done it that way" mind sets. Training and a mandate (with constant reminders) by the current MIS director to look for new solutions, rather than to reuse old ones, made the difference in the ultimate outcome of the project. As stable as the senior staff was, the programming-level staff had a significant amount of turnover. Computer center management eventually came to accept the turnover as the norm rather than as the exception. Previously, the MIS director had trained new people on both systems. The current MIS director quickly abandoned this concept, assumed that entry-level programmers would not be with CUA more than two years, and worked toward quickly getting them to be as productive as possible within the limits of their capabilities. The turnover in the university community provided another set of joys and concerns. The new managers that joined CUA all tended to be more "computer literate" than their predecessors. This was another double-edged sword -- they were more knowledgeable, but they were also more demanding. The turnover also meant that much institutional knowledge was lost -- again, a good news/bad news situation. It impacted the conversion in several ways: Computer Center personnel were frequently training office staff; they were also often in the position of having to tell users "no" or to expect a long wait for changes requested by savvy, new users. All too often, computer center personnel provided the details of how an office operated rather than the office itself specifying how it wanted to operate. IX. THE APPLICABILITY OF MURPHY'S LAWS Murphy worked overtime during both the academic and administrative conversions. Several of the problems were unexpected events in the lives of the staff: two people had to have unplanned major surgery; one was involved in a serious automobile accident; deaths occurred within extended families; one staff member (hardly unplanned) had a baby. These events provided challenges to management as they attempted to keep schedules and meet deadlines. There was also a false start in the selection of the relational database. Nearly a full year was lost selecting and trying to implement a third-party database. At that time, the database selected had been recently ported from an IBM system. It did not take advantage of the features and functionality of the VAX architecture. Further, staff found themselves one step ahead of the vendor's trainers who were supposed to be the experts. One of the biggest problems occurred in a user office. Three months had been allocated for parallel operation. During the first month, nothing was entered in parallel into the new system. When this was discovered by computer center staff, the office users were convinced to begin using the new system. The good news/bad news was that they liked the new system so much that they stopped entering data into the old system. At the point this was discovered, the audit trails had been lost and it was impossible to bring the data in the old system up to date. It took the controller several months to reconcile the new and old systems into the financial accounting system. Procedures were put in place to ensure that this did not happen when later systems entered the parallel testing phase. The previously stable hardware also began a series of failures about nine months before the project was complete. Given the age of the DECsystem10 and its complexity compared to today's systems, a week-long outage with lots of power interruptions started a downward spiral of crashes and repairs that caused long periods of system unavailability. A power outage even took the system down the day of the decommissioning party. X. CONCLUSIONS/RECOMMENDATIONS CUA is now in the happy position of having its student and financial information in a common, relational database. The legacy systems are gone and, among other things, there is no need to worry about systems breaking at the turn of the century because dates have been expanded to four digits. The effort to accomplish this task was a major one. As it was being accomplished, the demands for enhancements and improvements, not to mention the inevitable problem corrections, continued to build. MIS is now in the process of addressing these issues. Still remaining is the need for a good development/alumni system that will work with the common database. This system will probably be identified and bought within the next year. There is no question that CUA did not "do it by the book." However, the job was complete six months before vendor support ended. Although the expertise of the MIS staff was a large factor in the success, the effort could not have been accomplished without finding and nurturing interested individuals within the user community. These individuals not only helped make the re- engineering a success but they have provided an even greater service by raising the expectations for information use within their respective offices. During this process there has been extensive staff turnover in the administrative offices. Many of these champions within the departments have had the opportunity of serving on the search committees for new employees. This has helped to bring about a consistent improvement in the computer literacy of the newer employees which has also increased expectations of computer literacy on the existing staff. The last three years have seen a demonstrable change for the better in the relationship between the MIS department and the user community. A true collaborative process is starting to emerge for the first time and it holds great potential for the future. In summary, other institutions who undertake the complete redesign of their legacy systems, whether to another mainframe or to a client-server environment should be aware of, prepare for, and do the following: * Make sure the president and vice presidents all understand that the effort is an institutional one and not simply a technical change. * Get support from top management. * Establish a moratorium on all but absolutely necessary changes and development to the old systems. * Accept that the unexpected will be the rule and not the exception. * Keep current and force the staff to keep current with the tools and technology. * Enlist the support of the user community -- even if it is at a "grass-roots" level. * Empower the user community with desktop tools so they can perform as much of their own work as possible. * Learn from the mistakes of others. [1] It should be noted that CUA participates in Digital's Campus- wide Software License Grant (CSLG) program. This allows CUA to use most of Digital's software at no cost unless the product is royalty based, in which case it is deeply discounted. This heavily influenced some of the software choices that were made.