Automating Production Support Copyright 1993 CAUSE From _CAUSE/EFFECT_ Volume 16, Number 1, Spring 1993. Permission to copy or disseminate all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, the CAUSE copyright and its date appear,and notice is given that copying is by permission of CAUSE, the association for managing and using information resources in higher education. To disseminate otherwise, or to republish, requires written permission.For further information, contact CAUSE, 4840 Pearl East Circle, Suite 302E, Boulder, CO 80301, 303-449-4430, e-mail info@CAUSE.colorado.edu ATOMATING PRODUCTION SUPPORT by John Rogers ABSTRACT: In this time of fiscal restraint and technological (r)evolution, it has become very important to reduce the amount of time and money spent on supporting our "legacy" systems. At Dalhousie University, the small team that makes up the production support staff has implemented several automated innovations that are proving to be very time-saving and cost-effective. Legacy support Whatever support methodology has been followed, Dalhousie University's traditional legacy systems have been held together, one way or another, by those dedicated professionals who live in the realm of maintenance. With the exception of a few interfaces, most applications were designed and are maintained independently of each other, and support of each system has taken on its own unique characteristics. Myriad software tools and vendor-written code have meant that each application might require its own guru just to hold it together. However, within each system there are several aids to help the maintenance programmer, one of which is the "abend module."[1] Nearly every application, whether developed or purchased, has an abend routine. The abend routine usually produces an elaborate screen for the end users, to prevent them from seeing those cryptic T/P monitor messages. Besides online abends, most applications also involve some batch work, the results of which need to be checked. And in the bigger realm of "production support," all applications require management of their file space, and so forth. The problem With the limited resources in the department (we have four support staff to support all of the mainframe applications running on our IBM-4381), it became desirable to streamline those commonly found support processes mentioned above. Each system had its own online abend routine, and if errors were going to be caught and dealt with, first a programmer had to be aware the abend had occurred, and second, he or she had to be able to determine how it had happened. We soon realized that it would be very advantageous if we could somehow "capture" any and all abends in our systems and have the information "delivered" to the programmers in a timely fashion. While solving this problem, we realized the potential for also "delivering" information on batch jobs, and data management. Our support staff were soon to be inundated with information! The solution Electronic mail--the answer was so obvious and had been available for years! After consulting with one of our technical staff who runs the mail system, we found that we could mail interactive information or actual datasets to anyone; it was simply a matter of gathering, formatting, and passing the information into the mailer. We started with the abend routine for our student information system--it was already being called by every SIS program if a problem occurred and it received a lot of the pertinent information (the name of the program, the paragraph, the action taken, and the error). We combined that information with a system date and time stamp, along with the user ID of the person who encountered the error, and bundled the whole works up and passed it to our mailer, with a specified ID as the destination. It worked--and it only took about two hours to implement! We soon had all of the applications' abend routines tied into the mailer and had added the e-mail addresses of our support staff. Thus, as any abend occurred in the system, our support staff were mailed the results of the abend and could be on the phone to the client while the evidence was still fresh. Of course, on the down side, we found that there were a lot of loose ends in our applications and we were initially swamped with e-mail messages. However, after a few weeks of putting the major fires out, it became a very manageable and useful tool. We continue to be made aware of system bugs or our client's "deviant" processing techniques. Expanding the technique Once the online abend processing was in control, it was time to set our sights on batch job processing and data management issues. On learning what we had done with the online processing, our systems manager wrote a quick procedure that would scan the system logs and extract all the pertinent information we needed about jobs with non-expected results. Now our support staff are getting a daily early-morning report of any potential or real problems. Even our main client liaison in the finance office gets a copy showing the results of the previous night's financial processes. It wasn't long before we were talking about resource management and within a few days we were getting automated reports of any extensive spool wastage (large reports sitting online without being printed or deleted). As well, we tackled the diagnostic log of our aging SYSTEM 2000 DBMS and now receive a report of any database accesses that take longer than 10 seconds. This tool identified some rather nasty accesses that required some tuning! Conclusion To be honest, we don't know how we lived without this kind of management tool in the past. We have become more and more dependent on it and our clients are impressed with our seemingly ESP-like response to problems. The most impressive aspect of our solutions is that the total time involved to automate all of the above processes was under five days! That five-day investment has more than paid for itself. ======================================================================== Footnote: 1 An "abend module" is a procedure that is invoked when an application program does not complete to a normal end. ************************************************************************ John W. Rogers is Manager, Production Support, Administrative Computing Services at Dalhousie University in Halifax, Nova Scotia, Canada. He invites anyone who is interested in the specifics of the "good idea" described here to send e-mail to jrogers@adm.dal.ca or call 902-494-7129. ************************************************************************