Distributing CWIS Information Management Copyright CAUSE 1994. This paper was presented at the 1994 CAUSE Annual Conference held in Orlando, FL, November 29- December 2, and is part of the conference proceedings published by CAUSE. Permission to copy or disseminate all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, that the CAUSE copyright notice and the title and authors of the publication and its date appear, and that notice is given that copying is by permission of CAUSE, the association for managing and using information resources in higher education. To copy or disseminate otherwise, or to republish in any form, requires written permission from CAUSE. For further information: CAUSE, 4840 Pearl East Circle, Suite 302E, Boulder, CO 80301; 303-449-4430; e-mail info@cause.colorado.edu DISTRIBUTING CWIS INFORMATION MANAGEMENT Robert J. Brentrup Dartmouth College 6028 Kiewit Computation Center Hanover, NH 03755-3523 Abstract A campus wide information system is only as good as the information in it. Once a CWIS is in place a tremendous amount of effort is needed to "stock the shelves" with information. Removing obstacles in the data maintenance path is just as important as making the data accessible to users. Placing full control of the data in the hands of its "originators" increases the quality and quantity of information available. The Dartmouth College Information System (DCIS) has developed a number of tools used to enhance the information publishing and maintenance process. A locally developed client-server database update system provides easy to use links between standard personal computer applications used to develop and maintain databases and the database delivery systems. Automated format conversions, integrity checks and update authorization are the principal features provided by the update system. The Context Dartmouth College has been developing an extensive Campus Wide Information System (CWIS) for the last four years[1]. This client-server system is widely and heavily used and operates in an environment where virtually all faculty, students and staff have a networked Apple Macintosh computer. The system hosts highly structured data that can be located and retrieved through sophisticated database managers. The retrieved information can be displayed with extensive typography and embedded images (eg. Figure 1). The CWIS is much more than a shared file server, as it answers complex research questions for users. [FIGURE NOT AVAILABLE IN ASCII TEXT VERSION] The Dartmouth College Information System (DCIS) servers have been adapted to a number of different database management systems using commercial and academic software. Access to local information such the Oracle based administrative systems and gateways to other services on the internet are provided. DCIS is portable and installed at several other institutions. The two database management systems most frequently used at present to "publish" information in DCIS are the BRS[2] and PAT[3] commercial systems. Both of these systems run on UNIX computers. At Dartmouth these UNIX systems are now workstation-class computers with expanded disk storage capabilities. The Problem One of the primary factors limiting growth of DCIS was the rate at which new information sources could be developed and the amount of labor required to keep existing ones up to date. The computing staff at Dartmouth is small. As DCIS became more popular we found ourselves facing a long list of requests for assistance to place some portion of the institutional information base on line. Our ability to respond to these requests became a bottle neck and source of frustration. Even if adequate staff resources existed, their skills are best deployed on further extensions and development of the system. It became apparent that it is crucial to decentralize the development and maintenance of new information resources. At Dartmouth and probably most campuses, many offices and many people already generate and update valuable information. Harnessing that diffuse manpower in an organized way is the key to the continued growth of a CWIS. The primary obstacle to remove at Dartmouth was the complexity of "publishing" on the UNIX-based DCIS servers. Current Publishing Practices Many administrative departments are in the business of publishing information. In many cases the source data are already prepared and maintained on desktop computer systems. Periodically the information is printed and distributed. The CWIS is another and increasingly popular method to "publish" information. Its greatly enhanced distribution speed is the most important reason. By publishing on the CWIS the most current information is always available and it is easier to locate specific sections. In some cases the cost and labor associated with paper delivery can be avoided. There is, however, a need to identify the structure of the data in more detail to permit the computer systems to index and display the data more intelligently. Unfortunately the desktop systems used to enter and maintain the original information generally lack both the software and the processing power to provide database services to large numbers of users. For example DCIS typically supports thousands of database sessions daily and several hundred simultaneously. The workstation systems that can solve these problems are however more difficult to use and need to be relatively closely controlled to ensure high reliability and data security. For example, users would need to learn to use the workstation operating system and have accounts, while system managers would have to create and maintain these accounts. To resolve this dilemma DCIS has developed a new database-update system which enables the originating offices to add to and update information resources almost as easily as the rest of the campus uses these resources. Variations on the Theme Although not called a "campus wide information system" at the time, Dartmouth has had elements of this idea in the computing environment since the first computer systems were installed on campus in the 1960s. The Dartmouth College Time Sharing system (DCTS) supported a number of institutional databases and provided shared libraries of programs. In 1985 a large AppleShare file server capable of supporting several hundred simultaneous users was implemented on the DCTS hardware. Known as "PUBLIC" this server is a distribution center for the entire campus for programs and general administrative and academic information. This worked for Dartmouth because the campus also standardized on the Macintosh computer so everyone had access to this "CWIS"[4] and knew how to use it once they knew how to use the Macintosh Finder. These predecessors of the current client-server environment particularly the PUBLIC file server, had some interesting attributes when looked at as an "information publishing" tool. PUBLIC was the first place one would look for information. It had a very simple interface for users and information providers: you simply copied files from or to the server as you would with any other Macintosh disk. Locating information on PUBLIC is accomplished by manually opening and closing each folder or by using the Macintosh Finder's Find File functionality with which you can search for files by name. You typically browse through the various folders until you find what you are seeking. As the amount of information grew larger, this approach became more difficult and time consuming. But the simplicity of the PUBLIC file server, particularly for information maintenance, allowed it to be easily used by most members of the Dartmouth community. Locating information can be greatly enhanced by going beyond the shared file-server model. The primary step is to provide content indexing. While some information sources fit very nicely in the file-server model, there is another class of databases (eg. library catalogs and reference works) that are too large and complicated for this approach to be reasonable. This is the class of information that DCIS first addressed four years ago. The files are large, in most cases, and internally highly structured (ie. fields and records or tagged elements define semantic structure and display attributes). This structure allows users to formulate detailed questions and receive quite specific answers to their inquiries. An example best illustrates this point. "Rental Housing Alternatives" was originally a program to which one connected on the time sharing system. You answered a few predetermined questions, such as "Are you interested in apartments or houses? summer or year round?" Based on your answers an appropriate portion of the "database" was selected and transferred to your terminal. You then read through the list to identify the items of interest. Taking this same data and describing the regular internal structure of it for a database manager allows the user to ask specific questions such as "find furnished houses in town X and price range Y to Z" and subsequently end up with a list of items matching only those criteria. Such specific results are possible because of the additional effort invested in providing more structure and some uniformity in the data and having software that can use multiple indexes and value comparisons. Retrieving the whole list is still easy. Project Objectives and Results The goals of the updating tools project were to make the database creation and update process as simple as possible, to empower users with moderate computing skills to be able to control the process, to eliminate the need for knowledge of "mini/mainframe" database maintenance tools, to eliminate the administrative overhead of creating and maintaining accounts on server systems and most importantly, to eliminate the need for continued assistance from the software development staff. The solution DCIS developed is another client/server system that allows staff who are not computing professionals to "publish" their own structured information on the CWIS. The update client application runs on the same desktop computers used to prepare the data. Several other tools were developed to streamline the process of preparing, checking and transferring updates. Together these tools create an efficient link to move the data from the originators' desktop workstations to the CWIS central computers. The update clients and servers are now in production use and spreading rapidly. Some of the current examples include: * Student phone directory * Career Services, leave term jobs * Student newspaper * Art History department slide collection * Computing newsletter * Photo records catalog * Instructional Services media catalog * Rental housing alternatives * Student employment opportunities * Used computer equipment for sale Design Considerations The inspiration for this solution was motivated by having to handle these requests and be involved in the update process. After a while a pattern of usage emerged. We attempted to automate regularly repetitive tasks in order to make the process as simple as possible. We wished to address three important aspects of database maintenance. For reliability it is important to not allow incorrectly formatted data to be added to the database. For security it is important to prevent unauthorized modifications and to limit update access to approved personnel. To insure the integrity of the database it was most desirable to have its creator in control of its maintenance. The database creator is the best judge of correctness and timeliness. A number of different Macintosh software packages are being used to prepare the databases depending on their requirements. The most popular at present is Claris FileMaker. By adding an export script to the FileMaker database, the export process is accomplished in one command. The Acius 4D application is another popular tool used to create departmental databases. Export procedures can be programmed in 4D, and it has built-in functionality to produce single table output in common forms for export. On the CWIS server side a number of different database managers are being used. Each database manager has some specific strengths from which certain types of data sources benefit. Each has a required import data format that needs to be accommodated and specific loading programs to build the database and create the index points. These details of the database management system should be transparent to the update user. The database update requirements vary as well. Some of the files are added to on a regular basis (eg. the campus computing newsletter and the student newspaper). Others are periodically republished in their entirety. It is also desirable to be able to easily fix data errors in a previously posted update. This requires that identifying the proper portion of the information be straight-forward. How DCIS Update Works The database creator prepares the source data on his or her Macintosh. Some use a Macintosh database manager such as FileMaker as an input tool. Other applications use a word processor or even a document editor such as Aldus PageMaker. Next the database creator exports the data from these applications into text files. Then he or she converts the export files to the input form of the database manager used to publish it. This step describes the internal structure to the database manager. This operation is typically a mapping of database field names or the addition of SGML5 tags. Several Macintosh utility applications have been developed by DCIS to perform or assist this step. One converts FileMaker export files to the BRS load format. The file converter provides a table driven field mapping. Another checks the integrity of an SGML markup. The SGML analyzer has user configurable markup symbols, lists the nested tag structure of the document, the entity references used and the extended ASCII characters in the file. Structure errors are diagnosed and located by line number and context. The database provider then runs the DCIS Update client (Figure 2) to transfer the data to the server. The update client connects to the update server and gathers the list of databases available for update. The client prompts the database provider to select the database they wish to update (Figure 3). The client then checks the identity of the user to determine which of the databases available are open to updates from this individual. The person is authenticated by entering their network services password, which is checked against the campus wide "Dartmouth Name Directory." Tables in the update server list those people who have been given "write" access to a database. [FIGURES NOT AVAILABLE IN ASCII TEXT VERSION] The database provider then selects the update function desired from the available buttons. The system supports reload, add, delete and replace operations, which are enabled depending on the needs of the particular information source and the functionality of the underlying database manager. The update program prompts for any other information it needs, generally just an identifier for the update set and the data file to use. Only appropriate data file types can be selected preventing another possible cause of errors. The data file is transferred to the appropriate server machine, which starts to run the update script. The update server runs a script to handle the requirements of a particular database. The script handles both consistency checks and runs the various vendor-supplied server database maintenance utilities needed for this update. The update script first verifies the input data, often running a format checking program which must succeed in order to continue the process. Next the script attempts to process the update transaction. In some cases several periodic updates are merged into a single file. Next the script runs the database manager's loader or index building programs. A log window shows progress and provides diagnostic messages. Finally the success or failure of the operation is reported (Figure 4). The log messages can be mailed to a DCIS staff member for diagnosis in case there is a problem. On-line help is available. [FIGURE NOT AVAILABLE IN ASCII TEXT VERSION] Evaluation The DCIS update system has proved to be quite popular and has successfully accomplished the task of transferring the ongoing maintenance of the database to the originating office and freeing up the DCIS development staff. Several of the databases are being updated on a daily basis. Others monthly or as the need arises. Prior to this project "regular" updates were not getting processed on a regular basis. Some of the databases supported by the update system are among the most popular resources on DCIS. As the DCIS system continues to expand, an increasing number of uses for the update system are being identified. The client will run on almost any Macintosh. The servers are small, fairly portable and adaptable to other database systems. The system has also been easy to configure for additional databases. This project has measurably increased the usage and local satisfaction with the DCIS effort. Footnotes 1 Robert J. Brentrup, "Building a Campus Information Culture", CAUSE/Effect , Vol. 16 No. 4 Winter 1993, pp. 8- 14. 2 A product of Dataware Technologies 3 A product of Open Text Systems Inc. 4 For many other institutions their first experience with this kind of shared information resource was the Gopher system. For Dartmouth the main appeal and use of Gopher has been to share information with the world outside the campus. [5]Standard Generalized Markup Language Nov. 8, 1994 (RJB)