Observations on Benchmarking InformationTechnology Support Copyright 1995 CAUSE. From CAUSE/EFFECT magazine, Volume 18, Number 1, Spring 1995. Permission to copy or disseminate all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, the CAUSE copyright and its date appear, and notice is given that copying is by permission of CAUSE, the association for managing and using information technology in higher education. To disseminate otherwise, or to republish, requires written permission. For further information, contact Julia Rudy at CAUSE, 4840 Pearl East Circle, Suite 302E, Boulder, CO 80301 USA; 303-939-0308; e-mail: jrudy@CAUSE.colorado.edu OBSERVATIONS ON BENCHMARKING INFORMATION TECHNOLOGY SUPPORT by The Office Of Computer and Information Systems, The Pennsylvania State University ABSTRACT: Wanting to evaluate their own progress in information technology programs and practices, Penn State's organization collected information about successful operations in five similar institutions. Although differences in budgeting practices made it impossible to make direct numerical comparisons, they found four broad indicators of model practices. In the spring of 1994, the Pennsylvania State University's information technology organization began an aggressive program of benchmarking Penn State's information technology programs and practices against those of other major research universities. Over a three-month period, the University's Executive Director of Computer and Information Systems and various combinations of five of his managers visited five institutions determined to be "best-in-class" in their use of information technology (UCLA, the University of Illinois, the University of Michigan, the University of Texas at Austin, and the University of Wisconsin at Madison). These institutions were selected for two reasons. First, they are vigorously and successfully pursuing information technology solutions in a wide range of areas (e.g., academic, administrative, and library computing; telecommunications; computer security). Second, they closely match Penn State--a large, complex, public land-grant institution--in mission and size. The primary goals of this effort were to identify the underlying processes that facilitate excellence and to determine how similar processes at Penn State can be improved or reeingineered. While all six institutions are trying to adapt rapidly and effectively to technological change, it became evident very early in the benchmarking process that there is no single "right way" by which to address this need. Each institution has its own culture, and practices that serve one environment may not be applicable to others. Recognizing our own institutional biases, we made a conscious effort to factor out the Penn State view in identifying processes we felt to be most significant. Institutional differences were at least in part responsible for what proved to be our difficulty in comparing numerical data across institutions. Collecting "numbers" was never a limiting goal of this benchmarking exercise, but after analysis the numerical data proved to be the least valuable information obtained. In fact, we found that, with the exception of the most simple measures (e.g., percent of buildings with adequate inter-and intra-building telecommunications wiring), we were sometimes not just comparing apples and oranges, but apples and orangutans. For example, where some institutions include capital expenses in the information technology budget, others do not; comparisons would be irrelevant, at best. Thus, while benchmarking is usually synonymous with numeric comparisons, this article, which shares our benchmarking experience and findings, contains no specific numbers. Although our study did not harvest numeric measures, we were able to identify four broad principles that we believe offer major research universities sound guidelines for change. The degree to which institutions have internalized these principles seems to be directly related to the excellence that we perceive to characterize best-of-class institutions. ============================================================ Principle 1: Best-in-class institutions use policy, budget, and strategy measures to maximize the benefits of information technology. ============================================================ POLICY MEASURES We found that best-in-class institutions are struggling to address a number of pivotal policy issues. These can be characterized as follows: Defining the appropriate institutional roles and relationships for senior information technology management Many institutions are struggling to formalize the institutional role of the senior information technology officer, and few recognize the senior information technology officer as a member of the "president's cabinet." As the importance of information technology increases in all aspects of university operation, the senior information technology officer must be more directly involved with senior university management. For example, the broadening role of information technology tools in all disciplines means they will demand a larger proportion of a university's budget. Yet few institutions have formal relationships established to enable frequent exchanges between the chief financial officer and the senior information technology officer and their staffs. Information technology is a key factor in the future competitiveness of research universities, and all institutions in our study agreed that it is critical for senior information technology officers to have a "seat at the table" as a key advisor. At least one benchmark institution has moved to place the senior information technology executive as a member of appropriate university-wide, strategic/advisory/planning committees. This positive move recognizes the increasing importance and the increasing budget share that technological growth will require. (The benchmark institutions noted that if the most senior administrators are well-versed technically and aware of the many difficult issues surrounding information technology implementation in complex institutional environments, direct involvement of information technology officers may not be as critical. However, it may not be practical or prudent to expect such detailed knowledge from senior decision-makers.) Encouraging process reengineering Best-in-class institutions are attempting to reevaluate and reengineer both business and instructional processes to take advantage of technology. In business process reengineering, senior management support greatly enhances the probability of success. For example, the Senior Vice President for Finance at UCLA has taken leadership in eliminating paper and converting current business processes to electronic processes. Particularly noteworthy in this reengineering effort is the adoption of post-audit procedures for electronic forms, in which most forms can be acted upon without a lengthy pre-approval process. UCLA expects to save $500,000 per year in data entry costs and recognize significant reduction in key business office staff as a result of this practice. The large research institutions surveyed also recognize the need to evaluate and encourage the reengineering in instructional delivery that technology can enable. Two of the six institutions are strongly committed to developing innovative instructional uses for technology, with the support and personal involvement of interested faculty. The other benchmark institutions are in various stages of implementing similar programs. Nationwide, classroom instruction is evolving from the standard lecture format to collaborative learning, where the instructor is more of a mentor or coach than a dispenser of knowledge. This transition will rely heavily upon investments in information technology. Implementing incentives to overcome institutional inertia toward adoption of new technical standards We found near unanimity in the practice of offering financial incentives for users to migrate away from obsolete technology. The most widespread example of this is in charging for outmoded SNA connections to encourage use of TCP/IP as an Internet working standard. Five of six institutions currently charge for SNA network connections while offering TCP/IP at either no or greatly reduced cost. Internalizing or institutionalizing the development of information technology staff Those institutions that appear farthest along have managed to couple institutionalizing training with human resource policies that recognize training's implicit value. The University of Texas has a novel approach to administrative programmer development that implements a well-defined career progression from hiring through eventual placement of trained technical staff in the colleges and departments. This is only possible with the active cooperation and partnership of Human Resources. Engaging users effectively in application development We found two differing views of how to achieve the common goal of enabling or encouraging the user community to participate in the development of computing applications. The first model places technical responsibility closer to the level at which the end product will be used; the second maintains central responsibility but emphasizes coordination to ensure user satisfaction. The effort at the University of Texas involves direct participation of the user community in the development of administrative computing applications. In the Texas model, support at the user level is provided by programmer/analysts who began their careers in a very comprehensive, centrally managed training program. Penn State has also distributed to using organizations the responsibility for much administrative applications development, though the training infrastructure and hiring/promotion paths are not as clearly integrated into the institutional fabric and human resources policies as they are at Texas. The institutions that have developed this type of direct-support model believe that vesting technical capability and responsibility in user organizations allows better response to user needs. Other benchmark schools encourage user participation through close coordination and consultation with user organizations to identify requirements and necessary resources. However, actual development, testing, and implementation is done by the central information technology organization. Those that have adopted this model believe that as long as user requirements are adequately addressed through consultation, centrally managed technical efforts provide a uniform level of support, consistent standards, and common solutions that can be applied to more than the requesting organization. They feel that this helps avoid situations where programs are departmentally rather than institutionally optimized, resulting in redundant effort, incompatible solutions, and cost inefficiency. BUDGET MEASURES Each institution's budget figures reflect individual institutional accounting practices, making direct comparisons difficult. Our comparative data are necessarily approximate and no specific numbers are included below. Nevertheless, it is possible to make a number of general observations about budget practices that affect information technology. A hurdle all institutions are facing is the fact that information technology organizations must meet the unparalleled growth in demand for services in times when university budgets are tightly constrained. To achieve the needed levels of service in the current fiscal climate will require internal reallocation of institutional funds and innovative alternative funding strategies. Central vs. distributed funding issues While computing itself is becoming more distributed, distributing funding for computing is not universally advantageous. All of the benchmark institutions are investing heavily in central information technology organizations and, in some cases, are actually recognizing a cost benefit by centrally funding certain services. An important theme in our discussions was the need to find innovative solutions to funding needs while achieving an appropriate balance of central and distributed funding. Perhaps the most interesting budget observation was that some of the best-funded institutions not only spend a higher absolute dollar figure on information technology infrastructure, but they also invest a significantly larger percentage of their overall budget than some of the more poorly funded institutions. This may well be a result of a longer history and a better institutional understanding of the importance of investing in information technology as a critical institutional resource. A related and similarly important observation is that there is no direct relationship between the extent to which a university is centralized and the level of central investment in information technology infrastructure. Some of the more traditionally decentralized institutions make significant central investments in information technology infrastructure. Their level of investment in central information technology services frequently exceeds that of the more centralized institutions. Our study found that distributed funding of widely used services may, in some cases, increase the institution-wide cost. Central funding can result in de facto standards that reduce the cost of a widely used service, such as e-mail or site-licensed software, both for support and the actual cost of the service provided. Moreover, centrally funding items such as training may provide a baseline for user skills, improving overall institutional efficiency and productivity. The University of Illinois estimates the cost savings for centrally funding selected services may be three-to-one. However, the benchmark institutions were also sensitive to the fact that a balance must be achieved between that which is centrally funded and that which is distributed or cost recovered. A no-cost service, by its nature, will generate infinite demand. In implicit recognition of this fact, one benchmark institution has appointed a funding committee to evaluate the appropriate mix of central versus distributed funding. Student microcomputer labs The need to achieve an appropriate balance between central and distributed funding is particularly evident in funding student microcomputer labs. While no benchmark institution entirely funds microcomputer labs centrally, one does fund and operate all labs as college and academic department facilities. Where the primary model of funding is local (college or departmental), the support level available to students varies widely. In some cases, only students with junior or senior standing have access to microcomputer lab facilities. The result is very uneven, discipline-based lab support for undergraduates. At the other extreme, the University of Michigan provides a very large number of machines centrally so that access to microcomputers by undergraduates seems to be quite uniform and adequate. Michigan also facilitates college and departmental laboratories but recognizes the importance of centrally funded facilities in helping to ensure uniform support to the majority of the student population. The other benchmark institutions also employ mixed funding models. Two report that approximately half of all machines are provided centrally, with the remainder supported by colleges and departments. The issue of student microcomputer facilities highlights the observation that entirely distributing funding may not provide consistent support to the largest number of students. On the other hand, complete central funding (a strategy not used by any of the institutions surveyed) would not enable colleges and departments to run and manage information technology facilities unique to their needs. Most large universities, therefore, use a combination of central and distributed funding and operation; the problem lies in determining the optimum mix in a constrained fiscal environment. Numerically intensive computing Different approaches to resource management are also being pursued in numerically intensive computing. Two of the six benchmark institutions centrally support extensive numerically intensive computing capabilities; a third has a national supercomputing facility on campus; a fourth provides some central facility support but allocates operational support and coordination to the College of Engineering. Penn State has pioneered the use of "cluster" computing, or sharing cycles from clustered UNIX workstations among numerous participating departments. Departments have the option of adding their workstations to a virtual resource pool where computing cycles that are not in use at a given time may be provided to another application (perhaps unrelated to the original department). This cooperative arrangement maximizes the use of computing resources and relieves participating departments of system administration tasks for which they may not have qualified support staff. The clustering concept (decentralized procurement, central management, decentralized but centrally coordinated use) is seen as a "win/win" situation for all. Life-cycle funding The rapid pace of technological change has left all institutions struggling with life-cycle funding. Software has a useful life of about eighteen months; hardware becomes obsolete in three to five years; yet university fiscal pressures are usually too tight to provide the sinking funds common in business for such purposes. The most noteworthy means of addressing life-cycle funding is the use of federal indirect income. Federal indirect charges on research grants and contracts are a major source of funds for research universities. A parallel can easily be drawn between the services provided to researchers by libraries, traditionally recipients of such funding, and by centrally funded computing facilities. However, many have viewed them as different under the terms of OMB A-21 and related federal regulations. Two benchmark institutions have nevertheless addressed these concerns and are using this category of funding to help meet some of their critical life-cycle funding requirements. Other solutions At least two of the six institutions are pursuing innovative budget methods to fund good ideas mid-year. Many have found that the long lead-time requirements inherent in university budget cycles run contrary to the realities of the information technology industry. Precisely defining information technology resource needs eighteen to thirty-six months in advance is often impossible when the industry develops new products on a nine- to fifteen-month cycle. Significant flexibility in budget planning is necessary for best-in-class information technology resources. Other institutions are including information technology needs in capital campaigns. Information technology needs are capital intensive, requiring significant initial and continuing investment. Many institutions are embarking on aggressive capital campaigns over multiple years. Clearly, it is unrealistic to fund buildings without consideration of the technology that will be needed within them, or to endow chairs without endowing the equipment needed by the top- flight researchers who occupy the chairs. STRATEGY MEASURES The element that we have labeled "strategy" addresses the "how" in implementing effective information technology programs university-wide. Institutions are struggling to formulate their strategies to maximize the benefits of their investments in technology. The struggle is particularly evident in the areas of planning and resource allocation. Planning Formal university planning processes are generally not well adapted to critical information technology issues, although approaches to coordinated information technology planning are beginning to be developed. Based on the benchmark results, it appears that many universities currently do very little formal university-wide planning for information technology. Some are beginning to do "big picture" or vision planning for the next century that includes information technology in global goals. The institutions that are including information technology in their long-range vision plans are making an implicit (if not explicit) statement that information technology is essential to their future competitiveness. However, such planning is usually not sufficiently detailed to provide tactical guidance or to deal with the specific issues of individual academic units. Formal university-wide planning for information technology initiatives, addressing the needs of all academic areas, appears to be non-existent in the major research universities in our study. What detailed planning exists is normally assigned to the central information technology organization. Frequently the planning by the central information technology organization is coordinated (in some manner) with individual academic units. However, in the experience of at least one benchmark participant, the degree of true cooperation or coordination in planning is "directly proportional to the size of the pie and the slicing thereof." In other words, if units perceive significant dollar advantages to participating in coordinated planning efforts, meaningful plans will be drafted. If requirements for planning are not tied to increased funding, human nature dictates that plans will be superficial or nonexistent. Resource allocation Some institutions have recognized that the fast pace of technological change requires significant resources just to assess trends and their implications for the future. At least two of the benchmark institutions have dedicated staff resources within their information technology organizations to examine future directions on a continuing basis. Two of the six benchmark institutions have moved even more boldly to address future changes by restructuring their central information technology organizations. Both are moving toward flatter organizations and are strongly emphasizing responsiveness to customer needs and customer satisfaction as critical measures of success. One has taken a project and consumer orientation, organizing into teams that will remain with a task from inception to full implementation. The projects are in large part decided by the customers (the using organizations) who buy into the projects with a part of their computer budget allocation. The other institution has reorganized its entire information technology organization with the assistance of an outside consultant. It is restructuring based on a set of guiding principles. At the top level, the new structure is divided into: * "service bureaus" that deal with the operational and administrative aspects of the organization; * technologists who provide the systems engineering, application, and tools/methodology functions; * consultancies that provide users with information on services and products that are offered by the organization and, in turn, coordinate new user requirements; and * an architecture group to develop/advise on enterprise standards. It is too early to gauge success or failure of these efforts. Quality improvement Most benchmark institutions are seeing formal quality improvement efforts as a strategy for analyzing and implementing processes to enhance information technology support. Three of the institutions have adopted formal quality improvement initiatives to improve service to customers. Quality improvement is, thus, a routine element of the information technology agenda at half of the institutions studied. Also in an attempt to focus on customers and quality of service, at least two benchmark institutions are changing internal priorities to focus more directly on the needs of colleges and academic departments. One is directing more administrative development staff to departmental needs; the second is facilitating a market-driven approach to information technology services and believes that this will necessarily focus priorities where they will best serve the using organizations. ============================================================ Principle 2: Best-in-class institutions encourage early implementation of information technology infrastructure and standards. ============================================================ The benchmark institutions have, in many cases, adopted a two-level approach to the development and implementation of institution-wide networking. First, even in an era of constrained budgets, they have found methods to fund critical university-wide information technology infrastructure on a priority basis, and to implement that infrastructure in a timely fashion. Second, they have adopted funding strategies that encourage widespread use of more modern networking protocol standards (e.g., TCP/IP) as the basis for campuswide connectivity. Not surprisingly, placing a high priority on early and sufficient funding for information technology infrastructure leads to faster completion and availability. Five of six benchmark institutions have essentially completed installation of basic cabling infrastructure both between and within buildings and are now able to turn attention to other critical needs, such as facilitating greater use of the technology. Priority investment strategies, such as those used to implement the basic infrastructure, are now being used to extend full network connectivity to student residence halls. Some strategies for completion of residence hall networking projects take advantage of non-traditional funding opportunities; others involve formal institutional funding initiatives such as bond issuance. ============================================================ Principle 3: Best-in-class institutions emphasize customer service in order to integrate technology into the institutional culture. ============================================================ All best-in-class institutions strongly emphasize user support. Most of the benchmark institutions are actively encouraging a customer focus, meaning to "serve the customer," not "control the customer." Users have many choices to support their information technology needs as a result of the increases in desktop power and connectivity. They are not necessarily bound to solutions developed centrally. Therefore, the relationship between information technology organizations and both using organizations and individual users must be viewed as collaborative and cooperative, not dictatorial. Computing organizations exist to serve users, not vice versa. Some of the benchmark institutions are re-orienting their internal economies or funding strategies to further emphasize the role of the customer and the importance of customer service. HELP DESKS Those benchmark institutions that have most thoroughly internalized the customer service orientation within their central information technology organizations are placing increased emphasis on help-desk functions. For a number of reasons, help desks are seen as critically important in providing satisfactory user support. The help desk often provides the first-level contact for users with the information technology organization. First impressions are important. Moreover, the more assistance can be rendered by the help desk, the less secondary assistance is needed. A technical specialist should not be needed to answer a general question; many of these can and should be answered at the first, or help-desk, level. To further emphasize the importance of the help-desk function, two of the benchmark institutions have consolidated their help-desk functions, providing the user with a single point of contact and lessening user confusion about which office to call in a given situation. A consolidated help desk provides: * a single point of initial contact for faculty and students; * an entry point for "one-stop shopping" for information technology services; and * easy referral for more difficult problems to specialty areas within the organization. Not surprisingly, those that have made strong philosophical commitments to customer service as a measure of quality have also implemented large, centralized user support organizations. These centralized support organizations are seen as the primary coordinators of customer requirements and information technology services. They can handle training, software site licensing, sales and services, departmental services, student lab management, and planning with the schools and colleges, in addition to providing basic consulting. RESIDENCE HALLS The benchmark institutions are also seeking better ways to support student technology users within the residence hall environment. In some cases, on-site support is provided by fellow students hired as technical support and living in the residence halls. In other cases, student microcomputer labs are located in residence halls, either for exclusive use of the residents or for all students. In still others, site- licensed software is being provided for residence hall use. INFORMATION TECHNOLOGY TRAINING Traditionally, universities have provided only limited support staff for information technology training efforts. However, with the widespread use of information technology tools and the growth of the Internet, demand for information technology training has grown exponentially. Many of the benchmark institutions are developing creative means of coping with the increased demands for information technology training. Examples include: * One benchmark institution has consolidated information technology training into a single administrative unit. * For Internet access training, Texas has developed a model in which the operation of technical tools is taught by the information technology organization, but library personnel teach how to use the tools to access different subject areas. This is seen as an extension of the traditional library role of facilitating user access to information. * Some institutions are using commercial software packages to address some of their needs, rather than developing in-house training packages. Another approach to meeting the increasing demand for information technology training is to enlist the aid of students in instructing other students. Advantages of this approach are multiple. Most courses "for students, by students" are free; the content is driven by peer needs; and peer instruction provides a more collaborative learning environment. The training issue has been extended to address critical emerging needs, such as network security instruction. At Michigan, volunteers have been trained as ethics discussion leaders and Student Open Forums are held to discuss the ethical implications of actual incidents. In short, although institutions are still grappling with ways to stretch the traditionally small staff allocations dedicated to training, most are actively implementing enhanced information technology training programs. Such efforts increasingly involve extending the talent pool beyond just the resources of the central information technology organizations in order to cope with ever increasing demand. ============================================================ Principle 4: Best-in-class institutions use the elements of standards, security, and architectural planning to create a supportive environment for change. ============================================================ STANDARDS All benchmark institutions are attempting to articulate standards that will facilitate growth. However, there is divided opinion (sometimes even within an institution) on the value of open standards. Some are embracing a strategy of evolving to open systems standards (e.g., the Open Software Foundation's DCE) while some are choosing not to do so. Those in the latter category (a significant minority) believe that better support can be provided for growth by remaining with proprietary standards, at least for the near-term. Among the institutions committed to open, standards-based approaches, there was widespread sentiment that proprietary solutions are ultimately more expensive because the vendor(s) can hold the prices high. Moreover, applications will not be portable cross-platform, and there will be significant interoperability and compatibility problems. SECURITY Institutions are also examining strategies to provide secure technical environments that support change and growth. Many are seeking a reliable means to provide more secure authentication for the highly heterogeneous environments that typify large research universities. The strategy for many involves Kerberos authentication for those applications that can take advantage of it, centralized Kerberos server availability, and possible integration of token card support when that is technically feasible. Some go one step further and envision implementation of DCE in the future. Concerns have been voiced about cross-institutional compatibility and technical complexity of solutions. Most institutions seem to recognize that network security is an extremely complex problem to which the "answers" are evolving. The schools most actively engaged in examining network security problems and current security technology are beginning to explore options for encrypting at least some data. The concern driving examination of encryption options is not only with the confidentiality of the information but also with its integrity. At present there is no firm consensus on how to accomplish this objective technically, nor even on the degree of technical difficulty. Several institutions seem to be hoping that when Kerberos is more widely available in their environments session key encryption can be employed. Some are investigating encrypted Internet utilities (e.g., encrypted Telnet), and some are investigating public key encryption, particularly for e-mail (both signatures and data). Two of the institutions were less concerned with "network" security per se. Their strategy for providing a secure means for network growth was to vest all responsibility for security in the hosts interfacing with the network. Because a network is only as strong as its weakest element from a security perspective, this approach would necessitate, at some point, a minimum security level for all attached hosts and strong physical security for critical network components. Thus, again, there is not a universally accepted "right" way applicable in all institutional contexts. The majority of benchmark institutions are examining options for enhanced network security, but the degree to which security is an issue and the specific technical means being evaluated vary according to institution. ARCHITECTURAL PLANNING Architectural planning is the third area in which institutions are implementing or examining technical environments to facilitate change. Architecting a future institutional distributed computing path and transition strategy is on the agenda of three of six benchmark institutions and under discussion at the others. Moreover, there are at least incipient moves to examine cross- institutional architectural issues. For example, several institutions are looking at the same generic types of solutions for the same problems. Areas of shared interest include more secure network authentication (Kerberos, token cards), open client/server solutions based on DCE, Novell implementations of Kerberos, and eventual transition to DCE. For those institutions whose architectures are evolving in similar directions, there is interest in examining key common issues that could be solved collaboratively. While the cross- institutional differences in evolutionary strategy may preclude a truly common solution across the board, for those areas where interests and projected architectures do coincide, it may be beneficial and cost-effective to establish collaborative relationships. Establishing collaborative or cooperative partnerships may provide vendor leverage. At the very minimum, information in these areas of shared interest needs to be exchanged at the technical level to prevent possible conflicts in implementation that might affect all concerned institutions. An architectural issue for several institutions is that of "data warehouses" for administrative data. The single greatest challenge facing administrative computing is providing users with access to their own data. With the reasonable prices of mid-tier servers, it is possible to create "data warehouses" dedicated to providing better access to data through user-structured queries/reports based on SQL. While certainly many commercial database management systems could be applied to this type of problem, Oracle seems to have taken the most aggressive position with regard to the implementation of open software standards that run in a heterogeneous environment. There are acknowledged security issues that must be resolved before sensitive data elements are introduced. Nonetheless, the data warehouse concept does appear to be emerging as a common architectural element for several of the benchmark institutions, with Oracle most frequently mentioned as the database management software of choice. Another common architectural feature was the inclusion of MandarinTM as an element in implementing client/server solutions at several benchmark institutions. (Mandarin provides a "tool kit" for building client/server applications.) There are currently seventeen universities in the Project Mandarin Consortium and that number is expected to grow to nearly sixty by the end of the current fiscal year.[1] Three of the benchmark schools are members of the Consortium. Those applications currently in place or under development at the benchmark institutions using Mandarin are geared towards providing students with direct access to their own data. Mandarin is a proprietary product today, but it is evolving to compliance with OSF's DCE standard. It was designed with security provisions to help guard against unauthorized access. Mandarin represents another opportunity for those institutions with common architectural interests or strategies to collaborate for their mutual benefit. Library computing One critical architectural area in which all institutions appear to be struggling is in defining the path for library computing to transition to client/server technology. At present there is no clear-cut consensus on whether building or buying client/server solutions is the best strategy. Those that would opt for a "buy" strategy are troubled by the fact that no one single library automation vendor has all the needed components to support a major research institution. Further clouding the architectural vision for library systems is the fact that current library systems are mainframe-based with little hope they can be easily converted to client/server systems without extensive reengineering. For all of these reasons, the library system of the future will probably emerge as a combination of off-the-shelf components and locally developed subsystems, with standards providing the "glue" to make the pieces work together. Z39.50 is a particularly vital standard for future library systems. An interesting observation in the area of library system architectures was that few of the benchmark institutions rely completely on NOTIS. Even with a fairly comprehensive library system such as NOTIS, a local library development staff is needed to adequately support the needs of a large research university. Moreover, NOTIS and the other automation vendors are not providing the next generation of library information support, such as integration of CD-ROMs, imaging, and client/server tools. The University of Wisconsin has raised the lowest common denominator for the library computing environment. High-end personal computers are used for all public and staff access to library information facilities and tools. This architectural model provides the basis to support additional information services such as full text and image database access, as well as sound and other multimedia services. Overall, the most valuable result of this benchmarking effort was establishing relationships among the institutions and formally discussing common problems and solutions. Penn State and the five benchmark institutions have agreed that there are areas where collaboration could help address some of the common problems we are facing. One of the most pressing issues is how to support the incredible growth--in size and demand--in our user communities. The University of Wisconsin has agreed to take leadership in putting together a Union Catalog of User Support Materials and Tools. Our intent is to reduce duplication of effort by sharing user-support materials from each of our institutions. We hope that if this effort is successful, we will be able to apply it to other areas of overlapping concern. As noted in the introduction, our comparative numbers proved to be quite useless. The primary reason for this is that we were asking the wrong type of question; we were asking about "input measures" when in fact what we wanted to gather were "quality indicators." All of the institutions in our benchmarking study are interested in gauging the quality of their information technology resources and services. Thus, we have agreed to try to develop a set of quality indicators, questions that will provide a measure of the contribution of information technology to the fulfillment of the institution's mission. The University of Michigan has agreed to be the coordinating point for gathering and consolidating our suggestions for quality indicators. Was the benchmarking effort worthwhile? Yes. For Penn State, the effort has paid off by identifying a number of areas in which it can examine its own "soul" and possibly incorporate the best elements of the practices observed into its own institutional environment. As a group, we learned, first, that identifying useful quantitative measures of quality is extremely difficult. Second, we learned that there is immense value in recognizing the cultures of each institution and the ways in which each is accommodating change. Finally, both the benchmarker and the benchmarkees have benefited through sharing findings. For those considering the benchmarking process, the most valuable results may not come from gathering numerical data. Clearly, in this effort, the real value of benchmarking has been the synthesis of the experiences of the institutions, and finding common processes that may benefit all. ============================================================= Footnote: [1] For information about Mandarin, contact Nancy VanOrman, Project Manager of the Consortium, phone 607-255-2618; fax 607-255-1297; e-mail nlv1@cornell.edu ************************************************************* Author Acknowledgements: The following individuals contributed to this article: J. Gary Augustson, Executive Director, Computer and Information Systems; Ken Blythe, Director, Office of Administrative Systems; Eric Ferrin, Director, Library Computing Services; Kathy Kimball, University Computer, Network, and Information Security Officer; Steve Updegrove, Director, Office of Telecommunications; and Russ Vaught, Director, Center for Academic Computing. Thanks also to Gay Gragson for her editorial assistance. ************************************************************* Observations on Benchmarking InformationTechnology Support 2-(ô×fNºB -)ô×g°<{N­ HxWord Work File D 5lTEXTMSWDTEXTMSWD«-¬Br~\¾mjgâSt(´eÀÒ2;Nû„–¢²ÂÈî´2JZrÀ$Nbbbbbbbbbb´bbbrx´´~´´´´´’¢~{¾gzt}´g Jeff Hansen2¦Š¬!l2STR ¿ãÿÿª Ì