Building an Effective Organization to Run a Campus Backbone Network |-------------------------------------| | Paper presented at CAUSE92 | | December 1-4, 1992, Dallas, Texas | |-------------------------------------| Building an Effective Organization to Run a Campus Backbone Network C. D. Jordan The Ohio State University Columbus Ohio ABSTRACT The Ohio State University is the largest campus in the United States; consequently, its networking staff have experienced most problems that affect networking and communications groups on a campus. This paper will address the staff requirements of an effective and dynamic networking and communications organization. It will present a structure that can maintain a production-level network and still prepare for the future in a constantly evolving technology. The paper will emphasize the need for human funnels of information into and out of campus departments through coordination of departmenat LAN and network administrators. It will define the structure of such an organization as well as its cooperative relationship with supporting groups. ------------------------------------------------------------------------ BACKGROUND The Ohio State University (OSU) has 54,000 students on its main campus in Columbus. OSU was one of the first universities to develop a modern campus network to meet the needs of computer data communications. The campus network is called SONNET, for System of Neighboring Networks- -a name that reflects the interconnecting nature of the backbone network with building and departmental networks. The backbone network is a fiber optic, 80-megabit per second, Proteon token-ring network. This backbone is connected to over 110 main campus buildings, four regional campuses, OARNET (Ohio Academic Research Network) and CICNET (Committee for Institutional Cooperation Network--begun by the Big-Ten universities). More than 10,000 computers are connected to SONNET via building networks in the 110 SONNET-connected buildings. Even as the campus begins to plan for the next phase of networking, most of the original design and installation of the network is still working satisfactorily. The next phase of the campus network will have to meet the bandwidth demands of the growing number of users and the requirements created by more sophisticated network applications. Over the years, installing and maintaining SONNET has passed through the recursive Campus Networking Process: THE PROCESS 1. PLAN - Determine goals. 2. DESIGN - Develop a technical design that will meet the goals. 3. INSTALL BACKBONE HARDWARE - Install cables and network hardware. 4. INSTALL NETWORK SOFTWARE - Install central network services software. 5. INSTALL USER HARDWARE - Install user PC, MAC or workstation hardware. 6. INSTALL USER SOFTWARE - Install user networking software. 7. TEACH USERS - Teach users how to get started using the network. 8. MONITOR - Monitor the network, collect and analyze network traffic statistics, answer problem calls and initiate trouble-tickets. 9. REPAIR - Troubleshoot, repair and maintain network hardware and software. 10. DISSEMINATE INFORMATION - Disseminate information about new network services. 11. CONSULT - Provide "network problem" consulting. 12. STAY INFORMED - Stay abreast of the evolving technology. 13. DEVELOPMENT - Expand services or begin the next phase, (repeat 1-12) During the Process of installing and maintaining OSU's SONNET network, five key network function groups have evolved: KEY FUNCTION GROUPS - Network Engineering & Technician Support - Network Analysts & Application Programmers - Network Operations Center (NOC) - Network Information Center (NIC) & Consulting - Departmental Network Administration Liaisons Which department or departments perform these functions is not the critical issue. However, clear assignment of these functions and easy identification of who performs these functions has led to a smooth running organization and a stable network at OSU. This paper will define the roles of each of these groups in the networking Process, their interconnecting relationships, from which potential departments the staff can be drawn and how we have divided the roles at OSU. During the early phases of the Process these five key function groups can be formed. But before the Process begins, there may be many disparate, but very interested network players on a campus. THE PLAYERS In order to be successful all the "critical-to-success" players need to be involved in the planning. This list of people will come from a different list of departments for each campus. To avoid domain clashing or finger pointing, get all the critical-to-success players unique to your campus at the planning table. The following is a list of groups and departments that have been involved in the early stages of networking on some campuses: Computer users Academic Computing Center Administrative Computing Center Computer Science Department Math Department Engineering College Supercomputer Center Computer Repair Department University Telecommunications Department Library Physical Facilities - Electrical Shop ANYONE WHO MIGHT FUND IT START WITH A PLAN To get things started the critical-to-success players need to decide and agree on four issues: 1. What services will be provided initially on the campus network? electronic mail? remote login (telnet) and file transfer (ftp)? dynamic address resolution (Domain Name Server)? access to the Internet? newsgroups (USENET)? on-line Library card catalog? access to administration data? electronic white pages (Whois, ph)? campus-wide information system (CWIS)? security authentication (Kerberos)? protocols supported on the backbone? etc.? 2. To whom will the campus network provide connectivity/access? connect each building? connect each college or department? connect each faculty office? connect each faculty, graduate assistant and staff member? connect public computer labs? provide universal connectivity - provide access for every faculty, student and staff member? connect to the Internet? access from home? 3. Which leaders or departments will do the first four Key Functions? Network Engineering & Technical Support Network Analysts & Network Application Programming Network Operations Center Network Information Center & Consulting 4. And unless you are very lucky, who will pay for it? But before a group of "players" can decide who will do the Key Functions, in question #3, they need to have a better understanding of these network function groups: NETWORK ENGINEERING AND TECHNICAL SUPPORT Network engineers can be involved in the networking Process no later than step #2, physical design of the campus network. Technicians' involvement may not begin until step #3, the installation of backbone cable and hardware. It becomes apparent that there are two distinct skills in this function group; however, to be successful these two subgroups of people will have to work closely and comfortably together. Together, they will design, install, configure, troubleshoot, repair and maintain the network cable plant and hardware. The cable plant may involve several different types of cable (fiber, coax, twisted pair, other), and if so, this will require greater expertise of the staff. The hardware will include bridges, routers, hubs, protocol converters, terminal servers, and switches. This group will also evaluate network traffic statistics, determine and resolve routing issues, and may assign network addresses through their maintenance of the Domain Name Service tables. In addition, they will have to communicate with the Network Analysts--hardware and software people must always be in communication. They will report outages to the NOC (Network Operation Center), and follow-up on trouble-tickets initiated by the NOC. Routing or network addressing changes which affect user software will have to be communicated to the NIC (Network Information Center) & Consultants. Individuals may be requested to give presentations at DNA (Departmental Network Administrator) liaison meetings. Outside vendors, the university Telecommunications Department, the Physical Facilities Department, the Computer Center and others may have the skills, or can be trained satisfactorily to do some or all of the above. Since two distinct groups of personnel are described above (Engineers and Technicians), this functional group can also be very neatly divided between two departments. Or the functions can be divided among three groups, where one of the groups only installs cables. On a small campus, one energetic person and a few students may handle it all. With so many universities facing budget cuts, if more than one department on campus installs cables, consolidation of this job may be an area to review. Many Computer Centers, including the ones I have been part of, were leery of Telecommunications Departments installing coax connectors and Electrical Departments installing fiber. These fears were reasonable; originally. But numerous campuses have found that with proper training in data cabling installation procedures, these departments can do excellent work. As the line of "what is telecommunications" and "what is data communications" is merging, the function of Telecommunications Departments and Computer Centers have sometimes clashed. These are the clashes you want to avoid by coming to an agreement on who will do what. At OSU, the Academic Computing Services center has gradually decided that we don't really want to be "cable pullers". OSU's UNITS, the university's telecommunications department originally installed and maintained the campus fiber plant. Now that the majority of building networks are using twisted pair cable, a cable that telecommunications people have long experience with, it seems only natural to work with the telecommunications department for most cable installations. So the Academic Computing Services' Networking & Communications Division engineers the backbone and the senior engineer, (who maintains a close and successful relationship with the telecommunications department), orders cable installing service from the telecommunications department. Technicians of the ACS Networking & Communication Division install and maintain the backbone hardware. NETWORK ANALYSTS AND APPLICATION PROGRAMMERS Network Analysts, Network Application Programmers and Network Systems Support staff will develop, install, test and maintain the network software services that are identified in the Planning phase of the Process. These services may be maintained in a distributed fashion or on one machine. Sometimes this group of people may be the systems analysts of the central computers for the Computer Center. Sometimes a particular department such as the Computer Science, Math Department or Engineering College will maintain some of the services. The Network Analysts will have to work with the Network Engineers. The Analysts will have to keep the NIC and Consultants informed of how to access new services. They will need to report outages to the NOC, and they will have to investigate trouble-tickets reported to them from the NOC. On occasion Network Analysts or Network Application Programmers may be called on to give presentations at the Departmental Network Administrator liaison meetings. Two types of skills are needed for this area: 1) daily maintaining of application software and 2) testing, installing, developing, customizing and recommending new services. Usually these two types of skills don't come in the same person, but both are critical for a stable, but dynamic organization. Sometimes the staff in the second category are your brightest people, but they may get you in the worst messes--often they are not interested in properly documenting their work and they may get bored with a project once the initial challenge is over for them. This is where the staff in the first category will save your job. At OSU, these functions are distributed over several departments. Central Domain Name tables and Whois databases are provided by the Network Engineers. Some departments provide their own Domain Name Service, Mail and Newsgroups. Also many services, such as Kerberos authentication, Mail and Newsgroups are provided by the Systems Analysts of Academic Computing Services' Central Computing Division. OSU's University Systems, the university's administrative computing department maintains the Library catalog software. As information services is a developing part of networking, ACS's Technology Assessment staff provide some of the newer services, such as our Campus-Wide Information System and a Gopher service. NETWORK OPERATIONS CENTER, (NOC) The NOC (pronounced knock) is known to the outside world or the user community as a phone number to call to report a network outage. To the other network function groups this area is known as the group that monitors the network. The NOC has three main functions: 1) Electronically monitor the network 2) Initiate, dispatch and verify closure of Trouble-Tickets 3) Answer the NOC problem reporting phone line Two management decisions will have to be made. What kind of software monitoring tools will be provided to the NOC Operators, and what hours of operation will be offered by the NOC. These hours will have to be well publicized. There are about three levels of service to consider: 1) Monday through Friday, 8 am to 5 pm 2) Monday through Saturday, 6 am to 12 midnight 3) 24 hours a day, 365 days a year At first, level #1 works sufficiently. But as the number of users grows, choice #2 becomes a necessity. Starting at 6 am so that a problem in the night is resolved before 8 am is better than waiting until 8 am when 100 users will call the NOC to report the same problem. Also, many students and researchers tend to prefer the evenings to work, both from public computer labs and home modems. When people come to depend on network services just as they do their telephone service, then providing the third level of service will be necessary. The NOC Operators communicate with all of the other function groups: Network Engineers/Technicians, Network Analysts, Network Information Center & Consultants, Departmental Network Administrator liaisons and network users. Who can do this job? These people must have good phone skills, good hearing, dispatch and diplomatic follow-up abilities, and be able to adequately describe the nature of a reported problem. For the latter reason most NOC operators also do a "first line" of remote troubleshooting of problems and/or attempt to duplicate the user complaint. Also, to avoid hundreds of calls reporting a known outage, scheduled and unscheduled network outages need to be properly posted to user newsgroups and DNA liaisons notified. The Computer Center that runs a central computing mainframe may have the night staff and the Computer Center's staff may have the best software skills for the "first line" troubleshooting. But who has the best telephone skills? The Telecommunications Department might. If Network Engineering is handled by the Telecommunications Department and they have 24 hour operators this may be a consideration. However campus operators do not normally have the time to do first line software troubleshooting. At OSU the integrity of the campus network is now so critical to researchers that we feel it is necessary for our campus to have a separate, specialized group that runs a 24 hours a day, 365 days a year NOC service. NETWORK INFORMATION CENTER, (NIC) The NIC (pronounced nick) informs users of new information available on the network, through publications, workshops and by organizing and running meetings for the Departmental Network Administrator (DNA) liaisons. Consulting also needs to be provided for user problems with communications software and network applications. Often this group may be formed out of User Services or Microcomputing Consulting Groups. But increasingly some of these functions are provided by Librarians. These individuals will need to have great people communication skills. In addition, they will need a variety of platform skills in order to support: 1) DOS users, 2) MAC users and 3) workstation--UNIX or possibly VMS users. The NIC staff will need to get information from the Network Engineers and the Network Analysts to pass on to the DNAs. In turn, a feedback loop of information about user understandings and capabilities will pass from the DNAs to the NIC, and from the NIC to the Engineers and Analysts. At OSU, we have a separate NIC group in the Academic Computing Services' Networking and Communications Division. However problem consulting often occurs in ACS's Microcomputer Consulting group and the Distributed Services Division of ACS handles most problem consulting for workstations. As was said earlier these key network functions do not need to be placed all in one group or department, however they do need to be assigned and handled for a successful network operation. DEPARTMENTAL NETWORK ADMINISTRATOR (DNA) LIAISONS No matter how large or small a campus is a distributed group of DNAs is necessary. A Computer Center staff of 100 people can not reach 60,000 users and similarly, a Computer Center staff of 15 people can not reach 4000 users. As departments are connected to the campus backbone network, a liaison for each department or affinity group needs to be established. The choices for this person are often: 1) Department LAN (Local Area Network) Administrator 2) Faculty member 3) Student employee 4) Graduate Assistant 5) Secretary or Office Support Staff The LAN Administrator, is the ideal choice, but many departments may not have such a position, yet. Once the novelty has worn off, the Faculty member, doesn't really have the time to do the job right. The Student employee on whom everyone has come to depend, will leave the job at the end of a few semesters. The Graduate Assistant will write great customized code that will not get documented and when this person leaves, the department will have more problems than with the Student employee. Many people will think the Secretary is a terrible choice, but that isn't necessarily correct. A well-chosen secretary, who enjoys using computers and does use them daily, if given proper network training, can do this job. Actually where secretaries often "top out" in a department, this can offer a new avenue for growth. Also, secretaries don't have the problems mentioned for Faculty, Students and Graduate Assistants. At OSU, we have all of the above attending our Network Working Group (NETWOG) meetings and filling the role of DNA liaison. The DNA needs to be a liaison position to the NIC, and a point of contact for reporting network failures to the NOC. The list of DNA phone numbers, e-mail addresses, associated department names and their office locations needs to be widely distributed, updated once or twice a year, and made easily available to the NOC, NIC, Engineers, Analysts and to each other. Therefore the DNA liaisons are critical people for distributing information to users. They also must be able to get new users "up and running". Therefore, they need to know how to install user network hardware and communication software. They will assign network addresses to the computers in their department and then report Domain Name Service (DNS) information to the staff that maintains the DNS tables. Sometimes the hardware installation is done by another department on campus such as Computer Repair Centers or the group which installs the user's network cable. The DNAs need to know how to contact this group. But to avoid unnecessary service calls, they will want to know a first line of troubleshooting and testing for network failures from their network. USERS Need to know at least these three things: 1) Who their Departmental Network Administrator is. 2) What the phone number of the NOC is. 3) How to contact the NIC--phone number, e-mail or office hours. STAY INFORMED All staff in the network function groups need to stay informed of the evolving technology of networking. This can be done several ways: 1) Attend regional or state network organization meetings. 2) Read publications such as Network World, LAN Magazine Data Communications, ConnecXions and others. 3) Attend society conferences and computer network shows: CAUSE, EDUCOM, INTEROP, IETF, SIGUCCS, COMDEX or others. 4) Read newsgroups; assign the staff to read & reply to certain newsgroups, put this in their job descriptions. THE CAMPUS NETWORKING PROCESS REVIEWED Now that the roles of the function groups have been identified, listing where each group may fit in the networking Process becomes more apparent: 1. PLAN - Critical-to-success players 2. DESIGN - Network Engineers, Network Analysts & others 3. INSTALL BACKBONE HARDWARE - Network Engineers/Technicians 4. INSTALL NETWORK SOFTWARE - Network Analysts/ Programmers 5. INSTALL USER HARDWARE - Dept. Network Administrators, others 6. INSTALL USER SOFTWARE - Department Network Administrators 7. TEACH USERS - Department Network Administrators 8. MONITOR - NOC, Network Engineers 9. REPAIR - Network Engineers/Technicians & Network Analysts 10. DISSEMINATE INFORMATION - NIC and Consultants 11. CONSULT - NIC and Consultants 12. STAY INFORMED - Everyone 13. DEVELOPMENT - Everyone Today a campus network is a growing and evolving service. When staff stay informed the international sharing of new ideas leads to testing a new idea on your campus. Successful development and customizing of an idea leads back to the planning stage for new and expanded services of your campus network. The planning of new services will lead to building a design that will meet the goals of the plan. Thus the process begins over again which makes this the recursive Campus Network Process.