How to Build a Multimillion Dollar Network with No Money Down

Abstract

The University of Toledo wanted to replace its old, unreliable, patchworked broadband data communication network with a state-of-the-art network employing current technologies. A large and very significant impediment to fulfilling this "wish" was the lack of funding for this large-scale project. The Information Technology Department developed funding sources and convinced executive management, student and faculty groups, and the Board of Trustees of the viability of these funding sources to obtain approval and support to proceed. These sources are available to most colleges and universities and are not, in themselves, new, although the method of selling them may be. This presentation describes the state of the old network, identifies sources of funding, and outlines the "selling" process. The presentation will conclude with a brief description of the new network.

(Connie Schaffer and Louise Easly)

As of 1993, and even earlier than that, the broadband network at The University of Toledo consisted of three cables for data, video and energy management. The most useable of these was the video cable, with the quality of the energy management cable not far behind. The rationale for quality for these two cables may have stemmed from the need to deliver video for instructional purposes around the campus and for the efficiencies and cost-effectiveness of the cable for energy management. Regardless of how those two cables came to be "the chosen," the rationale for a complete data cable was never developed and implemented. As a result, the data network evolved in bits and pieces as answers to each next, very immediate problem -- a real Band-Aid and chewing gum approach. This was the state of the network as recently as four years ago.

As of today, we're in the home stretch of installing a $6 million fiber optic network on our campuses. This is going to be the story of How to Build a Multimillion Dollar Network with No Money Down. Actually, this is probably the best time and place for telling it. Sometimes, it has seemed to us like an early Christmas, as we made up our network "wish list." Other times, faced with seemingly insurmountable problems of time, money and everything else, it seemed it would take more than a bit of "Disney magic" to pull it off.

In order to tell our story adequately, we're going to lay out the whole process, from start to almost-finish. Some of it doesn't put UT in the very best light, but if we don't tell you the "bad news," you might not believe the "good news."

So, to continue... To add insult to injury, the electronics in the manholes were frequently under water. No matter how much we searched for the same electronics used on submarines, no matter how much we wrapped the equipment in plastic garbage bags, water still seeped in and shorted out the network. (Can you tell we still hadn't come up with adequate funding at this point?) Not only did the service disruption cause the user community untold aggravation, but we even started fighting among ourselves during the repair process. One notable incident took place at 2:00 a.m. one cold February night about two years ago.

The network failed according to Murphy's Law -- at the worst possible time. Our telecommunications staff found the offending equipment in a manhole near the Student Union. A call placed to the Physical Plant staff resulted in a maintenance worker being dispatched with equipment to pump out the water in the manhole. When he arrived, he said he had brought the wrong equipment and had to go back across campus to the maintenance area for the right machinery. The Telecom staff naturally questioned the wisdom of the maintenance worker in not being properly prepared to perform his duties. Words were exchanged, none of which can be repeated here.

According to our Telecom staff, the maintenance worker "took his sweet time" in returning with the right stuff. To make matters still worse, as he was positioning his equipment, he backed his truck up and ran over a tool box belonging to one of the Telecom staff. More angry words were exchanged, resulting in conflict management at higher levels the next day.

The cable system required frequent tuning because of "bleeding" from the video and energy management cables into the data cable. The situation was so unbalanced that, at one time, the Telecom staff shut off all data transmission equipment on a portion of the network and still found a signal coming through. The system was also temperature-sensitive. Throughout any year in northwest Ohio, the temperature can range from the 90's (for about two days in what we call summer) to near or below zero (in our too-long winter). After balancing the system in January or February, it was necessary to balance it again during the summer months. Each of these operations required the network to be made unavailable for several hours.

Not only did the data system fail at least once a week, but it was a Heinz mixture of 10Mbps Ethernet and 1.5Mbps T-1 lines. The task of extending the network to new areas was formidable, requiring great ingenuity. However, no matter how clever we were in solving connection issues, the problem of "who paid" popped up over and over again. Most times, it was easy to identify the "owner" of an office and to obtain approval and funding before proceeding with the network install. But if funding were not available from the user, the decision to do nothing was not always the best one. Some users resorted to buying their own inexpensive modem and using the University telephone system to dial out in order to connect back into the modem pool, a procedure which sounds like Rube Goldberg could have concocted it. This was a very expensive connection, resulting in increased telephone costs.

Another solution was for departments, particularly the technical ones, to install their own network, resulting in a hodge-podge of thin-net, token ring, or whatever was available and cheap at the time.

A particularly ugly incident (financially speaking) arose when a decision was made (without consulting Information Technology) to omit the video cable from a newly constructed building. The decision saved $10,000, but ultimately cost $100,000 to correct. The cable that had been installed was used as "pull" cable. The new cables for voice, data and video were connected to the end of the old voice-data cable and pulled through the building. It was apparent that, while we never appeared to have the money to do the job right the first time, we always managed to find the money to correct the problem.

At a meeting of the University's Academic Computing Committee, the chair of the committee (a professor in the College of Education) argued that it would be better in future construction and renovation projects to omit every other bathroom and drinking fountain than to omit any telecommunications under the banner of cost savings.

Budgets have been as tight at UT as they generally are at many institutions of higher education. State subsidies have been cut, largely due to declining enrollment. The enrollment was dropping due to a smaller pool of available high school students heading to college and also due to the fact that the strong economy meant they frequently chose to enter the job market rather than to start college. The falling FTE figure itself meant less income from tuition and fees. A few things were blatantly obvious. We desperately needed a new, reliable, high-speed data network, and we didn't have the cash to spend for it.

To make matters still worse in this chaotic situation, the functional groups which now make up the Information Technology department were spread all over the University's organization chart. Telephones were part of Administrative Affairs. Video was part of the library, in the division of Academic Affairs. And the data organization was part of Graduate Studies, Research and Economic Development. In the summer of 1993, for better or worse, all three units were combined into the Information Technology Department under the benevolent dictatorship of a Chief Information Officer.

Now, it's been said that "CIO" doesn't just stand for "Chief Information Officer" -- it also stands for "Career Is Over." Our CIO, Dr. Jerry Nogy, came to UT on his first day, all smiles and full of great plans for making the University a technologically-adept institution, one we could take great pride in. On the day he and his assistant moved into their offices in University Hall at the center of campus, Jerry discovered that all of our file servers were centrally located. In order for him to send files from his PC in one part of the office suite to the printer in the anteroom, a distance of only a few feet, the file made a trip from U-Hall to the server located in the Computer Center at the southeast corner, and then back again to the office suite in University Hall. If there were a "glitch" anywhere in the transmission, he simply couldn't print. Professors and administrative staff alike all seemed to have put his telephone extension on speed-dial for their complaints about the network and computing in general. It looked like his hopes for a brilliant career at UT were over.

Those of you who know Jerry or others like him will understand what we mean when we say that he began developing a "scam" -- we could, in fact, make plans for a great network in the not-distant future without any major outlay of funds if we could partner with a solid vendor, if we could, in effect, finance it ourselves through a reliable revenue stream, and if we could get all of the campus factions -- faculty, administration, students -- to buy into the plan and back us with their support.

(Ron Piotrowski)

So, where did we go from there? The first thing the new CIO did was to reorganize. As we indicated, Telecom reported to the VP of Administrative Affairs, video support was in the Division of Academic Affairs, and the rest of Computer Services reported to the Vice President of Graduate Studies, Research and Economic Development. (By the way, did you notice that we didn't even make it into the lengthy name of our own division?) Jerry brought all of these entities together in the Graduate Studies division, and we called the new organization Information Technology.

However, even with all the consolidating and streamlining, there still was not a lot of discretionary funding available for the network project. We needed to generate our own funding sources. The first thing we did to get into the resale of telephone service to students in our residence halls. First, we expanded the telephone switch, and then we purchased a call accounting system and a voice mail system. Finally, we secured a room for the new telephone switch.

In order to pay for the telephone switch and all its peripheral equipment, we decided to borrow funding based on future earnings. It was more than a bit risky, but we felt we had no choice. We identified six possible revenue sources and nine corresponding expenses to generate the revenue stream we would need. The revenue sources included: (1) providing dial tone to residence hall students, replacing Centrex; (2) resale of long distance service; (3) cable TV service; (4) voice mail; (5) providing data connectivity in the residence hall rooms; and (6) rental of personal computers. (Some of these have had less success than others, and either were never implemented or were dropped quickly.)

The extra expenses required to put these six "cash cows" into effect included an additional telephone switch and additional telephone trunks, the cost of long distance service, wiring the residence halls for data and for cable TV, hiring additional personnel, and purchasing voice mail equipment, personal computers for rental, and a call accounting system for student telephone billing. Projections indicated the potential for a positive cash flow in the near future.

With approximately 3,000 students who lived in the residence halls on campus, many of whom are international students or from outside Ohio, it was clear that the long distance resale program was the one to be targeted. We developed models of earnings based on average student long distance usage, and the number of students who would participate in our long distance program. I checked with my colleagues at other universities as to their participation rate in the student resale program and we determined (or guessed) that we could achieve about a 75 to 80 percent participation rate within the first two years. Based on those calculations and assumptions, we determined we could generate about $550,000 a year in net revenue.

After the first six months, we had around a 40 percent participation rate and we lost BIG money. As our beloved CIO would say, "That's not chump change!" But the administration had faith in us and let us carry a negative balance forward to the next fiscal year. In year two, participation increased and we only lost $15,000 or $20,000. Today, we are averaging around 75 percent participation and the resale program generates between $500,000 and $550,000 a year in net revenue.

We now could at least show the start of a trend of earnings. But it still wasn't enough to build a network. Even if we leveraged those earnings over ten years, we would still need more money.

Jerry Nogy wasn't done spending money yet. He decided that the residence halls all needed to be rewired for voice, data and video. He made his case to the University's Board of Trustees and borrowed another $800,000 to wire the dorms, all of this to be funded by future earnings. We finished wiring the last residence halls this summer.

The third initiative he took forward was to establish a student technology fund. The primary purpose of this increase in tuition was to generate revenue to be put back into student computing. Virtually everyone told him he'd never get it passed. Not only did he have to consider a tuition cap imposed by the State of Ohio, he heard repeatedly that administration wouldn't support it, the students would balk, and everyone else would oppose it as well. At a number of committee meetings, when this initiative was discussed, the plan didn't get much support. Jerry heard over and over again comments like, "Good luck," "Go for it if you have the time to waste," and "It'll never happen."

In at least one meeting with UT's President, Frank Horton, Jerry talked about the merits and benefits of such a fund. The President listened -- he's a very good listener and he doesn't forget much. But at the time, he neither supported nor rejected the idea. However, not more than a month later, on the way back from meetings in Columbus, President Horton called the VP of Administrative Affairs, Tom Repp, and had him run the number projections on a 1.5% increase for a technology fund. Within a month or two after that, the concept of the fund was being supported by a vote of the Board of Trustees, and today (with a 3% Student Technology Fund -- the Board even approved an increase since then), we generate $2.3 million annually.

Needless to say, all the original nay-sayers now wanted a piece of the pie for their own purposes. They got nothing. Money from the fund goes totally towards student computing needs. We've upgraded computer labs, purchased discipline-specific software, funded lab assistant positions, and much more. What does this have to do with the new network? We made a case that the students would use the new high-speed network, and that our professors would use the network to enhance classroom teaching.

We were able to convince the administration that part of the fund (known as TEAF, the Technology Enhancement Allocation Fund) should be used to support the high-speed network project. We now receive $300,000 a year for the next ten years.

While we're on the matter of finances, let me just say that part of the "sell" to the Board of Trustees was to present simple ratios to the Board. For the fact that we've laughingly and publicly referred to this project as Jerry's "scam," even in front of our own administration, the CIO laid out "the cost of the money" to our executive management and to the Board. He made a convincing argument that equipment expenditures would pay for themselves and, if financed, would not require additional University funding. The figures he presented were the payback ratio (in our case, our story showed a short recovery period to recoup the cost of the investment), the return on investment ratio, which showed the value of the investment in the capital expenditure (our figures showed the outlay to be an excellent investment opportunity), and the revenue to cost ratio (again, our figures showed a positive relationship). As you can imagine, these ratios played well with the business leaders on our Board of Trustees.

We were not shy about sharing our plans with student leadership, either. If we had any hope for success, it would hinge on the students buying into the program, literally and figuratively. Jerry and other IT management staff regularly attended and participated in Student Government meetings, as well as meetings of the Residence Hall Association. We told the students how much money was being generated, and how and why we were going to use the money in the fund for different initiatives for student computing. We got their support early in the process. We continued to meet with the student leadership, updating our progress and talking to them about new initiatives. On any number of occasions, the CIO was visited and interviewed by reporters from our student newspaper, The Collegian. Although they were usually friendly visits, we were held accountable for the uses of the student technology fund and for the other funds generated by student participation in IT ventures. It is more than fair to say that we have enjoyed the support of our students in these projects.

And where are we at now? We have generated $550,000 per year on student telephone service resale. We have a commitment for $300,000 per year from the student technology fund. We have support from both the President and the Board of Trustees, and we have support from the students.

Now all we needed was to move ahead with our new high-speed network. Unfortunately, there still was a bit of a cash flow problem. Paying for the new telephone switch, the voice mail system, the call accounting system, and all of the residence hall wiring left us broke. We didn't even have the cash to hire a communications consultant to design our campus network. In the words of the CIO, for our IT staff, this would clearly be "an opportunity to excel."

The Telecom engineers had a pretty good idea of what a high-speed network should look like. It would have a network infrastructure that would last ten to fifteen years and hardware that was scalable. By now, we knew how much money we had -- $850,000 a year -- but we didn't know if it was enough to build a network. We put together an RFP that was no more than four or five pages long for a high-speed joint enterprise network. We sent out the RFP's, and followed that up with a pre-bid meeting which was mandatory for all participating vendors. We told them our plans and asked them to come back with a recommendation. We told them that we had $850,000 a year for ten years to spend... now go design us a network.

The joint enterprise piece of the RFP was the vendor's opportunity to show how we could generate more revenues on services they could provide as part of the high speed network proposal. It also gave them the opportunity to include other things in the proposal which would differentiate theirs from others. Let me say here that the successful bidder was IBM. Under their proposal, we acquired a brand new mainframe IBM/9672-R14 computer for no additional funding, just the reallocation of service dollars. The reduction in maintenance costs from the old mainframe to the new helped cover the cost of the entire new mainframe. We even earned a few extra bucks by selling our old 9672 mainframe to a company which was in more dire straits than we were.

Some of the other vendors proposed in their joint network enterprise that the University would provide free modem access to students on one of our 200 modems. The deal was, if you wanted BETTER dial-in service, you could sign up with the vendor for its premium service. The net profit from this feature would be split, with the University's portion being used to offset part of the network expense. Other vendors proposed that they provide cable TV to residence hall students (for a charge, of course). Again, the profits would be split with the vendor over the life of the contract. The proposal to provide ethernet access from the residence hall rooms was the same as the cable TV deal -- we'd split the profits with the vendor.

We chose not to get into any of the revenue-sharing endeavors, although you may decide that they'd work for you at your institution. Since we had a technology fund established, we thought it would be inappropriate to charge the students on top of it in order to offset network expense.

The essence of the joint network enterprise is that we asked the vendors to share in the risk of future earnings. In the examples I've mentioned, the vendors had proposed that they would earn X million dollars over ten years. Let's say it's $8 million. The vendor cost over ten years on those earnings was approximately $3 million. This left $5 million to be shared by the University and the vendor. Based on the projected earnings for the University of $2.5 million, we would reduce the net cost of the network by that amount. Therefore, instead of having to borrow $8 million for our network, we would only have to borrow $5.5 million.

Here's the great part, another part of the "scam," and the part where the vendor rolls the dice. If the vendor were to fall short of meeting their projected earnings, the University would be held harmless. We would have already received the network at a discounted cost, and we would not be responsible for any additional funding or penalties. If, on the other hand, the vendor exceeded the projected earnings, then they would keep the excess revenue with no further obligation to the University. As an institution, we would support the vendor during the life of the contract, but we were not obligated to do any marketing or to collect any money for them.

As we indicated, IBM was chosen as the successful bidder. We set up an on-site office for them in our Computer Center, took a deep breath, and started the work -- the preparation and the installation -- of building ourselves a network.

Let's back up for just a moment to the old network. Although it was patchworked and unreliable, a number of our faculty were still managing to do great things with it in their research and in their teaching. Through the student technology fund, we had mediated a number of classrooms across the campus. We had installed distance learning facilities for some team-teaching endeavors. One of our professors, Bernie Bopp, was busily downloading data direct from the Hubble Telescope in order to bring the Hale-Bopp comet (no Bopp relation) right into his classroom. Professors of Pharmacy and Biology used the network for interactive and up-to-the-moment lectures on the computational chemistry of drug interaction and on the latest in sheep cloning. These individuals have also been among the leaders in mentoring their colleagues about the greater possibilities which would come with our high-speed fiber optic network.

Throughout this entire time, Jerry and the IT managers continue to meet and work with our Capital Projects gang, the Faculty Senate, the Executive Cabinet, the Board of Trustees, and various student organizations for hand-holding, status reporting, and reassurance.

The fiber optic network will connect all eighty-plus buildings on our four campuses (the main campus in the center of Toledo, our community college which is about a mile and a half southeast of that, our downtown facility a few miles farther, and the brand-new Lake Erie Research Center on Maumee Bay). It will connect all classrooms, all administrative offices, all residence hall rooms.

(Joe Drees)

In many ways, the financing of this project was the most challenging task. For me, it was difficult to wait for the go-ahead to start my job to make this project a reality.

I am the network engineer who put together the design of the network products and applications. I've been at The University of Toledo for six years and have spent most of my time keeping the old network running. In this portion of the presentation, I'll talk about my "bag of tricks" and the many creative solutions we used to maintain the aging network infrastructure. Experience being the greatest teacher, much of that information was used to set our path for the new network. As I explain where we've been and where we're going, I don't profess to be the expert in all areas, but we have blazed many new trails and have learned much along the way. Rather than go through the technical details of network design, we'll go through a building-block approach to chart our course from here to there.

They say that necessity is the mother of invention and, believe me, we invented a lot to keep the old network running. That experience was very useful, as we could document knowledge learned and directions followed, and then use that information to set criteria for the new network.

You've already heard about the worst problem with our old network -- the physical cable plant. We were charged with building a fiber infrastructure with a life expectancy of ten to fifteen years and, with this being the majority of the cost involved, it had to be done right.

Although we could spend a full day discussing and justifying the products and protocols that we chose to implement, I'll simply describe the choices made. Incidentally, I should note that our decisions were made almost two years ago, with one equipment change made early in the project. But, in spite of the many technology advances made since then, if we had to go through the process again, we would make the same decisions.

Our old network was a Chipcom Ethermodem network which linked many of the buildings on campus. This system is basically standard 10-megabit ethernet transported over a cable TV system. You may have heard of this, or you may be in an area where the cable companies are using this technology to provide Internet access to homes. It does work but, using it as a campus backbone, we exceeded the capacity on the system and then created a second channel to double our bandwidth.

The total length of the broadband cable already exceeded the manufacturer's specifications, so all new campus buildings and remote locations had to be connected into the network by way of T-1 connections, mostly over campus copper. There are several technical reasons behind this, but the buildings with 1.5-megabit T-1 connections proved to have faster and more efficient access than those on the 10-megabit ethernet.

While cable TV systems are generally very reliable, our system was not. The company which did the design was two hundred miles away and they didn't realize that our campus is built in a natural flood plain. Our manhole and conduit system also seems to function as a backup drainage system. The combination of salt, water and ice creates an environment particularly harsh to electronic equipment. But that's where the equipment is, and I've seen cast-aluminum housings turn brittle and shatter, and other alloy fittings reduced to a glob of red gel.

The failure rate was growing exponentially as the equipment aged and failed faster than we could replace it. At the direction of our new CIO, we moved from quietly fixing the failures to a more open position that this was inevitable and that the only true fix would be a new network infrastructure. After the "right" people (that is, the administration) were inconvenienced due to network outages, the ball began to roll to find a real solution.

To say that I enjoyed the years of that network would be an outright lie, but we did do much more than just function as caretakers. We also learned how to make the most out of what we had and, by necessity, learned to control traffic flows, create redundant paths, and set priorities for protocols and utilization.

As we have already described, the inherent characteristics of the broadband/T-1 network couldn't be changed. We could, however, minimize the impact of the problems. At one time, broadcast traffic chewed up 22 percent of the already-limited bandwidth. Broadcasts are controlled by routers, so we put some in. Our latest count was 34 routers with 57 distinct routed segments. Most network analysts would describe this as excessive but, when it was done out of necessity, it solved the problem. There were some disadvantages to this approach, primarily in the complexity of maintaining an outdated and overloaded routing protocol but, overall, routing became a positive characteristic that we wanted to continue.

Failures on the net meant that traffic from one location couldn't go across to a server or host machine. (As we mentioned earlier, the CIO himself found this out when he couldn't print to the printer right outside his door if the network failed, because the print queue was on a server on the other side of campus. After that embarrassing incident, regional servers went in the very next week.) We couldn't prevent the network failures, but we could minimize the impact by moving the services out into the various buildings on campus, closer to the user.

AppleTalk and IPX are very "chatty" protocols, and we tried to discourage their use in order to conserve the limited bandwidth. Additionally, it was determined that the most crucial traffic was IP and, during segment outages or Band-Aid patches, only IP was enabled on extremely congested links. With all the creative routing and redundant linking we were doing, supporting multiple protocols was an administrative headache. Experience can be a cruel teacher, but the lessons are well learned.

As the saying goes, "KISS -- Keep It Simple, Stupid!" As far as our new network was concerned, we had to have an information technology infrastructure for voice, data and video that was, above all, reliable. It also had to be fast, fault-tolerant and flexible. We had to build something that would last for at least ten years (since that's how long it would take to pay it off).

Our only choice was fiber. Resistant to everything but fiber-seeking backhoes, this medium is guaranteed for twenty years. Currently, there is no upper limit on the speed at which it can carry data. Voice, video and data technologies are merging and can easily co-exist on a fiber infrastructure.

Fiber was the easy choice of media, but the actual layout or topology was a subject of discussion. We needed to find a compromise between the economics and applications. Our original design called for a ring of stars, but the consultant had previously created a full ring proposal that he wanted to re-use. After months of negotiations, we agreed to a physical hybrid, to a ring with spurs. By back-feeding the trunk cable, we were able to implement it logically as a ring of stars.

There are six locations on campus which host attachments to the main ring. Six to eight neighboring buildings use that facility to connect to the ring. The ring layout creates two paths between any devices, so that a single break in the fiber would not interrupt communications. The many stars in the layout prevented any single point of failure. Even a catastrophic failure of one region would only affect the neighboring buildings.

Each building on campus will get a star-wired riser plant. All data closets are fed with a bundle of fiber cables from the building entrance facility. The only copper in the plant is in the horizontal runs from the closet to the desktop.

The backbone contains over 48,000 feet of cable (nine miles of 72 strands). Add that to three miles in length of spur cables and five miles of riser, fuse the strands end-to-end, and you'd have a fiber link reaching from Toledo to Orlando.

Now came the easy part for me -- the network level design. The electronics fit easily onto the physical layer design, which was an obvious advantage of having a cohesive singular plan.

As a brief overview, our network now circles a dual OC-12 ATM ring. Running LAN emulation, this gives us a 1.3-gigabit total switched bandwidth with the ability to sustain traffic after a cable break or a single equipment failure. The Cisco 5500's are native ATM switches with ports available for future connections of voice and video sources throughout campus.

The basic 7500 router configuration includes an ATM interface to attach to the backbone, as well as six full-duplex fast ethernet interfaces. With an interface per building, this gives us the routing capability we formerly had in thirty separate routers.

Each building takes the 100-megabit feed and, through the Catalyst 5000, delivers a switched 100Mb segment to each of the distribution closets. Catalyst 3000's then feed the network to a combination of shared and switched 10Mb ethernet ports to the users' desktops.

Information Technology is a multifaceted work group containing voice, video and data departments. We believe that these groups will continue to converge and that a high-speed fiber ATM network will best serve the long-term interests not only of this group, but of our entire University community as well.

(Louise Easly)

And there you have it -- some facts, figures and foibles about our old network and about how we built "a multimillion dollar network with no money down." Time constraints didn't allow us to present the most minute of details about the process and the network, but we hope you'll feel free to contact any one of us (or anyone at UT) about any facet of it.