Main Nav

I wanted to informally survey what is being done on other campuses for power backup and conditioning for the network across the campus. We have tried to install small APC UPSes (the Back-UPS Pro models) in all network closets; we want to have 5 – 10 minutes of backup for the small power hiccups that seem all too common. However the time, cost, and organization required to keep up with aging and failing UPSes, bad batteries, etc  sometimes seems to be more trouble than it’s worth. After a number of extended switch down incidents caused by failed UPSes or batteries, I’ve considered slowing replacing them with quality power strips (for surges/spikes, etc) and letting them reboot when the power goes out. The concern by others is this would be somehow hard on the switches and cause increased failures.

 

So I had a few questions about what others are doing:

 

·         Do you use UPSes for all network gear across campus?

·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?

·         What is your experience with keeping up with many UPSes?

·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues?

 

Thanks in advance for your input,

Thomas Carter

Network and Operations Manager

Austin College

903-813-2564

 

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

AttachmentSize
image001.gif1.74 KB

Comments

·         Do you use UPSes for all network gear across campus?


For most closets yes, not all. We have well over 300 closets on campus.


·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?


IT is responsible for upkeep.



·         What is your experience with keeping up with many UPSes?


Not good, money is typically not allocated adequately for battery replacement, nor is enough consideration given to the amount of man hours required to maintain 300+ UPSes. Ive been here 15 years and UPS failures (due to lack of maintenance) are the single largest cause of network outages during that time.



·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues?


I've seen switches fail after power outages, but not nearly as often as I've seen UPSes fail due to lack of maintenance.





From: Thomas Carter <tcarter@AUSTINCOLLEGE.EDU>
Reply-To: The EDUCAUSE Network Management Constituent Group Listserv <NETMAN@LISTSERV.EDUCAUSE.EDU>
Date: Wed, 25 Sep 2013 10:11:17 -0500
To: The EDUCAUSE Network Management Constituent Group Listserv <NETMAN@LISTSERV.EDUCAUSE.EDU>
Subject: [NETMAN] UPS for network across campus

I wanted to informally survey what is being done on other campuses for power backup and conditioning for the network across the campus. We have tried to install small APC UPSes (the Back-UPS Pro models) in all network closets; we want to have 5 – 10 minutes of backup for the small power hiccups that seem all too common. However the time, cost, and organization required to keep up with aging and failing UPSes, bad batteries, etc  sometimes seems to be more trouble than it’s worth. After a number of extended switch down incidents caused by failed UPSes or batteries, I’ve considered slowing replacing them with quality power strips (for surges/spikes, etc) and letting them reboot when the power goes out. The concern by others is this would be somehow hard on the switches and cause increased failures.

 

So I had a few questions about what others are doing:

 

·         Do you use UPSes for all network gear across campus?

·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?

·         What is your experience with keeping up with many UPSes?

·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues?

 

Thanks in advance for your input,

Thomas Carter

Network and Operations Manager

Austin College

903-813-2564

 

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

We have seen more issues with UPS devices failing after a power outage (probably due to lack of maintenance).  We probably don't maintain them as well as we should and these devices are often in undesirable environmental conditions (very warm, dusty).  




--

Jeremy L. Gibbs
Systems Administrator / Network Engineer
Utica College IITS

T: (315) 223-2383
F: (315) 792-3814


·         Do you use UPSes for all network gear across campus? 

o   YES

·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?

o   IT is responsible.

·         What is your experience with keeping up with many UPSes?

o   We have them all set to notify us if they have issues.  We also PING and SNMP monitor the UPS units centrally.  APC and Liebert supply software for centrally managing and maintaining their UPS devices.  We make sure all have network management capabilities. 

·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues?

o   We put UPSes in so we do not have power issues except in cases of extended power outages.  Our UPS protect against short term (1 hour or less) power outages, spikes and brown outs.   We have not had any switch issues related to power. 

 

 

T.R. Knight

Director of Enterprise Infrastructure/CISO

Taylor University
236 W. Reade Avenue
Upland, IN  46989
Office:  765-998-4902

Fax:  765-998-4640
tr.knight@taylor.edu

 

 

When we implemented our VoIP solution, we installed new PoE switches.  We put Liebert UPS’s in each closet (we only have 15 or so) and they are network attached.  We monitor them with Liebert NForm which reports on maintenance issues as well as generates alerts when UPS’s have a status change (loss of city power, battery depleted, power restored, etc).  We haven’t had a unit fail in the last two years since we put them in production (fingers crossed).  We will budget for battery replacement for them next year using a local 3rd party battery supplier, most likely Batteries Plus since they will deliver.

 

My answers are underlined below.

 

 

·         Do you use UPSes for all network gear across campus? – YES, in all closets that have PoE switches for VoIP phones.

·         Is IT responsible for the upkeep, or is facilities/physical plant responsible? IT

·         What is your experience with keeping up with many UPSes? We haven’t done much of anything, though we’ve only had them 2 years.

·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues? In closets that don’t have UPS’s, we have had switches die after a power loss (OLD Cisco switches)

 

 

 

Brandon Riffel – IT Manager, Technical Operations

Ottawa University

 

 

 

·         Do you use UPSes for all network gear across campus?

o   Yes, every stack (~100, ~320 swtiches) is connected to at least one UPS. (APC SmartUPS 3000 XL)

·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?

o   Yes, network team is responsible.

·         What is your experience with keeping up with many UPSes?

o   Generally no issues. It is usually budgeted as network stability.

 

 

 

Tim Cappalli, Network Engineer
LTS | Brandeis University
x67149 | (617) 701-7149
cappalli@brandeis.edu

 

·         Do you use UPSes for all network gear across campus?

About 30% of our 286 closets have UPS.  They are predominately located in the MDF.

·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?

IT is responsible

·         What is your experience with keeping up with many UPSes?

Service contracts on large UPS with periodic service visits keeps these running well but it is an expensive approach.

SNMP monitoring on small IDF locations works well but occasional outages and failure to reboot after extended power outage.

·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues?

Switches tend to be much more reliable than the UPS systems on restarting after a hard failure.

We are moving away from IDF-based UPS and to dual-power supplies for all switches (even the 1U stuff).

One cord in street power, one cord in generator/street (where available), 

Yes, we'll lose network service for the amount of time it takes the generator to spin up and the ATS to kick over but that's usually pretty short.

Our facilities staff load-test the generators every month during the daytime so unless one cord stays on pure street, users will see outage on generator-only sources.



·         Do you use UPSes for all network gear across campus?

We do in all locations with Network equipment.  We have pretty dicey utility power (although it’s gotten a little better in the last few years) and a large number of buildings given our student population. We run about 100 buildings/135 closest and ~2300 students.

·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?

For our edge and distribution locations, ITServices manage the UPS devices.  With the datacenter UPS, our Facilities guys work with our maintenance vendors – but ultimately I believe IT foots the bill for maintenance/upkeep.  Our Datacenter UPS/Generator was financially proposed as a campus benefit though – not only supporting the Datacenter but a few key buildings across campus with the hopes of maintaining a few safe/warm buildings for residents in the event of an extended utility outage.

·         What is your experience with keeping up with many UPSes?

        These are deserving of attention, but easy to ignore.  Battery health needs to be monitored and we regularly replace batteries.  IMO the effort has been worth it – With our not-so-dependable utility power, I can’t imagine the service interruptions we would have experienced and/or equipment lost or damaged due to brown/blackout events.

 

 

Kevin Schoenfeld
IT Services
Hobart and William Smith Colleges
315-781-3711
schoenfeld@hws.edu

 

 

 

 

·         Do you use UPSes for all network gear across campus?
No. Only in our Data Centers.
·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?
IT is responsible. 
·         What is your experience with keeping up with many UPSes?
In my previous organization we had UPSes in all of our closets. It was time consuming and a headache keeping track of the maintenance done (e.g battery change). It was also very expensive to keep them up and running. 
·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues?
At my present organization we do not have UPSes in our closets. We are looking into "heavy duty" power surge protectors to move away from plugging into the wall. I have seen a hand full of devices fail do to power plant issues but for the most part they just reboot (we had a power issue in a dorm this morning and the switch rebooted). 

Thanks,
Gonzalo

---

Gonzalo Cervantes

Associate Director Network Services


Barnard College, Columbia University

gcervantes@barnard.edu

212-854-8795

barnard.edu/bcit



We have APC UPSs (SUA1500RM2U and the newer SMT1500RM2U) in any communication rooms that have generator power available and/or drive VoIP phones. Our facilities people do monthly generator tests, but IT maintains the UPSs.  Some key distribution closets also have UPSs even if they all don't have a generator.  Like you say we might get 10-30 minutes of run time for most loads.

All UPSs are monitored with Nagios for internal temps, power transfers, and battery failures.

I think we have around 95 and try for a 3-year replacement cycle on the batteries, but often see them swell and fail before that time.

We have had some switch failures after a power event when they did not have a battery or power strip.


Message from csmith6@swarthmore.edu

>Do you use UPSes for all network gear across campus? Yes. All network closets (~60) have a UPS of some sort. This was treated as a life safety requirement when moving to a VOIP system a few years ago. In places where the building has a backup generator they are sized to run ~15 minutes. Those places without a generator are sized to run at least an hour. >Is IT responsible for the upkeep, or is facilities/physical plant responsible? ITS manages the UPS's including upkeep. >What is your experience with keeping up with many UPSes? A necessary evil. I would rather replace the UPS than all of the network/phone gear. Having network cards in the UPS (Tripplites are currently being used to replaced aging Powerwares) is the biggest help in keeping on top of them. The cards let us monitor each UPS for failed batteries or other problems by setting up syslog alerts. This also lets us know how often a space losses power, which might be reported or noticed otherwise. >If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues? Even with UPSes there can still be power problems. UPSes don't protect the switches from surges coming from the copper connections out to client devices. We have lost switches, or just the POE function, after big electrical storms that included lightning strikes nearby. Aaron Smith Network Engineer ITS Swarthmore College
On 09/25/2013 08:11 AM, Thomas Carter wrote: ... > · Do you use UPSes for all network gear across campus? As the central network provider for UCSB, we provide core and external connectivity with a demarc at the MPOE of each building. We also have service in wiring closets where necessary for support of campus wireless service. In nearly all cases, a UPS is installed with an SNMP management card, with a minimum run time of 30 minutes. Switches with redundant power supplies are connected to commercial power and UPS power, with the UPS connected to generator-backed power where practical. A nightly cron job checks the load on each UPS and reports changes to the NOC, which helps us follow up with departments that may have semi-randomly plugged into our UPS. Sometimes a bit of adjustment of the UPS settings is necessary for proper operation in a given location. For example, default high/low voltage thresholds may be too wide for the attached gear, so we routinely make these as tight as possible. The sensitivity -- or duration outside thresholds -- trigger for switching to battery may also need adjustment. High sensitivity works best in most places, but sometimes an electrically-noisy upstream generator can cause a UPS to frequently toggle, possibly resulting in early failure. All that said, the clear majority work best with the tightest settings. Now that everything is on UPS gear, we rarely need to take any action after a power event. > · Is IT responsible for the upkeep, or is facilities/physical > plant responsible? IT. > · What is your experience with keeping up with many UPSes? It takes some time, but it isn't difficult. We've talked about contracting battery replacements to our Communications Services group or hiring some students, but there isn't enough work to make it worthwhile. The SNMP cards are key to the process. We could have skipped the SNMP cards, but then we'd have to run around doing manual testing and/or pre-emptive battery replacements. By having the UPS gear do its own periodic testing, we maximize battery life and minimize staff time. Also, during a power failure the cards give us clear feedback regarding the scope and duration of the event, which can help determine whether a NOC staff response is necessary or a call to the on-call Facilities staff is in order. There have been a few outages due to UPS failure, but a more common problem is switch failure due to non-redundant power supplies. > · If you don’t use UPSes, what is your experience with this? > Have you seen switch failures after power issues? We saw some switches hang during brief outages, requiring a site visit and manual reboot. Damaged switches were rare. -- Kevin Schmidt Office of Information Technology University of California, Santa Barbara North Hall 2124 Santa Barbara, CA 93106-3201 805-893-7779 kps@ucsb.edu ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Just to throw in my 2¢

·         Do you use UPSes for all network gear across campus?

Yes, where VoIP phones are concerned. We ocassionally have other access switches (2960's, etc.) outside of our 3750X stacks we're using for VoIP that may not be on UPS power due to load. We expect approximately 15-30 minutes of runtime in the buildings to keep phones online for emergency/disaster/business continuity purposes. Our larger facilities, and VoIP core is on larger UPS infrastructure, with much longer runtimes.

·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?

IT, and more accurately our network operations department, is responsible for the procurement, installation, and maintenance of UPS equipment across campus. 

·         What is your experience with keeping up with many UPSes?

Generally, you get what you pay for. We rely on the APC 1500, 3000, and 5000 XL model UPS and have generally had good luck with them. Batteries do fail, but we get battery replacements here in town for pretty inexpensive, and they're relatively easy to install. Our student techs are about to embark on a project across campus to inventory each UPS and make sure the batteries are healthy. 

·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues?

Even with UPS', switches/power supplies can still fail after the juice runs out. We don't view UPS as a savings of our gear, but to keep phones up and running after building power cuts out. That's where the 3750's come in, with modular redundant power supplies making it easy to swap them if they fail.




Britton Anderson |  Senior Network Communications Specialist |  Office of Information Technology |  907.450.8250



> * Do you use UPSes for all network gear across campus? We've built up over the years to having UPSes in 90% of our network closets across campus. We've aimed for about 1 hour of backup historically in our larger closets, but after recent upgrades we're down to 15-30 minutes. In many cases these are on generators so this is ok. Our primary concern historically has been to avoid equipment failures due to bad power. Nowadays, life safety concerns are more important due to VoIP, security/card access/fire alarms, and other infrastructure communications needs (e.g. building management systems/HVAC). To this end, all UPSses we are buying now are of the online, double-conversion type. We have mainly APC Smart-UPS RT1500, RT2200 and APC Symmetra RM 2-6kVA units. The data centers have large 3-phase Symmetra units and generator backup. Historically, we've used APC Smart-UPS SU1400 and SU2200 units. > * Is IT responsible for the upkeep, or is facilities/physical plant responsible? IT. > * What is your experience with keeping up with many UPSes? All UPSes have SNMP monitoring cards that we monitor with Nagios, Cricket and email alerts, which is key to keeping up with them and fixing any battery issues before they get so bad that the batteries expand and become stuck inside the units rendering the UPSes trash. The newer ones also have the environmental monitoring capability, so we keep track of temperature and in some cases humidity in our network closets. On more than one occasion this has helped us avoid network downtime due to failure of air conditioning units, which we've come to experience as the #1 risk to network downtime these days, so catching these issues quickly has been helpful. Mechanical systems are just not as reliable as electronic systems. We've been getting about 10-15 years lifespan out of the UPS equipment, and between 2-6 years typically for batteries, depending on environmental factors. A bunch of our APC SU2200's are still going after 14 years--their original batteries lasted 9 years before we started seeing failures. I think the older UPSes with the larger 12v18Ah batteries are more durable than the newer ones which all seem to come with the smaller 12v7Ah or 12v5Ah batteries. We rebuild all of our battery packs in-house (but not the actual batteries themselves). It is a lot of work--our rate of battery failure is maybe 1-3 a week during "peak" season when the temperatures are high (not all of our closets have cooling unfortunately). Part of the reason is that we've unfortunately not been able to budget for proactive battery replacement, but we've been able to get maintenance budget allocated to keep up with replacements after failure. In our last few network equipment upgrade cycles, we've also made sure to add UPS upgrades and/or battery replacements as part of the network upgrade cost. > * If you don't use UPSes, what is your experience with this? Have you seen switch failures after power issues? We still have some locations with no UPSes. We see many more network issues in locations without UPS backup, although not too many "permanent" failures that were not recoverable by reloading the switches software/configuration. Our newest switches don't like to lose power suddently, or they corrupt their filesystem (Juniper EX line) but this is somewhat mitigated by their dual-resilient root partitions which back each other up. There is also the "peace of mind" that comes from not having the "unknown variables" of power issues come into the picture when debugging network issues. We've had weird cases in the past where brownouts have caused strange, hard to debug behavior in network equipment. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
UPSes…  We are blessed by very stable campus power, and this colors our perspective.  

We don't have many, and typically they only support the building router (not edge equipment).  Our data centers/NOCs of course have large UPS systems.

The different brands we've tried in buildings have caused more outages than they have avoided (we have them reporting battery states for maintenance to little avail).  So our new deployment guides call for the use of an automatic transfer switch between routers and UPSes to avoid long site outages due to UPS failure (best of both worlds) for single power supplied devices.



-William


********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Thomas,

For new construction where we have buildings with more than one data/wiring closet, we are installing a single, larger UPS (or if the building is significantly large, a couple UPS's) that handles all (or some) of the closets.  Typically this is in the MDF allowing us to only cool the MDF, but vent the IDF's (in general). 

In existing spaces, we use the SmartUPS line (typically SU1500's) with network adapters so we can monitor health.

IT (Networking) is responsible for the maintenance/upgrade/replacement of all units with the exception of a couple that feed our Telecom Frame Rooms (Networking and Telecom are separate IT units).

I actually have a student worker handling the battery replacements.  APC has made it pretty easy (and safe) to do.

We have had failures from power outages, or more accurately brown-outs, in areas where a UPS wasn't used, or the UPS failed.

We are of same philosophy -- we want to prevent the quick up/down/up situation from a power flicker.  Those seem to be more problematic.  Even if it doesn't damage the equipment, there is a higher chance of the system not rebooting properly.

-Brian Helman
Salem State University

I recognize that I am late to the party, but will add my $.02:

·         Do you use UPSes for all network gear across campus?


 No.  We do use rack-mount in perhaps 25% of our most vital closets, and two buildings have central UPS (the library and administation building.)


·         Is IT responsible for the upkeep, or is facilities/physical plant responsible?


 IT is responsible for our rack-mounts.  IT is responsible for the Data-center ups.  For two other buildings with a central UPS, Facilities has taken responsibility over the past year.  (This is a work in progress :-(



·         What is your experience with keeping up with many UPSes?


 It requires diligence.  We have a staff member who is extremely strong in keeping a spread-sheet with scheduled battery replacements (three year cycle, I think) and testing all rack-mount ups at six month intervals.  As others have mentioned, if the UPS are not maintained, they become more of a liability than an asset.

 

 

·         If you don’t use UPSes, what is your experience with this? Have you seen switch failures after power issues?


 We see this sometimes, a couple of times a years after a power outage a switch will fail, tho it hasn't been a major problem.  We more often see switch failures during lightning storms when the electricity feeds back into the switch through the building copper.  

 

Tho we are starting to implement some Juniper switches, and are starting to be aware of the tendency of these to become corrupted with a sudden power outage.  We will be paying attention to this.  We just had our first power outage since installing a few Juniper EX2200s and they did (thankfully) come back up without a problem.

 

 

best,

Dennis Bohn
Manager of Network and Systems
Adelphi University
bohn@adelphi.edu
5168773327 
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.