Main Nav

Was wondering if anyone else out there had significant issues with the freak October snowstorm this past weekend.
 
Drew was hit particularly hard.  We were in an area of New Jersey that got a lot of wet, heavy snow, and we have thousands of large oak trees that lost huge limbs.  In addition, we have an interesting situation with electrical power--the town of Madison has its own electric utility and we get our power from them, and they in turn bulk purchase electricity from JCP&L, who had hundreds of thousands of customers without power. 
 
Drew's power (and the entire town of Madison) went out on Saturday afternoon around 4:30.  All our server rooms have backup generators and switched on cleanly and remained running all day Sunday.   Power hadn't been restored by Sunday night and the University made the decision to cancel classes, recommended students go home if possible, take a friend who could not easily go home with them, and the remaining students were shipped to a nearby school and slept on cots in the gym.
 
Monday a few of us in IT went to campus to determine the scope of the situation.  The generators were still running, and we were running temporary power to some switches so we could do work as one ran out of fuel early in the morning.  We were able to migrate most essential services out of that data center before the UPS power gave out, and most things were shut down cleanly.   However, that room also houses our Internet connection and thus we were offline for about half an hour until our (excellent) Facilities staff refueled the generator from 5 gallon yellow diesel cans (we have a diesel storage tank on campus, but no fuel trailer to transport the fuel).  The 30 gallons they put in would last us at least 8 hours, we were told.  The other generators were checked and had enough fuel to last overnight if necessary.  At any rate, a fuel delivery was made that afternoon and all the generator tanks were topped off.
 
Power was restored to campus Monday afternoon, but it turned out only 2 of 3 feeder cables were energized.  As we were in an undervoltage situation, the generators didn't shut off.  We made the decision not to manually switch the server rooms back to utility power.
 
Tuesday morning, however, the power situation changed.  One part of our campus loop started behaving oddly--it turned out some high voltage equipment was damaged but we didn't know that at the time.  At any rate power started fluctuating wildly, and for some reason the generator supporting that data center turned off and wouldn't reactivate.  We had about 45 minutes of our server room UPS switching on and off utility power, and we were afraid it was going to get damaged.  We again migrated services out of the room, turned off our redundant disk array in the building, and shut down the core network switch to avoid any damage.  The equipment was down for several hours while Facilities brought in the municipal electric authority and high voltage contractors to assess the loop and make temporary repairs to restore normal voltages and service.  Permanent repairs are awaiting the delivery of parts--high voltage equipment is not something electric supply places just have in stock.
 
A brief assessment shows that we had little or no permanent damage due to the power issues.  We may be having issues with a UPS powering an aggregation switch but it was also 6 years old and in need of replacement.  Most importantly, except for the brief generator outage and the corresponding loss of our Internet connection, we had little or no disruption in services for the duration.  The University remains closed tomorrow, with students asked to return after 4pm Thursday for regular Friday classes.  It is likely we will extend the semester, which was previously scheduled to end December 14th, but no official determination has been made.
 
I'm extremely proud of my staff and colleagues who came in over the last few days to respond to issues as they were occurring and our response likely avoided damage as well as downtime.  I'm also thankful for our Facilities staff who were all working 24-7 keeping things together.  This was the longest unscheduled shutdown we've had, far worse than Hurricane Irene, and longer than when our main administrative building was destroyed by fire in 1989.  The power issues we experienced were unprecedented.  Interestingly, I did not see many people referring to our university emergency response manual, and most of our response was coordinated by a few mid-level administrators who knew their areas and made the hard decisions.  We were prepared to shut things and move to an emergency website but we never had to activate that option.  We used Facebook, Twitter and campuswide email to communicate technology issues, and the University used our emergency notification system to great effect. 
 
It's still too early to talk about lessons learned but I hope we have the time to sit down and do so, and although it's unlikely we'll have this scenario occur again, we probably have gained knowledge that will help us for the next unimagined crisis.
 
--Mike
 
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Comments

Thank for sharing, Mike. Sound like an acceptable ending to an unforeseen event. Good luck. If you ask, I'm sure the community (CIO list) will help. 

Dr Robert Paterson 
V. P. I T, Planning & Research Molloy College 

Sent from my iPhone

Message from ingerman@vassar.edu

Mike:

Thank you for sharing your experience.  I know that we all learn something from each of these stories about campuses coping in the face of unexpected circumstances.  Thankfully, while Vassar lost tree limbs, we did not lose power.  But your story makes me wonder about our generators.  They are exercised regularly, and I know that our facilities staff makes sure that they have fuel.  But in an emergency, when they a being pulled in many directions, I wonder where refueling is in their priority list? A good question to ask now.

  --Bret

Sent from my iPad

Bret,
 
What I learned is that we have 18 generators on campus, and most are in fact to operate the life safety systems such as emergency lights and sprinkler pumps.  In fact, the ones that power our server rooms also power those needs, and one of our server rooms is in our public safety building and that building is always powered during an outage.  Because of that, Facilities has a plan to monitor and refuel generators during an extended outage.  I think this is the first time we've been without power for more than 24 hours.
 
--Mike


 
 
Mike Richichi
Director of Computing and Network Services
>>> On 11/3/2011 at 7:34 AM, Bret Ingerman <ingerman@VASSAR.EDU> wrote:
Mike:

Thank you for sharing your experience.  I know that we all learn something from each of these stories about campuses coping in the face of unexpected circumstances.  Thankfully, while Vassar lost tree limbs, we did not lose power.  But your story makes me wonder about our generators.  They are exercised regularly, and I know that our facilities staff makes sure that they have fuel.  But in an emergency, when they a being pulled in many directions, I wonder where refueling is in their priority list? A good question to ask now.

  --Bret

Sent from my iPad

Message from ingerman@vassar.edu

Thanks, Mike.  What it comes down to is that the best of intentions can be compromised by the unexpected.  To help combat that we have a Crisis Response Planning Group who meets monthly.  At each meeting, we engage in a small table-top exercise that puts a different campus officer in the role of "incident commander" and that plays out a possible crisis scenario.  Then once a year we have a larger tabletop exercise with local law enforcement and other first responders and that includes all of the directors from areas that would likely be involved should there be a crisis (such as IT, dining, academics, residence life, etc.). Working through issues on a monthly basis has really helped us to see gaps and ask questions such as how will the generators be filled (which has not yet come up).  In fact, it was one of these tabletop exercises that led us to purchase a campus-wide automatic computer backup solution (from Crashplan) when we realized that key offices were not necessarily storing their key documents on the central file servers available to them.  Rather than fit the uphill battle of telling people where to store things, we decided to make the backup process effortless.  I can also say that the tabletop exercise where we learned this also helped my colleagues on senior staff to see the need for such a solution and to encourage its use among their staff and faculty.

  --Bret

Sent from my iPad

Message from dthibeau@post03.curry.edu

Mike,

 

Generators are great, but you’re right, they do need fuel.  We opted for natural gas powered generators, which offer underground fuel supplies.  Those are less likely to be impact by snow, wind, ice, or high winds.  Though, they are certainly susceptible to earthquakes.  We have a permanent generator at each of the three buildings where are data centers are located, and they are setup to power on automatically in the event of an outage.  The UPS systems are in place to handle any short momentary outages.

 

If people start thinking about adding generators – consider natural gas.

 

You also make a good point about testing.  I wonder how many of us have periodic DR testing scheduled?

 

Dennis

 

From: The EDUCAUSE CIO Constituent Group Listserv [mailto:CIO@LISTSERV.EDUCAUSE.EDU] On Behalf Of Bret Ingerman
Sent: Thursday, November 03, 2011 7:34 AM
To: CIO@LISTSERV.EDUCAUSE.EDU
Subject: Re: [CIO] Snowtober

 

Mike:

 

Thank you for sharing your experience.  I know that we all learn something from each of these stories about campuses coping in the face of unexpected circumstances.  Thankfully, while Vassar lost tree limbs, we did not lose power.  But your story makes me wonder about our generators.  They are exercised regularly, and I know that our facilities staff makes sure that they have fuel.  But in an emergency, when they a being pulled in many directions, I wonder where refueling is in their priority list? A good question to ask now.

  --Bret

 

Sent from my iPad


Although data centers are only one aspect of business continuity that needs attending to, I thought you might be interested in seeing some data about higher education institutions' disaster recovery provisions for their data centers. The 2011 Core Data Survey results show the following: - 23% of institutions have no disaster recovery provisions in place for their data centers - 81% of those with a disaster recovery site have not tested it in the past year This is an estimate for all higher education institutions nationwide. The margin of error is approximately plus or minus 3%. I've attached a brief excerpt from the forthcoming report that provides more details if you are interested. -Susan Susan Grajek Vice President for Data, Research & Analytics EDUCAUSE Uncommon Thinking for the Common Good http://educause.edu 4772 Walnut Street, Suite 206, Boulder, CO 80301-2538 202-331-5350 (phone) sgrajek@educause.edu From: Bret Ingerman > Reply-To: The EDUCAUSE CIO Constituent Group Listserv > Date: Thu, 3 Nov 2011 07:53:16 -0400 To: > Subject: Re: [CIO] Snowtober Thanks, Mike. What it comes down to is that the best of intentions can be compromised by the unexpected. To help combat that we have a Crisis Response Planning Group who meets monthly. At each meeting, we engage in a small table-top exercise that puts a different campus officer in the role of "incident commander" and that plays out a possible crisis scenario. Then once a year we have a larger tabletop exercise with local law enforcement and other first responders and that includes all of the directors from areas that would likely be involved should there be a crisis (such as IT, dining, academics, residence life, etc.). Working through issues on a monthly basis has really helped us to see gaps and ask questions such as how will the generators be filled (which has not yet come up). In fact, it was one of these tabletop exercises that led us to purchase a campus-wide automatic computer backup solution (from Crashplan) when we realized that key offices were not necessarily storing their key documents on the central file servers available to them. Rather than fit the uphill battle of telling people where to store things, we decided to make the backup process effortless. I can also say that the tabletop exercise where we learned this also helped my colleagues on senior staff to see the need for such a solution and to encourage its use among their staff and faculty. --Bret Sent from my iPad
Thanks so much for taking the time to write out the details of your experiences.  We all learn from these posts.  Our experiences with electrical power failures and generators have been very difficult to manage.  Every situation has been different, so you really need thinking, problem-solving people on hand, rather than a scripted action.  You've given us more to think about.

Theresa

Our campus lost power on Saturday evening and our generators took over without any problems.  Like Dennis, our generators are fueled by natural gas, so as long as the supply lines are not disrupted, we can remain on generator for a significant period of time.  Fortunately, power was restored to most of the campus on Sunday, so cleaning up the downed trees and branches and students attending classes happened simultaneously on Monday.  Unfortunately, for many of the folks here, the power outage situation at home is now into its 5th day. . .

 

Bill

 

From: The EDUCAUSE CIO Constituent Group Listserv [mailto:CIO@LISTSERV.EDUCAUSE.EDU] On Behalf Of Thibeault, Dennis
Sent: Thursday, November 03, 2011 8:28 AM
To: CIO@LISTSERV.EDUCAUSE.EDU
Subject: Re: [CIO] Snowtober

 

Mike,

 

Generators are great, but you’re right, they do need fuel.  We opted for natural gas powered generators, which offer underground fuel supplies.  Those are less likely to be impact by snow, wind, ice, or high winds.  Though, they are certainly susceptible to earthquakes.  We have a permanent generator at each of the three buildings where are data centers are located, and they are setup to power on automatically in the event of an outage.  The UPS systems are in place to handle any short momentary outages.

 

If people start thinking about adding generators – consider natural gas.

 

You also make a good point about testing.  I wonder how many of us have periodic DR testing scheduled?

 

Dennis

 

From: The EDUCAUSE CIO Constituent Group Listserv [mailto:CIO@LISTSERV.EDUCAUSE.EDU] On Behalf Of Bret Ingerman
Sent: Thursday, November 03, 2011 7:34 AM
To: CIO@LISTSERV.EDUCAUSE.EDU
Subject: Re: [CIO] Snowtober

 

Mike:

 

Thank you for sharing your experience.  I know that we all learn something from each of these stories about campuses coping in the face of unexpected circumstances.  Thankfully, while Vassar lost tree limbs, we did not lose power.  But your story makes me wonder about our generators.  They are exercised regularly, and I know that our facilities staff makes sure that they have fuel.  But in an emergency, when they a being pulled in many directions, I wonder where refueling is in their priority list? A good question to ask now.

  --Bret

 

Sent from my iPad


Message from shelw@berkeley.edu

Mike, Thanks very much for sharing. Sounds like you had a good plan in place and glad you had the support of a strong facilities team. We in California are known to have our fair share of business continuity challenges on occasions (although thankfully Snowtober isnt one of the!) For those that are considering your business continuity planning activities, I would suggest looking at the community built and supplied Kuali Ready tool . This light weight easy to use SaaS tool allows every department on campus to develop its own plan, create action items, store reference materials, and share or clone other plans within your campus. All information is secured and hosted by higher education institutions. There are 70+ universities utilizing this low cost service and more signing up all the time. You can take get more information at http://kuali.org/ready Regards, Shel Shelton Waggener Associate Vice Chancellor Chief Information Officer University of California, Berkeley mailto: shelw@berkeley.edu phone: 510.642.4096 Regards, Shel

At Assumption, although the College was closed Sunday through Tuesday due to the power outage, our natural gas-powered generator worked beautifully.   We had a couple of battery issues, but redundancy kept those from affecting our services.   Since the ice storm of 2008, we have been undergoing testing of our various generators and emergency communications systems each semester.  We learn something new every time!

Dawn

-----------------------------------------------------------------------

Dawn R. Thistle, D.A.

Executive Director of Information Technology and Media Services

Assumption College

500 Salisbury Street

Worcester, MA  01609

 

508-767-7095

508-799-4502 fax

 

From: The EDUCAUSE CIO Constituent Group Listserv [mailto:CIO@LISTSERV.EDUCAUSE.EDU] On Behalf Of Thibeault, Dennis
Sent: Thursday, November 03, 2011 8:28 AM
To: CIO@LISTSERV.EDUCAUSE.EDU
Subject: Re: [CIO] Snowtober

 

Mike,

 

Generators are great, but you’re right, they do need fuel.  We opted for natural gas powered generators, which offer underground fuel supplies.  Those are less likely to be impact by snow, wind, ice, or high winds.  Though, they are certainly susceptible to earthquakes.  We have a permanent generator at each of the three buildings where are data centers are located, and they are setup to power on automatically in the event of an outage.  The UPS systems are in place to handle any short momentary outages.

 

If people start thinking about adding generators – consider natural gas.

 

You also make a good point about testing.  I wonder how many of us have periodic DR testing scheduled?

 

Dennis

 

From: The EDUCAUSE CIO Constituent Group Listserv [mailto:CIO@LISTSERV.EDUCAUSE.EDU] On Behalf Of Bret Ingerman
Sent: Thursday, November 03, 2011 7:34 AM
To: CIO@LISTSERV.EDUCAUSE.EDU
Subject: Re: [CIO] Snowtober

 

Mike:

 

Thank you for sharing your experience.  I know that we all learn something from each of these stories about campuses coping in the face of unexpected circumstances.  Thankfully, while Vassar lost tree limbs, we did not lose power.  But your story makes me wonder about our generators.  They are exercised regularly, and I know that our facilities staff makes sure that they have fuel.  But in an emergency, when they a being pulled in many directions, I wonder where refueling is in their priority list? A good question to ask now.

  --Bret

 

Sent from my iPad


Close
Close


Annual Conference
September 29–October 2
View Proceedings

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

Digital Badges
Member recognition effort
Earn yours >

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2014 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.