Conferences & Events
Events for all Levels and InterestsStay
Jump Start Your Career GrowthStay
Get on the Higher Ed IT MapStay
Uncommon Thinking for the Common Good™Stay
EDUCAUSE Southeast Regional: Weathering the Storm Panel
EDUCAUSE Southeast Regional: Weathering the Storm Panel
Summary: Weathering the Storm -- Preparing for, Responding to, and Recovering from Emergencies
This session was recorded for podcast.
Tuesday, June 12, 2007
Kathryn F. Gates, Chief Information Officer, University of Mississippi
Betty Hawkins, Program Manager II, University of South Carolina
David J. Sliman, Director of Technology-Gulf Coast, University of Southern Mississippi
Frank O'Quinn, Deputy Policy and IT Disaster Recovery Officer, Louisiana State University
Along with Charley, Ivan, and Katrina, hurricanes and other natural disasters are a part of life in the Southeast. Beyond the weather, many other calamities might hit us at any time. While many of us have been involved in efforts to develop emergency preparedness, disaster recovery, and business continuity plans, often these sit on a shelf and have little to do with how the institution really responds in an emergency. How can we plan and prepare in a way that is realistic and meaningful? How can we collaborate in advance so that we will be prepared to help each other when disaster strikes? What are the critical functions of IT before, during, and after an emergency? How can IT best facilitate long-term recovery efforts? This panel will feature several IT professionals who have experienced disastrous situations first-hand and can offer valuable insights to others.
Kathy Gates introduced the three perspectives of disaster recovery and business continuity from three different institutions. The themes of the panel will cover planning, collaboration and partnership, and managing technology in emergencies.
David Sliman, whose campus (University of Southern Mississippi) and home were totally destroyed ("everything south of 100 miles of railroad track was gone"), began the session with descriptions of the hurricane damage, his gratitude for all of the support that continues to be offered, and his lessons learned from the experience.
1) Have your personal life in order (insurance docs and all)
2) Plan how far to evacuate - no more than Â½ tank gas away
3) It may be difficult to get back so who will be in charge and who has the right documentation to be in charge. Get in touch with officials ahead of time for the right documentation to make sure you can get back on site.
4) Health and welfare of co-workers should be first and paramount
5) Know how you will contact your employees. It may be that you'll have no cell phone towers. Perhaps only text messages will work. Have alternative meeting places.
6) Communications are important:
a. Develop a phone tree for communications (divide and delegate) NOAA has a call in service (1-800 number)
b. Make sure you have your employee's spouse or relative's contact information
c. Also have your employee's children's information
d. Activate text messaging for everyone
e. Have off-site web services arranged or answering service check-ins
7) Have good records of your inventory for insurance (home and work) and take them with you when you evacuate
8) Secure intellectual property and this includes faculty data/papers/books/
9) Disasters come in many flavors. It is the planning not the plan that is most important
10) Be prepared - ahead of time!
There were a few good things like
- Slab Boils instead of Crab Boils (parties on the slabs of concrete that were left)
- Trunk'r'treat - Halloween was trunk by car trunk instead of door to door
- Sculptures have been created from many of the damaged trees
Frank O'Quinn, who was appointed the LSU IT disaster recovery guy one week before Katrina, hunkered down at home with gas in the car, filled propane tanks, batteries and candles, had peanut butter and extra water, games to play with the children, and waited for the power to go out. After the flooding the campus closed and everyone stayed home and cleaned up from the storm and waited for power to return. Then it was a matter of being of service to those critically affected.
LSU's role in these situations is to:
1) Take care of students and employees first - closed the campus
2) Provide a social services location for a special needs shelter.
3) Use their basketball arena to house the shelter which always opens prior to storms for people to use.
LSU & Katrina was a different situation
- There were more people affected and many more responders arriving.
- No one was in charge of these "many more people"
- Determined tasks, organized volunteers, activated an emergency operations center
The temporary medical operations and staging area (TMOSA) that they set up processed over 40K patients. They had to keep the TMOSA open until these people had some place else to go. They were still open when Rita hit and finally closed after 52 days.
LSU had no detailed written Memorandum of Understandings (MOUs) and was not a federally recognized emergency operations center and so they spent over $1 million on everything from telephones to tracking patients. LSU absorbed that 1 million in hard cash - not reimbursed.
Everyone was very emotionally involved but it was also very important to return to the primary mission
- Classes resumed Tuesday following labor day
- Home football game switched to away
- Need to encourage guests of dorm residents to find other arrangements - Faculty staff students swap to get families out of dorms into homes of LSU community - Everyone gathered on the football field with signs of who they could take
Board of regents - declared any student enrolled in an affected state 4 yr school permitted to enroll in any other state 4 year school
For the enrolled displaced students
- Timing was just right - few missed days
- IT had to modify processes, no fee admits
- Existing WEB portal unfeasible
- No dorm rooms, apartments, houses
- Constantly learning lessons and applying them immediately into the process.
On top of enrolling displaced students they were still running the shelter and the requests after Katrina increased for services such as phone, networking connections, pc laptops printers and more. They went with VOIP phones and felt it was a good decision. Their vendors delivered equipment and Frank reminded us of the importance of maintaining good relationships with vendors.
They took on many tasks that were outside their normal roles - like broadcast emails, call center, patients database, volunteer database (many students), map of campus for media, responders. They continued to adjust administrative applications and had twice daily meetings with the chancellor for status checks.
A few of the changes since Katrina - They now have a formal LSU Emergency Operations Center and Formal MOUs and agreements with state agencies and private sector services for 4 days of food, fuel, and water on campus - with secondary suppliers backing up primary ones.
They also have a full time generator for their basketball arena and have pre-planned emergency logistics.
In 2006 they prepared for repeat hurricanes, especially critical now that people were living in mobile homes. IT disaster recovery now has a tested hot-site for mainframe activities and the Chancellor requested written plans from all units on campus. These plans are being updated for 2007
Observations from their experiences:
- Things will be unpredictable. They won't unfold as you planned and you need to be fluid.
- Priorities change by the minute
- Some projects will be aborted after much effort which will cause disappointment and hurt feelings.
- You have to watch out for ideas being suggested that are illegal (ie HIPPA)
- Keep your supplies stocked
- Think about what people still aren't thinking about. People still don't comprehend the unpredictability and they never believe it will happen to them
- Don't wait for the money
- Do the right thing.
Know the big picture
- your mission
- your resources
- your people
Betty Hawkins had no direct experience with Katrina or Rita but as she discussed,
Katrina was just one disaster but things happen every day to different people around the world. She has worked in the area of emergency preparedness for many years and has seen both disasters and incidents of many types.
Disasters: long term or permanent damage
- Power failures
- Blizzards & ice storms
Incidents: short term interruption
- Hardware failures
- Software failures
Experiencing a disaster throws you into continuous process improvement.
- People don't plan to fail - they just fail to plan. Lack of planning adds to the mayhem.
- It is important to build relationships with all groups in your setting. This may include groups beyond the local area.
- Plans are effective if people know about them
- Your emergency planning should include marketing avoidance or otherwise stated: how to prevent an incident or disaster.
- Each of us should also have a plan for home emergencies.
Creating the plan is the easy part. It's important to engage others to think about the following layers of the plan:
- Incident recovery
- Disaster recovery
Lessons she has learned over the years include:
- If you complete the plan today, it will be obsolete tomorrow. Continuous planning and preparation is needed.
- Someone needs to own "disaster recovery" at the institution level and the unit level. They must do it and practice it
- Plan on at least one FTE for your IT organization.
- No two disasters are alike
- Be prepared for emotional fallout
- Understand that panic spreads quickly
- Every organization is different so try different things
- Planning and conducting mock disasters is the best way to demonstrate preparedness
- Create an email suggestion box
- Provide positive recognition - volunteers, best ideas for improvement, certificates
Ideas to share information
- At the University of South Carolina she publishes a newsletter every semester. She brings in local EMS, Fire, Police, Red Cross, etc., to write articles or speak on campus and they love to do this.
- Changing the mindset is important so that people think about the little things they can do everyday - her illustration was to put a spare car key with your ID badge.
- University wide business impact analysis (BIA) - circulating this form will generate thought
- Give brown bag presentations to Business managers and keep it easy to understand
- Start a users group and invite various experts to come and speak
- Get the DR & BC people talking with their counterparts at other colleges and universities. (USC's role in Katrina - in less than 24 hrs the IT group set up a place to do IDs for evacuees and talk to state services)
- Reciprocal agreements between organizations should be looked at very closely and determine what are the resources to manage over time
- Create a web site and remind people to visit often. Make sure it includes what IT will provide for recovery and how IT will help in an emergency.
Topic: CIO perspective on emergency planning - How should the CIO prioritize?
- Betty suggested that CIOs think about it now and brainstorm with others/stakeholders to develop plan
- Frank believes that communications comes first (from the business side of the house receipts come first but employees think payroll comes first)
- David said communication plans are first
Comment from Monte Luehlfing: You find yourself doing things you never thought you'd be doing - like running a day care center. You have to take care of people first.
Topic: Disaster Exercises -
- Betty indicated that they do hot site exercises (strictly IT) and they find something wrong every time. The exercises are to find out what's wrong. They are not to document what's right. If your exercise is perfect then you haven't succeeded. You should set up a different goal each time you do an exercise.
- Frank said the minimum is that the smoke detectors go off once a month. However they pay visits to their hot site to bring up the backup regularly. Their A Team tests and finds a number of things that they then work on tweaking.
Question: What should the DR technology be? Should we have a black box or a remote site or something else?
- Frank said if main building goes down the alternate site goes up. However, distance is important, 200 yards away not enough. Multiple sites and farther away may be best.
Question from Sandy Schaeffer: Where does the external motivation come from to do emergency planning?
- Kathy indicated it was the State in her situation
- Betty said it was the recognition of a vulnerable point - (happens because it is important)
- Frank suggested that it comes from within the organization but people are "too busy" to do anything individually so organizations have to bring it down from the top. Plus it is "almost equivalent" to a requirement in the state in LA.
Final Question: What should we take away as action items to do immediately upon return to our institutions or key lessons?
- Depends on where you are - look at your own group and where you are vulnerable - low hanging fruit
- People became leaders after Katrina: grow from where you are
- Emotional impact is significant so you must figure out how to deal with the emotional and psychological issues. Keep staff busy so they don't think about other things.
- LSU has changed everything they do and now asks for a DR budget as well (see http://www.educause.edu/SERC07/Program/12202?Product_Code=SERC07/SESS01)