To Hell & Back in Five Days: a case study in disaster recovery and emergency response
A talk by at ISA WWAC Symposium
About the Talk
August 4, 2010 12:00 PM
Orlando, FLOrlando, FL
Between February 7th and March 14th, 2009, more than 400 bush fires across the state of Victoria, Australia scorched over a million acres of land, killing 173 and injuring 414. Goulburn Valley Water (GVW), provider of urban water and wastewater services to 54 towns and cities on the outskirts of Melbourne, watched as their telemetry system from the Kilmore Dissolved Air Filtration plant reported an ambient control room temperature of 142 degrees Fahrenheit before going silent on the afternoon of Saturday, February 7th. A site visit on the following day revealed that while the treatment plant survived the fire, its control room was completely incinerated, destroying the electrical switchgear, plant HMI, laboratory, instrumentation, and chemical dosing systems. With only five days worth of water stored, an emergency response plan to rebuild the control room and recommission the plant went into action. What followed over the course of the next five days is a case study in how proper backup and change management procedures, strong vendor relationships, and dedicated, cross-trained employees can pull off near miracles. Armed with a full set of accurate as-built drawings, up to date PLC programs, and HMI computer backups, GVW was able to assemble a portable control room in the parking lot of their corporate headquarters, deliver it to the site, connect thousands of control points, commission the plant, and resume delivery of 10ML per day of water to the towns of Kilmore, Wandong, and Heathcote Junction in five days. This presentation will discuss the challenges faced by management, employees, and vendors, while outlining the policies and procedures that allowed them to execute their emergency response plans so successfully during this catastrophe.