By Patrick J. Lynch P.Eng.
By Patrick J. Lynch P.Eng.
September 12, 2016 – This was the local U.S. utility’s worst nightmare. Its control room had lost all computer systems, UPS power, utility power, back-up generator power and phone systems… and smoke was quickly filling up this now pitch-black room.
They completely lost control of the grid.
This utility has failure contingency plans stacked on top of other failure contingency plans. They conduct failure contingency meetings six times annually. A system risk assessment reliability engineer is also on staff. They have triple redundancy on many of these systems, yet everything seems to be completely failing all at once.
Smoke was detected on most floors, and the entire building was evacuated. How could this happen?
The final back-up was a small, old unoccupied control room left over from the “old school days”, which was located about two hours away. They would have to “patch together” some sort of makeshift control system from there.
Was this a terrorist attack?
It would appear everything had been perfectly timed and coordinated: loss of all power, crashed computer and phone systems, and the building filling up with smoke… all of this had occurred at the exact same time. Was this simply a crazy coincidence? Was there a logical explanation?
Electrical engineering forensic investigation begins
Our company was commissioned to investigate this rather bizarre and tragic sequence of events. To begin with:
• Smoke damage in the building was minimal.
• No employees were injured.
• Grid control was reacquired within 6 hours.
This was the Good News. Now for the Bad News:
• The main electrical switchboard is a complete disaster area. It suffered a major electrical hit and was hanging on by a thread.
• There was no longer any electrical backup diesel generator power available at this site. Uninterruptible power supply and computer systems have also been compromised.
We arrived at the site a day after the disaster had struck, where we discovered the severely damaged switchboard was still energized, powering the entire 20-floor building. All staff had returned to work. The utility carried on business as usual.
Who had authorized this? This was completely unacceptable!
Electrical busbars had melted within this switchboard (see photo above). Electrical busbar standoff insulators and all the switchgear were full of black, electrically conductive, carbon soot flash residue. It could flashover again at any time. This was a potential second catastrophe waiting to happen, and it could be much worse than the first one.
We met with the general manager to explain the severity of the situation. Bottom line? Power down this main switchboard as quickly as possible.
Their four diesel generators (each 750kW/480V) and synchronizing switchboard had also been involved in this electrical failure. The client had no backup power, so an additional 2MW diesel generator was quickly rented and set up at the site. With the site completely powered down for 8 hours, this generator was electrically spliced into the electrical system, downstream from the failed switchboard. The site was running on diesel power alone, 24 hours a day until the main switchboard could be repaired.
(Utility power was finally restored 23 days later to the failed switchboard, which now sported newly fabricated switchgear components.)
The results of our failure investigation revealed the following…
Following the clues
We learned diesel generator operational power transfer switching tests were never performed at this site. Staff were concerned that power blips may cause computer system operational problems.
The transfer switch scheme allowed all four synchronized diesel generators to be directly connected (in parallel) to the utility power system before transferring to pick up the electrical load within the building (make-before-break switching arrangement).
A building water pipe had burst nine weeks earlier. Some of this water had found its way into the synchronizing/paralleling control circuits for these generators, blowing up the circuit boards. While the damaged circuit control boards were replaced at the time,
• The backup generator system was never tested after those repairs. Again, there was a staff concern over a potential power blips affecting operations.
• The overall backup electrical system was never thoroughly examined for any other additional failure points.
It appeared the electronic paralleling sequence control boards for this main switchboard transfer switch were also damaged during the water leak, but went undetected until the complete power failure occurred.
Moments before the disaster, this commercial area had a “temporary loss of power” situation. This “defective” main switchboard transfer switch attempted to operate, then erroneously connected 3 MW of full diesel power out of phase directly onto the now fully energized utility power grid system.
This led to severe electrical arcing in the main switchboard, phase-to-phase and phase-to-ground flashovers, resulting in complete electrical destruction.
The enormous amount of electrical flashover energy then transferred through to all the other building electrical loads, knocking out the UPS, phone and computer systems, etc. In several instances, electrical components on each floor caught fire, producing smoke.The damage to this building and operational downtime was extreme.
For every problem, a solution
In the end, the entire electrical system required a major redesign to eliminate all single points of failure, including:
• System grounding
• Electrical isolation points
• Wraparound/bypass electrical options
• Load banks (permanently installed for system testing)
Note that a single point of failure in the initial electrical design crippled the entire grid control computer system. Both utility and diesel power were left stranded with no alternate electrical path to power up these critical loads.
Finally, staff and management were made to understand that mandatory monthly full-power system transfer tests for a critical site (like this one) is never optional, but a requirement!
Patrick J. Lynch, P.Eng., has been the president of Power Line Systems Engineering Inc. since 1986. He graduated Electrical Engineering from the University of Waterloo in 1975, and has successfully directed Power Line’s completion of over 1100 complex electrical engineering site disturbance investigations around the globe. Visit www.powerlinesystems.ca.