RCA TRAININGRoot Cause Analysis training by Sologic provides the tools, skills, and knowledge necessary to solve complex problems in any sector, within any discipline, and of any scale. Learn More
SOFTWARESologic’s Causelink has the right root cause analysis software product for you and your organization. Single users may choose to install the software locally or utilize the cloud. Our flagship Enterprise-scale software is delivered On Premise or as SaaS in the cloud. Learn More
This is a challenging RCA*, to say the least, because so many things happened in a short amount of time. Essentially, the system was at capacity when it experienced a few failures that, under normal circumstances, would have been relatively easy to manage and likely would not have resulted in such a catastrophic outcome. And although one employee was injured, thankfully no one was killed.
On February 9th, 2017, the Seattle area was experiencing a significant amount of precipitation. Much of Seattle relies on a combined storm/waste sewage system. This means that during heavy rains, the wastewater treatment facilities experience significant volumes. In such events, roughly 90% of volume is attributed to storm water and the other 10% sewage. The West Point Treatment Facility is located on the Puget Sound in the Magnolia neighborhood. It has a capacity of approximately 440 million gallons per day (mgd).
On this night, the plant was operating near capacity when an electrical fault shut power down to part of the plant, including Effluent Discharge Pumps 1 & 2 as well as a hydraulic system which was the single source of hydraulic power to Effluent Flow Control valves. The plant uses a total of four Effluent Discharge Pumps, but only at high tide. At low tide, the pumps are not needed. When the hydraulic system lost power, the valves (by design) failed to the closed position. Of the four total Effluent Discharge Pumps, at the time of the power loss, pumps 2, 3, & 4 were operating while pump 1 was in standby mode. Effluent Pump 2 shut down when it lost power. And Effluent Pumps 3 & 4, pumping against the closed flow control valves, experienced high vibration, which caused them to shut down shortly thereafter. At this point, very little effluent is able to flow out of the plant. Yet millions of gallons are still flowing in.
When the Effluent Flow Pumps shut down, effluent began building up immediately upstream in the Effluent Pump Station (EPS) wet well until the EPS high-high indicator triggered. This caused weir gates to close upstream in the Primary Sedimentation tanks, stopping flow at that point. But then the Primary Sedimentation Tanks began to fill. These tanks are equipped with two high-high float switches which shut down the upstream Raw Sewage Pumps and open the system bypass. However, one of them did not function properly due to a bent float rod. These switches have a history of failure. The rods are easily bent, which then prevents the floats from tripping the switch.
The operators on shift at the time had a difficult time accurately troubleshooting exactly what was happening. They were unaware that the flow control valves downstream from the Effluent Discharge Pumps had failed closed due to no hydraulic power. And they also did not realize that the weir gates on the Primary Sedimentation Tanks had been closed (and locked in place) by the high-high safety switch in the Effluent Pump Station wet well. And there were so many alarms (2,100 in an hour) that the operators in the Control Room were simply overwhelmed with information.
As the sewage continued to be pumped in (and not out), the Primary Sediment Tanks overtopped and began flooding the plant. Operators realized that they were in danger and quickly evacuated the area. However one operator partially fell into a tank through a grating that had been forced out of place and became injured. Other operators helped extract her and move her to safety where they were then able to provide first aid.
Operators then were able to correctly troubleshoot the fact that the flow control valves were failed closed due to the common hydraulic system. They then installed a backup hydraulic system which allowed them to open the valves and restart the Effluent Pumps.
Operators were also able to manually shut down the Raw Sewage Pumps. This caused the raw sewage wet well levels to rise, which then triggered the plant bypass. Downstream levels began to fall. And raw sewage began flowing directly into Puget Sound.
* NOTE: SOLOGIC DID NOT INVESTIGATE THIS INCIDENT IN ANY OFFICIAL OR PROFESSIONAL CAPACITY. THE SOLE PURPOSE OF THIS EXAMPLE IS TO BE USED AS A LEARNING OPPORTUNITY TO DEMONSTRATE HOW A HIGH-VALUE, HIGH-RISK INCIDENT COULD BE ANALYZED USING THE SOLOGIC METHOD OF ROOT CAUSE ANALYSIS AND CAUSELINK SOFTWARE. PRIMARY SOURCES OF INFORMATION FOR THIS EXAMPLE RCA ARE THE (EXCELLENT AND DETAILED) INDEPENDENT AECOM REPORT COMMISSIONED BY KING COUNTY AND SEATTLE TIMES REPORTING. WE HAVE SIMPLY RESTATED INFORMATION PREVIOUSLY PUBLISHED BY OTHERS. ENJOY!