On the 25th of May 2021, a switching operation was taking place
at the Callide C Power Station. During a planned step in the process, there was an unexpected loss of power
to the systems critical for the safe operation of the turbine generator. In the switch room. the lights went out. In the control room, The screens went black. Within 2 seconds this unit had gone from normal operation,
to something entirely different. And with no screens, the operators had
no way to determine what had gone wron
g. Over the next 34 minutes,
despite the operator's best efforts to understand
and regain control of the system. The turbine generator
ultimately tore itself apart. This is how it happened. The Callide C Power Station is located
near Biloela, Queensland. The site has two coal fired
power stations. Callide B with generator units, B
1 and B 2 and Callide C with generator units C3 and C4. C3 and C4 are adjacent
and share a common control room. Stations B and C are separate
and independent from one
another. It was unit C 4 where the incident occurred. In a coal fired power station, incoming coal is burned in a boiler, heating water and creating steam which drives a turbine. This spins a generator rotor at 3000 revolutions per minute to generate electricity. This electricity is stepped up
through a transformer and is exported
to the CalVale substation. The substation is operated by Powerlink and is part of the Queensland power grid. The safe operation of unit C4 relies on two key electrical
systems the AC system and the DC system. The AC system powers key equipment required for the operation of the turbine
generator. Whereas the DC system provides
monitoring control and protection. As well as backup functionality
in the event of a loss of AC power. The AC system is connected to the grid and supplies
large equipment. It powers the hydraulics
that open and close the steam valves, which regulate the flow of steam
to the turbines. It powers the pumps that provide
lubrication oil to th
e bearings, allowing the rotor to spin freely
without metal on metal contact. And it also powers
the pumps that create an oil pressure seal, preventing the hydrogen gas
that cools the generator from escaping. Meanwhile, while the DC system runs
a range of control and monitoring systems critical
for the safe operation of unit C4. In simple terms, it powers the brain
and life support system of the unit. The DC system runs the 'unit protection',
which monitors the turbine generator for issues
and t
akes appropriate action in response. The DC system also provides
power to emergency backup systems. For example, it supplies emergency lubrication oil and seal oil pumps in the event of a loss of AC. This system is primarily powered by a battery charger,
which also keeps a battery fully charged. This battery provides
important redundancy. If ever there was a loss of AC power,
which would result in the charger ceasing to operate, The battery could continue
to power the DC system. The DC therefore
would still monitor,
control and protect the unit, and its backup pumps
would continue to supply lubrication oil and seal oil to the turbine generator. Unit C3 has its own identical but separate electrical system. There is a third electrical system
called Station. Station AC provides supply for plant common to unit C3 and C4, while Station DC primarily provides redundancy to both units. Station DC has its own battery
charger and battery, while Station AC receives
its power from the units. In th
e 18 months leading up to the Callide incident. An upgrade program had been initiated
to replace the battery charges at C3, Station, and C4. The C3 and Station
battery charges were replaced and successfully
brought back into service by May of 2021. The C4 battery
and battery charger had been disconnected and the battery charger had been replaced. During this time,
the C4 DC system was configured to receive power from Station
through a switch called an Interconnector. Now the battery charger and
battery were ready to be reconnected to C4. The planned
switching sequence had five key steps. Firstly,
the battery charger would be connected directly to the battery
to restore it to a full state of charge. Once the battery was charged,
the battery charger would be disconnected. It would then be
connected to the C4 system. Then, the interconnector from Station
would be opened, disconnecting Station and C4. Finally, the C4 battery would be reconnected. These steps would restore the system to its
typical configuration. The plant has physical safety measures
specific designed to prevent two batteries
from being connected to the same system. This means Station must be decoupled from C4 before the battery is reconnected. But, this sequence does require that the C4 battery charger
be the sole source of supply in between these two steps. By 1:32 p.m. on the day of the incident,
the first three steps of the switching sequence had been carried out
successfully. The Battery had been fully charg
ed. The Battery charger
had been disconnected from the battery. It had
then been connected to the DC system. At this
point in time, the turbine generator was still spinning at 3000 R.P.M. and exporting power to the grid. Then, as the fourth step was completed
and the interconnector was opened, all of this changed. The opening of the interconnector
initiated an almost instantaneous loss of DC power
and AC power to the unit. Without power, the turbine generator had lost critical primary and emerge
ncy lube oil and seal oil pumps, but it was still connected to the grid.
Without protection, the unit could not be disconnected,
nor could it be shut down safely. And without power to their screens the operators in the control room
had no visibility or control of unit C4 and no way of telling
that the plant was on a trajectory towards catastrophic failure. But, what had actually happened
when the interconnector was opened? How did this lead to the loss of DC and why was the AC system lost as wel
l? First,
let's look at the voltage level in the C4 DC system. Up
until the interconnector was opened this voltage was being supplied
from Station. A voltage level of between 190 and 242 volts is required for the system
to operate as designed. So when the interconnector was opened,
the system required the C4 battery charger
to maintain the voltage at this level. This did not occur.
When the interconnector was opened, the voltage in the DC system
instantly collapsed. To understand why, let's look
at how the C4 and Station
battery charges behave when they're connected to the same system.
Before being connected, each battery charger maintains the voltage
level in its respective DC system. This level is determined by each battery
charges configured output voltage. But when they are connected together it's only the charger
with the higher output voltage that supplies the system
lifting the voltage to this level. Meanwhile, the lower output
charger detects that the voltage level in the syste
m has increased and responds
by decreasing its own output. Since the higher output charger continues
to maintain the system voltage at this higher level,
the voltage inside the lower output charger continues to decay. This is precisely
what happened in the C4 charger. In the 74 seconds between the two charges being connected to the same system
and the interconnector being opened the voltage in the C4 charger had decayed to nearly half of what was required. So the instant
the interconnector was o
pened, the voltage in the C4 DC system
collapsed to the level of the C4 chargers internal voltage, just 120 volts. And it was the specific nature of this collapse
that led to the loss of AC power as well. But in order to understand
how this happened, we need to look at how a mechanism
designed to protect the AC system inadvertently led to its loss. A major hazard with high voltage electrical systems is the occurrence
of an electrical arc flash. An explosion caused by electricity
passing through
the air. When this occurs, it is critical
to shut down the power source to prevent continued arcing and further
damage to equipment. The system used to protect
against parking in units C4's high voltage AC
cabinets is called 'ARC Flap Protection'. The ARC flap protection works by applying a DC voltage
to a switch at the top of the cabinet. The presence of this voltage
is monitored by a protection relay. If an arc occurs in the cabinet, the explosive pressure will blow open
a flap on the top. Whe
n the flap opens, it opens the switch,
collapsing the DC voltage to the relay. When the protection relay detects
a voltage collapse below 164 volts, it determines
that the ARC flaps must have opened and then sends a signal to the circuit
breakers to trip the AC power. On the day of the incident,
however, no such arc occurred. Instead,
because the voltage in the DC system collapsed, the protection
relay incorrectly determined that an arc had occurred
and the switch had opened. It then sent a sign
al to the circuit breakers to trip the AC supply. These circuit breakers are powered
by the DC system and in order to trip successfully, they need to be supplied
with at least 101 volts. This is how the specific nature of the DC collapse
led to the loss of AC. Because the DC voltage had collapsed
below 164 volts the protection relay: interpreted
that an arc had occurred; and sent the trip signal to the circuit breakers;
which tripped the ac supply to the unit. All before the voltage had decayed
below 101 volts. If the voltage had remained above 164
volts, the protection relay would not have determined
that an arc had occurred and it would never have initiated
a trip of the AC. If it had collapsed below 101 volts, there would not have been sufficient
voltage for the breakers to operate. If it had collapsed below
80 volts, the protection relay would have powered down
before it could initiate the trip. But the decaying DC voltage was at
just the right level to misidentify
an arc and to tr
ip the AC power. Had the AC power not tripped, the battery charger would have recovered
and restored the system voltage to the required level. However, without AC power,
the voltage inside the Battery Charger decayed to zero, leading to a complete
loss of the DC system. Within 2 seconds
of opening the interconnector, both the AC and DC power systems to Unit C4 had been lost. When AC supply is lost, there's an emergency diesel generator that starts automatically and restores
power to the Station
and Unit AC emergency boards. But the loss of DC supply also managed to circumvent
this backup system. When the emergency diesel generator
detected that power had been lost, it automatically powered on. The Station
DC system then configured. the Station AC switches
so that the Station Emergency board was being supplied
by the diesel generator. However, without C4 DC power, the C4
AC switches could not be configured, preventing the generator from restoring
power to the C4 AC Emergency board. So n
ot only did the loss of DC directly
cause the loss of AC, it also prevented any automatic recovery. There is also a mechanism
in the C4 DC system that automatically responds
to a loss of supply. This is called the Automatic Changeover Switch or ACS. The ACS sits
between the main board and distribution board and monitors
the DC voltage in the main board. If this voltage falls below the required
level, the AC is automatically changes over to supply the distribution
board from Station. However, the
automatic
switching capability of the C4 ACS had been damaged in a previous incident and it could only be operated manually. In this state the ACS had no way to automatically reroute power from Station to the C4 distribution board. So seconds ago unit C4 was functioning normally. Steam was driving the turbine,
spinning the generator at 3000 R.P.M. and exporting electricity to the grid. But the sudden loss of both
the AC and DC systems would lead to the destruction of the unit over the next 34 m
inutes. As soon as AC power is lost, the steam valves slam shut. But this loss of driving power
from the steam doesn't result in the turbine
generator slowing down significantly. Instead, because it is still connected to the grid, the unit changes from exporting power
to importing power. And as it continues to spin with its field
switch open, the generator is now
an asynchronous electric motor. And the unit protections
that would normally prevent this by disconnecting the unit
from the grid and
safely shutting it down, are unavailable due to the loss of DC. This motoring of the generator will continue for the next 34 minutes. Without AC power the bearing lubrication, oil pumps
and hydrogen seal oil pumps stop working. And without DC power,
the emergency DC pumps don't work either. Without all these pumps,
the oil pressures in the bearings drop and the shaft begins
grinding metal on metal and producing heat. And without seal oil pressure,
hydrogen gas begins escaping from the generator
and combusting in the air. On top of this, the loss of AC power
means that none of the cooling systems critical for the safe
operation of the turbine generator and generator transformer are available. These begin to heat up. Over in the control room the C4 displays go blank
because of the loss of power, and the operators are immediately
bombarded by control system alarms. They can hear violent crashes and bangs
coming from the plant. Something is seriously wrong with C4. Within minutes the decis
ion is made to evacuate the site. While some operators will stay behind
to try and understand
what's taking place on unit C4. Has the boiler tripped? Is steam still driving the turbine? Are they still connected to the grid? But with the screens blank,
they have no visibility of what is actually taking place in unit
C4. 10 minutes later at 1:43 p.m., The generator hydrogen
has completely leaked out and the fires at the generator have stopped.
But the unit is drawing 50 megawatts and 350 megavars
from the grid and continues motoring. Because the white metal layer of
the bearings has completely melted away, The shaft begins to lose its center. Ongoing, grinding generates immense heat. At this stage, the shaft has reached
at least 730 degrees Celsius. 20 minutes later, the Operators manage to restore their displays
using another power source. But it's clear to the operators
that the incoming data is inconsistent and can't be relied upon to make it safe
and informed decision. Their major co
ncern
is avoiding an overspeed event. If they decide to ask Powerlink
to disconnect the unit from the grid at CalVale, while it's still being driven
by incoming steam, it will rapidly accelerate and tear itself
apart in a matter of seconds. The Operators ask Powerlink to stand by as they continue
to try and make sense of the situation. It is at this point that the event enters its final stage. At 2:06 p.m., the excessive wear on the shaft causes
the turbine blades to catch on the casing, and the
shaft tears itself apart at nine
locations, ejecting chunks of shaft
from the generator unit. A piece weighing more than 2000
kilograms is thrown five meters across the ground
like a spinning top. The barring gear weighing
300 kilograms is launched 20 meters into the air,
punching through the turbine hall roof. With the generator
still connected to the grid, large electrical arcs start to form,
vaporizing the copper conductors: causing it to pull a massive 300 megawatts
and over 1400 megawatts
from the grid;
nearly three times its rated export power. After 40 seconds of this, the arcing causes an electrical fault,
which is detected at the Calvale substation, leading to its protection
systems operating automatically: Finally,
disconnecting unit C4 from the grid. By this stage, the generator and generator transformer are destroyed. The remaining people on site
are then evacuated with no loss of life. But the incident destabilizes the grid, initiating a cascading failure
that trips nine
major generator units across
multiple power stations in Queensland. A number of factors led to the incident at unit C4. When the interconnector was opened, the battery charger did not maintain
the voltage in the DC system, despite the switching sequence
requiring it to do so. Because the battery charger was the sole
source of DC power at the time, this led to a voltage collapse
in the DC system. And this voltage collapse incorrectly triggered ARC flap protection
which tripped the AC system. Then
, without AC power,
the battery charger did not recover, leading to the complete loss of DC. Because the Automatic Changeover Switch was unable to function
in automatic mode, DC supply could not be rerouted to C4. This loss of DC meant
the unit could not be disconnected from the grid
and would motor for the next 34 minutes. It also prevented the AC system
from being reconfigured to receive power
from the emergency diesel generator. Operators had no visibility of the unit and were bombarded
with
more than 15,000 alarms. With no way to safely regain control
of the unit. At 2:07 p.m.,
the incident reached its conclusion.
Comments