Dispute Erupts Over What Sparked an Explosive Li-ion Energy Storage Accident

A little after 8:00 p.m. on April 19, 2019, a captain with the Peoria, Arizona, fire department's Hazmat unit, opened the door of a container filled with more than 10,000 energized lithium-ion battery cells, part of a utility-scale storage system that had been deployed two years earlier by the local utility, Arizona Public Service.

Earlier that evening, at around 5:41 p.m., dispatchers had received a call alerting them to smoke and a bad smell" in the area around the McMicken Battery Energy Storage System (BESS) site in suburban Phoenix.

Sirens blaring, three fire engines arrived at the scene within 10 minutes. Shortly after their arrival, first responders realized that energized batteries were involved and elevated the call to a Hazmat response. After consulting with utility personnel and deciding on a plan of action, a fire captain and three firefighters approached the container door shortly before 8:00 p.m., preparing to open it. The captain, identified in a later investigation as Captain E193," opened the door and stepped inside. The other three stood nearby.

The BESS was housed in a container arranged to hold 36 vertical racks separated into two rows on either side of a 3-ft-wide hallway. Twenty-seven racks held 14 battery modules manufactured by LG Chem, an 80 kW inverter manufactured by Parker, an AES Advancion node controller used for data collection and communication, and a Battery Protection Unit (BPU) manufactured by LG Chem.

The battery modules in turn contained 28 lithium-ion battery cells of Nickel Manganese Cobalt (NMC) chemistry. These modules were connected in series, providing a per-rack nominal voltage of 721 V. The entire system had a nameplate capacity to supply 2 MW of power over one hour for a lifetime energy rating of 2 MWh. With 27 full racks, there were 10,584 cells in the container. After a full day of charging, the batteries were around 90 percent of capacity.

With the door to the BESS container open and Captain E193 at its threshold, combustible gases that had built up inside since the incident began several hours before received a breath of oxygen and found an ignition source.

The gases erupted in what was described as a deflagration event." Firefighters just outside of the incident hot zone said they heard a loud noise and saw a jet of flame" extend some 75 ft out and 20 ft up from the door.

Photo: Arizona Public Service View of debris and damage to the rear door, HVAC systems, and the container.

In the explosion, Captain E193 and firefighter E193 were thrown against and under a chain-link fence surrounding the facility. The captain landed more than 70 feet from the open door; the firefighter landed 30 ft away.

The captain's injuries included a traumatic brain injury, an eye injury, spine damage, broken ribs, a broken scapula, thermal and chemical burns, internal bleeding, two broken ankles, and a broken foot.

The firefighter suffered a traumatic brain injury, a collapsed lung, broken ribs, a broken leg, a separated shoulder, laceration of the liver, thermal and chemical burns, a missing tooth, and facial lacerations.

The timeline and series of events is not generally disputed. However, a dispute has erupted in recent weeks over what exactly happened inside the BESS container at around 4:54 p.m. that initiated a thermal runaway that cascaded across multiple battery cells.

Photo: Arizona Public Service The side door to the McMicken BESS was ejected in the explosion and can be seen laying against the fence in the foreground.

In a report released in late July, the utility and its third-party investigator, DNV-GL, said that their review of the evidence pointed to the failure of a single lithium-ion cell as triggering the events.

In a separate, preliminary report filed days later with state officials, LG Chem, which supplied the li-ion batteries, challenged that finding. The South Korea-based battery supplier said the APS report missed a number of details about the accident. Those details, LG Chem told regulators, indicated that the cell thermal runaway began due to intense heating" caused by a heat source such as external electrical arcing" on one of the battery racks.

Scott Bordenkircher, who served as APS' Director of Technology Innovation & Integration at the time of the accident, said in an interview that the utility accepts the findings of its third-party accident investigation, which was completed by Davion Hill, Ph.D., the U.S. Energy Storage Leader for DNV GL. We have confidence in our third-party investigator," Bordenkircher said.

Image: Arizona Public Service According to the APS report, a single cell failure propagated through Module 2, then consumed the whole rack, releasing a large plume of explosive gases. The report said this process could have occurred without visible flame, which could explain why the gases were not burned as they were emitted.

In its 78-page report [PDF], DNV GL said that what was first thought to be a fire was in fact an extensive cascading thermal runaway event within the BESS. That event was initiated by an internal cell failure within one battery cell, identified as cell 7-2 on Rack 15. The failure was caused by abnormal lithium metal deposition and dendritic growth" within the cell, the report said.

Once the failure occurred, thermal runaway cascaded from cell 7-2 through every other cell and module in Rack 15 via heat transfer. The runaway was aided by the absence of adequate thermal barrier protections" between battery cells, which otherwise might have stopped or slowed the thermal runaway.

Image: Arizona Public Service As thermal runaway moved through the rack, flammable gases continued to concentrate. Fire suppression devices were ineffective on the event.

As the event progressed, a large amount of flammable gas was produced within the BESS. Lacking ventilation to the outside, the gases created a flammable atmosphere within the container. Around three hours after thermal runaway began, when firefighters opened the BESS door, flammable gases made contact with a heat source or spark and exploded.

It was a tragic incident," Bordenkircher said.

It also was not the first time that a lithium-ion battery had failed.

The APS report listed events reaching back to 2006 that involved thermal runaway events in lithium-ion batteries. In one widely report incident in January 2013, a Boeing 787-8 experienced smoke and heat coming from its lithium-ion battery-based auxiliary power unit. It was later determined that the failure was caused by an internal cell defect, which was exacerbated as thermal runaway cascaded through all the cells in the battery pack, releasing flammable electrolyte and gases.

The state of the industry is that internal defects in battery cells is a known issue," said Hill. Even so, problems with the technology have not been well communicated between, say, the personal electronics sector and the automotive sector or the aerospace industry and the energy industry.

Overall, across the industry there was a gap in knowledge," Bordenkircher said. The technology moved forward so quickly, he said, that standards and knowledge sharing had not kept up.

The McMicken BESS accident also was not the first for APS. In November 2012, a fire destroyed the Scale Energy Storage System (ESS) at an electrical substation in Flagstaff in northern Arizona. The ESS was manufactured by Electrovaya and consisted of a container housing 16 cabinets containing 24 lithium-ion cells.

An investigation into that accident determined that a severely discharged cell degraded and affected a neighboring cell, touching off a fire. The root cause of the 2012 accident was found to be faulty logic used to control the system.

The control logic had been updated more than two dozen times during the 11 months that the BESS operated. But several missed opportunities could have prevented the fire that destroyed the unit, the accident report said. It pointed in particular to an event the previous May in which a cell was severely discharged" even as the logic was continuously charging the cell against the intended design." After the May event, the logic was not changed to address that improper behavior.

An APS spokesperson said that lessons learned from this 2012 incident were incorporated into the design and operation of the McMicken BESS.

In its 162-page rebuttal [PDF] of the McMicken accident LG Chem refuted the utility's finding of fault with its battery.

Image: LG Chem LG Chem's accident report included this chart showing the voltage profile of the Cell 7 pair during the thermal runaway event. A post-incident test set up to match the pair damaged in the accident did not match, leading the company to conclude it is unlikely that a single cell failure caused the voltage excursion.

The battery supplier said that based on available evidence, metallic lithium plating did not cause an internal cell failure leading to the initial thermal runaway event" at the McMicken BESS facility. Instead, cell thermal runaway began through intense heating of the affected cells caused by an external heat source, such as external electrical arcing on Rack 15.

LG Chem said that its own third-party investigator, Exponent Inc., tested the internal cell failure theory. It did so by forcing a parallel cell configuration into thermal runaway. It then compared the resulting voltage profile to the voltage profile recorded during the incident. It found that the two did not match, leading to the conclusion that the explosion's cause was unlikely to have been an internal short within a single cell."

The battery maker also said that data recorded during the incident showed a discharging current of 4.9A (amps) present during the voltage excursion. It said that although the APS report acknowledged that the current flipped from -27.9A charging to 4.9 A discharging,it offered no explanation for the event." To LG Chem, however, the fact that the discharging current was at 4.9A, instead of zero, means the current indeed flowed to somewhere else," supporting what it said was a likely double-point electrical isolation failure and not an internal cell short.

(Complicating the post-accident investigation was the fact that the fire destroyed system control electronics within the container. That left dozens of battery modules energized with no way to discharge them. It took seven weeks for the utility to figure out a plan to remove the modules one by one and bleed off their stored energy.)

The reports and their divergent conclusions signal the start of competing interpretations of available data as the utility and its battery supplier work to find a single cause for the accident.

We don't want a public argument about it," said DNV GL's Davion Hill. For him, the main point is that we had a cascading thermal runaway that led to an explosive atmosphere" at the APS McMicken BESS. The goal now should be to make storage systems safer through standards development and information sharing.

After the accident, APS placed a hold on BESS deployment across its service territory. The technology is seen as key to meeting the utility's announced goals to produce 100 percent clean energy" by 2050. Two other BESS systems that had been operating at the time of the April 2019 accident were taken offline; they will remain idle until retrofits can be designed and installed.