Tech Stories

Issue #68

Hello Reader,

I'm travelling for work this weekend, so I'll keep this brief.

Two data centre disasters struck Asia Pacific this week. On Wednesday, a ruptured liquid cooling pipe caused chaos at a Southeast Asian facility. On Friday, a fire at one of South Korea's government data centre knocked out hundreds of crucial services.

Let's talk about data centre disasters this week.

My friend Junhan is running a 1-day paid masterclass in Singapore called "Visualise Your Value" this week.

He is extraordinary in doing visuals - this image is an example, and he'll teach you to express your ideas better.

Learn more

Data centre 'water feature'

On Wednesday, a trusted contact sent me a news tip. It was a video showing workers squeegeeing away a massive pool of water... in a data centre aisle. Oh dear. I was finishing my post for the day and briefly considered pivoting to cover this instead. But I decided to wait, using the time to make additional enquiries.

I'm glad I waited. After making further enquiries, I had to omit mention of the affected GPU server system affected, the country, or details of the data centre involved. Same goes for the CDU brand or sharing the actual video, despite multiple requests. I wanted constructive learning and discussion, not get someone fired or naming and shaming.

There were a few self-promotional comments, but the almost 80 comments are worth reading over coffee. Here are two if you're strapped for time.

Joe Capes of LiquidStack argues that such incidents are very rare and likely caused by human error, inexperience, or corner-cutting. He wrote: "Any water cooled close coupled cooling systems have the potential to leak, as do the chilled water pipes serving CRAHs... Piping, tube and interconnection components have been perfected. Water leaks in the DC stopped being a major issue 15 years ago."
KJ (Kwang Joo) Lee might have hit the nail on the head though, from what little I know. He wrote: "Subcontractors usually end up hiring mostly unskilled engineers. The contractor must allocate materials and manpower according to the prescribed construction methods..."

Lithium-ion batteries on fire

On Friday, a lithium-ion battery fire in a South Korean data centre disrupted hundreds of online government services, prompting the Prime Minister to apologise. I read over a score of reports, then dug into the history of the National Information Resources Service (NIRS) data centre and fire regulations. Turns out the facility is close to two decades old, with UPS and batteries installed alongside servers at the rack level.

On the one hand, the batteries aren't concentrated in a battery room, so the fire probably spread slightly slower. On the other hand, given that this was a government data centre, firefighters were likely instructed not to perform dampening operations with copious amount of water to preserve IT systems.

But as you likely know, lithium-ion battery fires have this nasty habit of reigniting after being extinguished. So the poor firefighters had to repeatedly attack fires with CO2, itself dangerous at -78°C release temperature. They contended with ambient temperatures up to 160°C, wore full breathing apparatus, and yes, tried to unplug or unscrew UPS batteries in the dark. No wonder over 200 firefighters were activated - they had to keep rotating teams due to exhaustion.

The ultimate irony? The battery explosion and fire happened as technicians were attempting to migrate the lithium-ion UPS to the basement for safety. This incident differs significantly from the fire at Digital Realty's data centre in Singapore last year, which housed its lithium-ion UPS systems in a battery room. Firefighters truly have their work cut out for them in data centre fires involving lithium-ion batteries.

If you have any thoughts about this fire, would love for you to drop a comment here.

Disasters that can happen, will happen

I'll conclude with this thought: Disasters that detractors say can happen in the data centre will likely happen. We've seen it this week - liquid cooling can leak, lithium-ion batteries can catch fire, and both can bring operations to their knees.

These incidents are sobering reminders that as we build data centres at breakneck speed, we must prepare for the worst with clear emergency protocols and proper training. Because when things go wrong in a data centre, they go very wrong indeed.

You can reach me by hitting reply.

Regards,
Paul Mah.

You can now comment on stories published on techstories.co. If you received this newsletter from me, your account was created automatically.

All you need to do:

Scroll to Member discussion and sign in with the email used for this newsletter.
An email message with a magic link is sent immediately. Click to log in.
Comment away.

Spotlight