While today started off like any other, by the end of the day it was one that will surely be remembered for some time here at the Pole.

The morning started off like any other. I got up after a night of restless sleep, threw on some clothes, washed up a bit, and headed to the galley for a light breakfast. After breakfast I took care of some chores that had accumulated in my e-mail inbox overnight before calling family back in the States to wish them a Merry Christmas. While I was on the phone, the fire alarm went off indicating a fire emergency in the New Power Plant (NPP). There were no instructions for non-emergency personnel, so I decided to remain out of the way in my berth and continue my phone calls.

Minutes passed and the alarm was repeated several times. Eventually, an announcement came over the Station’s All Call telling all personnel in the Elevated Station to report to the galley. Just moments after that announcement was made, the power went out to my room, dousing the lights and killing my phone call. I grabbed my water bottle and headed for the galley.

Upon arriving the galley I gathered with some other IceCubers while we waited for news. Eventually, the station manager made an appearance and gave us some information.

The Station uses a clever system whereby waste heat from the diesel generators that provide the Station’s power is circulated through the station in a glycol loop to provide heating.

Unfortunately, this morning the heat exchanger system failed and glycol started leaking out on to the running generator, Generator #3. Once the glycol touched the hot generator it was vaporized, producing a thick cloud of toxic smoke and vapor. Soon, there was a toxic could of glycol in the air and a thick lake of glycol on the floor.

The two workers in the power plant at the time were overcome and while the Station’s emergency response crews worked to control the mess in the power plant, trauma teams rushed the two workers to Medical. Volunteers were needed to help carry the litters up to Medical (on the second floor of the Elevated Station; so a carry of three floors). I’d rather do almost anything in an emergency than just sit around, so I volunteered to help carry the injured, even though it meant combining two of the hardest activities here at a high elevation, carrying something heavy and climbing stairs.

Once the injured were in Medical, I helped one of the cooks unload the Station’s freezer and place the food outside so that it wouldn’t defrost while the Station’s power was out. It wasn’t sexy work, but it kept me busy.

Meanwhile, other teams of people were tasked to find Material Safety Data Sheets (because of the glycol), flashlights, fans (to help clear the air in the NPP), and other sundry items.

Simultaneously, technicians were dispatched to the Emergency Power Plant to get it up and running. Crews couldn’t begin to assess the damage in the NPP until they had power to run fans (to clear the air) and produce light. The Emergency Power Plant is located in my berthing pod, and soon the pod was filled with the very loud sound of the Emergency Power Plant and the distinctive scent of diesel emissions.

Once the Emergency Power Plant was on-line the call went out to turn off every non-essential item in the Station, including the coffee makers. It wasn’t long before another fire alarm sounded, summoning emergency response crews to the Emergency Power Plant because it was running too hot and producing too many fumes.

Sometime during the preceding, myself and another IceCuber volunteered to work as runners because there was serious doubt about how much longer the Station’s handheld radio system would be able to run since no generators were running and the UPS to which it was connected was rapidly draining.

We were first tasked to go wake people up in Summer Camp, which is the name for the Jamesways. Suffice it to say that we didn’t make any friends rousting people out of the Jamesways and mustering them in the galley. There are three shifts here, and many of the people in the Jamesways had only been sleeping for an hour to two before we rousted them out of bed.

Next we were detailed to run out to the skiway and act as human beacons to keep vehicles from crossing the skiway while a Herc from McMurdo was on approach and landing. Two others had originally volunteered for the job, but as the plane entered its base leg, they were nowhere to be found and out we went. As we were heading out, we were passed by a snowmobile carrying the two missing human beacons so we returned to our stations as runners in the large conference room where resources were being marshalled.

While waiting there for jobs to do, we could hear what the Emergency Response teams were doing over the radio. Four more people ended up in Medical, for a total of six, with smoke inhalation.

Since much of the air inside the NPP was toxic, those entering the room had to be on oxygen until the air could be cleared. In addition, those dispatched to Medical were placed on oxygen. As a result, the station was running out of oxygen bottles, so a flight from McMurdo was dispatched with a variety of emergency supplies requested by the Pole, including another power plant engineer.

It was about lunchtime at this point. There were five runners available (all IceCubers) and we took turns going down to the galley for lunch. Since there wasn’t any power, lunch was cold cuts, cheese, bread, tortillas, salad, bars, and cookies. Runners were dispatched to the NPP, Medical, Comms (the Station’s nerve center), and other locations bearing sandwiches, hot and cold drinks.

Once the air in the NPP was somewhat cleared, and most of the thick glycol lake had been swept and shoveled by the Emergency Response crews, runners were dispatched to muster volunteers to mop the NPP floor so that power plant crews could work on getting Generator #2 up and running.

Soon after the volunteers started mustering in the Barn, the Station’s smoking bar, I was tasked to round up as many mops as possible. So, I started digging through janitor closets on my way down to the Dome. Janitorial supplies are kept in the Dome, so myself and another IceCuber started rummaging around looking for mops. We came up with something like fifteen mops all told, which was plenty.

Once the mop crews got the floor in the NPP cleaned up, maintenance crews got to work bringing up Generator #2. The Emergency Response crews were stationed in and around the NPP in case they were needed.

After a few fits and starts, Generator #2 came on-line. It was determined at this time that the trauma teams were no longer needed, and neither were the runners, so we were all dispatched to our regular jobs.

Tonight, the Station is operating on Generator #2 while crews work overnight to put Generator #1 into service and restore the glycol heat exchanger system. The generators can work without the glycol system, but not very well or very hard since they have a harder time shedding their waste heat without the glycol system. The glycol system drained on to the floor of the NPP early on, which means that 35 barrels of glycol have to be retrieved from storage and fed back into the system. Then, the system has to be pressurized and tested before it can be put back in to service.

To ease the load on Generator #2, the Station is operating under an aggressive power reduction regime tonight. Non-essential systems are disconnected from Station power and won’t be turned on until tomorrow morning at the earliest. Almost all of IceCube’s component bits are turned off as part of this shutdown. The Station’s heat is currently being generated by boilers that were installed to back-up the glycol system.

What can be learned from this incident?

First, Emergency Response and trauma crews here at the Pole, while mostly staffed by volunteers drawn from around the Station, are committed and skilled. They worked hard, without complaint or rest, from 09:01 this morning when they were summoned, until 17:00 when the Station officially stood down. Victims were treated and moved to the professional care of the Medical staff as quickly as reasonably possible.

Second, five years after 9/11, the importance of testing emergency equipment is still unlearned. While the Station’s radios lasted throughout the incident, there were serious doubts about the endurance of the backup power supply that provided power to the radios at critical points during the day. Several times I heard how rechargeable flashlights intended to be used by Emergency Response crews weren’t charged, and hence were worthless. The equipment that could have been used to recharge the oxygen bottles couldn’t be used because it didn’t have a supply of power.

Third, it is not enough to have a backup generator or two, they must be frequently and realistically tested. Most people who have a backup generator fall in to three categories:

  1. Those who test their backup generator frequently, but not realistically.
  2. Those who rarely if ever test their backup generator, being content to simply have one on the premises.
  3. The small remainder that frequently test their generator under a realistic load and fastidiously perform maintenance on the system to ensure its readiness.

My experience with backup generators was hard-won while evaluating data centers for some of my past jobs. Most data centers fall in to category one. They may test their backup generator(s) monthly or even weekly, but they almost never test it under any sort of load. Then, when their primary power supply fails and a huge load falls on to the backup generator, they can’t understand why it collapses under the strain shortly after it fires up.

The fire alarm created in the Emergency Power Plant shortly after it fired up is a dead giveaway that Raytheon and the NSF have not established a rigorous testing regimen for the Emergency Power Plant’s backup generators and that they clearly fall in to category one.

Fourth, while we joke about Antarctica being a “harsh continent” while we drink milk made from powered milk in the galley or pour wine into empty beer cans instead of schlepping down to the galley to get some glasses, that statement has more than a bit truth to it. McMurdo provides the nearest help during an emergency and it is an absolute minimum three hours away by plane. That three hours is an almost ludicrously low figure since it is the flying time from McMurdo to the Pole. In all likelihood, you’d be looking at four to five hours before a plane could be dispatched from the coast with needed help and supplies until it lands on the skiway.

If someone is seriously injured, the nearest hospital is in Christchurch, New Zealand, which is a minimum eight hours away. That’s like being seriously injured in Boston and but being forced to fly for medical help to London or Paris with a stop-over and a change of planes in Iceland. Realistically, you’d be lucky to get to Christchurch in less than fifteen hours since they don’t keep C-17s spooled up on the runway at McMurdo for just such an incident. Fortunately, none of the six people who suffered injuries today required a medevac flight.

Hopefully, tomorrow will see the power plant issues resolved and life here at the Station can return to normal.