Fighting Power Outages at SATVPosted: May 29, 2010
The month of May at SATV has been …
[BEEEEEEEEEEEEEEEEEEEEP] [Sound of UPS alarms] [Sound of cell phone message notifications] [Sound of cell phone ring] [Repeated]
We’ve had several power interruptions in Downtown Salem in the past month, and several others earlier in the year, one of which happened in the middle of an update to the Exchange server. We lost half a day’s email on that one.
This was one of our live shows:
I was working for that show at the time and I was just about to hit a key on our Inscriber to start the show introduction when the power hit, or didn’t.
A week later, I was at SATV again around 6 PM, processing the video for the Salem Commission on Disabilities meeting on which I sit. Darkness. Alarms. My cell phone message tone sounded over and over. I was in a dark room, which was not fun but at least I had the light of my laptop to see by. This was a longer outage, lasting about 25 minutes.
SATV doesn’t have backup power building-wide. We have a UPS in the server room, and two more in Cablecasting. These worked as well as can be expected during a power failure. We used to have a Comcast-managed demarc equipment room in the furthest corner of our facility, but this has been literally ripped out and replaced by 6U worth of fiber optic equipment in our Cablecast racks.
A notable gap is our phone system; the PBX itself is protected but the individual phones are not—they stop working during a power incident. It’s a very sad reminder of the days when all phones and even some PBX’s were provided by Ma Bell and powered by the central office and stayed up no matter what. I have investigated PoE (Power Over Ethernet) switches—SATV is planning a refresh of our network hardware—but there are few affordable options to us.
We are too small to have generators—we are in leased space and the only place they could go is the roof, if our landlord would even allow it; as well, it would not be cheap. (I have never been approached about running a Home Depot generator in the building. Fortunate, since I’d then have to explain about enclosed spaces, fire regulations and CO.)
We thought about using a spare UPS we have to power the control room; Our control room rack power is supplied by two power cords with NEMA 5-20 plugs. Our UPS doesn’t have compatible sockets, in fact none of ours do. Unless we want to completely rewire the rack, we can’t use our UPS..
In talking with Sal, we’re reluctant to get higher-capacity UPS units, since the costs rise very, very quickly when you go beyond the usual 1500 VA units, the biggest one can buy for regular office power. UPS’s are not made to power everything for a long time, but just to carry over brief interruptions and keep the loads up until the generators start and power is switched in.
If our facility disappeared and I was asked to design a new one from scratch, I would have definitely put the server space close to the cablecast space and close to the Comcast & Verizon demarcs. And configured for 208V three-phase in those areas. And had a locked door. And working HVAC. But very few of us have been fortunate enough to get that blank slate. Not me, either.
It’s just about impossible to predict what will happen next and how National Grid will deal with it. Our power situation—Downtown Salem in general—reminds me of Salem 35 years ago when we had a very old water and sewer system. Then, water mains would rupture seemingly every month. One May evening in the late-70’s, Salem’s water main downtown let go.
It was in the train tunnel under Riley Plaza. There was, at one point, several feet of water in spots on street level. Never mind the tunnel itself, which was completely submerged. I note with cheer that virtually all of Salem’s telecom cabling ran through this point connecting with the New England Telephone (now Verizon) CO that was and still is just nearby.
DSL service—my DSL service—now runs through that tunnel. We don’t have a map of Comcast’s cable junctions, but we do know their major cable vault downtown, near Summer & Norman Sts., was prone to flooding, too.
What surprises do our aging electrical systems have for us? I already know that a car with a stuck accelerator near the Ward Two Social Club (on Bridge St. Neck, five blocks NE of my apartment) can and has dropped power downtown, along with the splintered pole he ran into.
National Grid is frustrating in its lack of communications; they have a cute Google-based outage map on their webpage that is useful for National Grid itself, but not for its customers or residents. Why can’t they do what the MBTA does? I get service notices to my email and cellphone for bus routes I regularly take. Why couldn’t National Grid have a system that had you put your zipcode and email address or cell number into its system and get notifications of outages or planned maintenance.
For that matter, National Grid already generates internal reports on power incidents. Why not let us see them in a daily or weekly summary? (“There was a 2-second drop on one phase at the Canal St. Substation, affecting some customers in South Salem…”)
This is not a problem for SATV to solve as much as it is the city’s. Downtown Salem loses power several times a year; we just go off the air, but the many other businesses lose money. Big money.
Unfortunately, no neighborhood association will ever solicit for improved electrical distribution or refurbished substations.