This past spring, we had our annual rite, the Annual Meeting.  This is a board function required by law where we explain our finances, and our past year, and our future plans.  Kevin Walker has a few kind words for me in this video.

The mistake with the cable was forgotten very quickly.  Ten years ago, I wrote a strategic plan for SATV in which I explained that IT would be a very important part of our success in serving the community of Salem.  It was received politely, but it would only be taken seriously by our new, and current executive director, Sal Russo.  When he was hired in 2002, he and I would work to realize many elements of my plan.

Then, we were in the final years of our contract with Comcast and we had limited funds;  we also had to tally up our assets and figure out what needed to be improved.

It was a long list and SATV was a tired place in many ways.  A caretaker director and an unaware board had made life very discouraging;  in 2002 you could be forgiven for thinking our best years were behind us.  Our first executive director had unknowingly contributed to this decline when he was first hired;  in a very technological business like ours, he had no knowledge of technology, not even superficially.

While our first director, Bob Miot, was second to none in making the contacts SATV needed to thrive in its early life, the technology was neglected to our detriment.  Members like myself who had expertise were not really encouraged to be involved in that area.

Very soon after Sal came on board, I got the money to run new phone wiring for DSL.   Six months after that I dusted off my plans for our new network and in early 2004, we had our new Category 6 gigabit network, which has been nearly unchanged to this day.  (In 2006, we renovated our studio space so we performed another round of network improvements.)

More gradually, in recent years, our server technology has been updated;  we had SBS 2000 on our first Dell , followed by SBS 2003 and now, as of a year ago, SBS 2008.  We installed a video server in 2008 and our network is busy with Mac editing workstations and a public WiFi network.

We have modern Dell PC’s running Windows 7, and a new VoIP PBX that replaced a very elderly Panasonic key system.  We are as progressive as our budget will allow us to be.

Gone are the days when I didn’t know something was down until I came in or staff would call.  We have an environmental monitor, our SuperGoose, and numerous notification systems that go out to my cell phone.

We now have a modern IT-centric system that sits in the background and does its work for us, helping our government connect with its community, our community connect with their government and our citizens connecting with each other, over video and now online.

Now, as it was 10 years ago, we are nearing the end of another contract cycle with Comcast.  The monies we get from them will be lesser and our budget tighter.

Now, as then, I have been asked to write another strategic plan for SATV.  Using what we have learned over the years and what I have learned over 10 years, I am once again responsible for helping SATV navigate the next few years in IT.

We’ve seen many changes, too many to list. Ten years ago, I was on a tour of WHDH-TV, Channel 7.  We were shown two large robotic tape machines.

They were each used to cue and play commercials, one acting to backup the other.

It is safe to say those tape machines are gone now.

Five years ago, the first video servers came into use at high-end broadcast facilities.  We’ll not get these for a while, we thought.

Nearly two years ago we got our first video server and it has forced a change in how we deal with video, so much of a change that I have had to dedicate a section of our plan to recommend policies on how long a member’s video needs to be in our servers, for example.

It has changed backup procedures;  five years ago we were on DDS, then DLT tape.  Then external hard drives just as you would find at Best Buy.  How do we backup a terabyte of video?

Can we afford to?  Afford not to?

What more will we see over a few short years?

Whatever happens, I have been given a mandate and an endorsement to continue, perhaps for another ten years.

More errors, more mistakes, but also more triumphs, and successes.

And, one hopes, the continuation of SATV’s mission.

All one can ask for.

UPDATE: Phil Elder has some kind words.

There have been ups and downs over the years at SATV, not a few have been down to my own mistakes.  This is one of them.

This is or was a wiring harness used in our video server.  Because PC cases don’t have a lot of room for connectors, particularly not for BNC video connectors, it is very common in the broadcast and professional audio fields to have cable snakes—a series of BNC or XLR pigtails that go into a DB connector where it can be DB-15 (like the old IBM Joystick connectors) or DB-25’s (serial).

This is one I broke.  I was sliding out the video server for cleaning—I had wanted to check the machines for dust ingestion, which was a problem when we renovated our studio area a few years ago.  The slide rail got stuck and I pulled a little harder.

It snagged the cable without my realizing it and broke off several cables in the snake which we only found out about later on.  Fortunately, our program director, Dave Gauthier, had been able to work around the problem.

I offered to pay for the cable, but was declined.  "You do more good than bad”, I was told.

Another amusing picture:

When we got our new Dell server, it opened, as most Dell tower servers do, with a key inserted into the lock on its bezel.  The power button and case release button and screws are all under the bezel so you must unlock it and take it off from time to time.  This key was in the bezel while it was propped up against a cabinet while I worked on the server.  As you may know from the history of that machine, it had spent a lot of time being open.

I knocked the bezel down and it landed flat on the floor on the “good” side.  The key snapped off in the lock.  Lucky I had another key and a pair of pliers to remove the stub.  Keeping honest people honest is all this does.

We use Spiceworks to manage our IT assets and I’m a regular in the community there.  There was a thread going a little while ago, “What is Your Most Recent IT Screwup?  Be Honest”, and I contributed two posts I will reproduce here:

DCM_SATV

David Moisan

Network/Systems Administrator at Salem Access Television

Salem, MA

My worst mistakes are in the home. You know, you’re more likely to trip on a cat toy and fall down the stairs and die at home, that sort of thing.

I have an SBS box at home with backups. Good thing too. I’ve done these:

1) Changed security inheritance in c:\Windows in such a way that the next step was to get out the restore disk and the backup drive–or break down crying. I no longer change security on Windows directories, ever. (and I did have that backup!)

2) Torn apart my new server motherboard five times on a dead system indication, not realizing the BIOS has a blank screen for the first 30 seconds and there was nothing wrong with the board.

3) In the days of hard sectored floppies, inadvertently inserted a CP/M distribution disk as the "destination" in an old NorthStar machine and realized it only after hearing clickclickclick. (Fortunately I had a copy of that disk itself so I copied it back over…)

3A) Fired up a very, very loud daisywheel printer on that NorthStar for testing–during a staff meeting in that same room! It would have been a good way to drown out boss ranting had I thought of it…

4) Flashed the firmware on a managed switch through the serial port and didn’t take into account that 9600 baud is not fast to transfer an 8 meg image. Switched over to backup switch very very quickly and quietly. Configured TFTP the next day.

5) Rebooted video server in the middle of a program during Patch Tuesday. Embarrassing. We run from the satellite in the mornings for exactly this reason.

6) Ignored a call on the cell during afternoon sleepytime thinking it was a wrong number. Nope. My boss was on the line and he never calls me just because, and it WAS something I needed to have acted on 15 minutes ago.

Jun 10, 2010 at 10:24 AM

DCM_SATV

Oh does that bite! I was on the beta for SBS 2008 and every time a refresh came out I would save a PST as I put all my personal mail on Exchange. I migrated the mailstore twice in the course of the beta.

The RTM comes out and I have a copy.

I say, bleep it, I won’t bother with a PST, I migrated the store twice how hard could it be?

Hahaha

Could not migrate the mailstore from my beta machine for love nor money.

No PST.

I go and buy some program to read OST’s. It doesn’t get all my mail but just enough to triage it and get the important stuff. I try yet another OST reader to get one piece of mail that has an important voucher code in it, the one email I most need.

Ain’t going to do that again…

There you have it!  Those are my mistakes and I don’t expect to stop making any soon, even though one never wakes up in the morning with the intent to screw up.  But, there have been good moments, next.

The end of May marked a milestone quietly passed:  I have been managing the IT at Salem Access Television since the spring of 2000.

I had been at SATV as a member since 1994, and served on its board as one of the three member representatives from 1999 through 2002.   I had learned, on my own and with the help of SATV, video production, editing, and my personal favorite skill, broadcast graphics design.

But I was never involved with our Macs (we were a Mac shop when we opened), nor our Amigas (which we used for TV production and graphics).  I had to be involved with the Amigas out of necessity since our graphics machine was very very flaky.  I had learned AmigaOS, Broadcast Titler, SCALA (which was, and is still, a superb digital presentation/signage program) and the Video Toaster, the legendary video production machine, but this out of necessity.

The executive director at the time was not so much interested in my help, or even aware of it, being very nontechnical and more of a social networker than anything.  I never bothered to be involved.  The only reason I paid attention to computers at all was to deal with the cranky Amigas;  despite the mystique around these nice machines, they were still computers all the same and I applied my computer skills no differently to them than to IBM’s or DEC’s.  There wasn’t anyone else in the building who could help when the Amiga CG went down before a show so…

In 1998 SATV moved to a Windows network and Small Business Server 4.0 (Back Office Server 4.0, Small Business Edition, was the name for it.)  I didn’t know it at the time, but the consultant’s expertise or lack of would make me more involved with our network whether I knew it or not.

When I served on the board in 1999, we were dealing with the financial repercussions of the new network—our director wasn’t really happy with the process nor the consultant and there were some disputed bills.

The staff wasn’t happy either:  Machines would crash and the network would often stall.

We hired a new executive director after our first director, the nontechnical one, felt burned out and wanted to do something else.   After this happened, I spent a little more time on staff machines, essentially diagnosing slowdowns from crapware that people would install and finding out just how flaky the client machines were.  Many of them had fans that failed quickly (a staffer told me once, “the machine sounds like it’s in pain!”) and networking problems that were all too often solved by, removing the NIC and the drivers and reinstalling.

These clients were all Windows 95 machines.  The consultant didn’t want Windows 98, “it was unproven”.  (Given our good experience with Win98 machines later on, I was livid remembering this  but that is off the point…)

We had used a third party NAT program to access our (then dialup) internet connection and it was flaky.  The network continued being intermittent—it was a 10 M shared Ethernet hub.  (Long-time Ethernet experts may know where this is going but hold those thoughts for the moment.)

Sometime in May 2000, I don’t know the exact date, our executive director, Jen Casco, had had enough.  I was babysitting a graphics project, or fixing the CG machine, can’t recall which, when she asked me that afternoon if I would look at the server.  “If you’re comfortable doing that.  It’s driving us crazy.”

Me:  “OK, give me the password…”

It took me two years to get all the problems out and get us a good system.

The network problem, I much later found out, was due to the same junky NICs in the clients;  if you are that Ethernet expert, you probably guessed our problem was due to a duplex mismatch, which can and does affect everything on a hub.

The server had never been backed up.  It was built with a Travan backup drive, an abomination in itself.  It had never seen a service pack.  It had never seen any routine checkups.  So much of my time over that first year was spent learning about our particular network and configuration.

Over that year, these matters were slowly resolved.  NT 4.0 Service Pack 6a was applied and when the upgrade to SBS 4.5 was published, I applied that.  I borrowed tools and parts from home to fix things.  I lent out, for a time, a 10/100 Ethernet switch that I had used to connect computers in my home network.  I even managed to fix the noisy fans by finding surplus CPU cooling units that would fit the (then-slotted) Pentium II processors in the workstations.

Eventually, I had to replace that Frankenstein server our consultant had built, in one final insult.  Over the year that I managed it, it would bluescreen continually.  That’s probably where my interest in Windows internals came from.  It was not a magic fix that I could do, but rather a bad motherboard.

A cheap workstation motherboard. 

Not a server board.

I would be lying if I said I never used a client board as a server, because I have.  But for a paying customer?  This consultant had the brass to suggest he had built and sold us a “Mercedes Network”!

Facepalm.

It was nothing like that.  There’s really no such thing as “Mercedes networks”, I told the board, only larger and smaller, simpler and more complex depending on the needs of the facility.

The network that guy left us wasn’t even that good—just five runs into the Engineering room, where my network core sits now.

(Years later, I would rip out that old network literally with my bare hands, and watch coils of cheap cable fall out of the ceiling, not even secured…)

The end of my year’s work (really year and a half) came when Jen listened to my request to refurbish the server with a new motherboard, and instead authorized the purchase of a new Dell server, the first of three Dell machines we’ve owned.

That is when I knew I had been a success.  (I also knew I loved this sort of work more than I do board work—no disrespect intended!)

Next:  Ups and downs, mistakes and revelations at SATV.

The month of May at SATV has been …

[Darkness]

[BEEEEEEEEEEEEEEEEEEEEP] [Sound of UPS alarms] [Sound of cell phone message notifications]  [Sound of cell phone ring]  [Repeated]

We’ve had several power interruptions in Downtown Salem in the past month, and several others earlier in the year, one of which happened in the middle of an update to the Exchange server.  We lost half a day’s email on that one.

This was one of our live shows:

I was working for that show at the time and I was just about to hit a key on our Inscriber to start the show introduction when the power hit, or didn’t. 

A week later, I was at SATV again around 6 PM, processing the video for the Salem Commission on Disabilities meeting on which I sit.  Darkness.  Alarms.  My cell phone message tone sounded over and over.  I was in a dark room, which was not fun but at least I had the light of my laptop to see by.  This was a longer outage, lasting about 25 minutes.

SATV doesn’t have backup power building-wide.  We have a UPS in the server room, and two more in Cablecasting.  These worked as well as can be expected during a power failure.  We used to have a Comcast-managed demarc equipment room in the furthest corner of our facility, but this has been literally ripped out and replaced by 6U worth of fiber optic equipment in our Cablecast racks.

A notable gap is our phone system;  the PBX itself is protected but the individual phones are not—they stop working during a power incident.  It’s a very sad reminder of the days when all phones and even some PBX’s were provided by Ma Bell and powered by the central office and stayed up no matter what.  I have investigated PoE (Power Over Ethernet) switches—SATV is planning a refresh of our network hardware—but there are few affordable options to us.

We are too small to have generators—we are in leased space and the only place they could go is the roof, if our landlord would even allow it;  as well, it would not be cheap.  (I have never been approached about running a Home Depot generator in the building.  Fortunate, since I’d then have  to explain about enclosed spaces, fire regulations and CO.)

We thought about using a spare UPS we have to power the control room;  Our control room rack power is supplied by two power cords with NEMA 5-20 plugs.  Our UPS doesn’t have compatible sockets, in fact none of ours do.   Unless we want to completely rewire the rack, we can’t use our UPS..

In talking with Sal, we’re reluctant to get higher-capacity UPS units, since the costs rise very, very quickly when you go beyond the usual 1500 VA units, the biggest one can buy for regular office power.   UPS’s are not made to power everything for a long time, but just to carry over brief interruptions and keep the loads up until the generators start and power is switched in.

If our facility disappeared and I was asked to design a new one from scratch, I would have definitely put the server space close to the cablecast space and close to the Comcast & Verizon demarcs.  And configured for 208V three-phase in those areas.  And had a locked door.  And working HVAC.  But very few of us have been fortunate enough to get that blank slate.  Not me, either.

It’s just about impossible to predict what will happen next and how National Grid will deal with it.  Our power situation—Downtown Salem in general—reminds me of Salem 35 years ago when we had a very old water and sewer system.   Then, water mains would rupture seemingly every month.  One May evening in the late-70’s, Salem’s water main downtown let go.

It was in the train tunnel under Riley Plaza.  There was, at one point, several feet of water in spots on street level.  Never mind the tunnel itself, which was completely submerged.  I note with cheer that virtually all of Salem’s telecom cabling ran through this point connecting with the New England Telephone (now Verizon) CO that was and still is just nearby.

DSL service—my DSL service—now runs through that tunnel.  We don’t have a map of Comcast’s cable junctions, but we do know their major cable vault downtown, near Summer & Norman Sts., was prone to flooding, too.

What surprises do our aging electrical systems have for us?  I already know that a car with a stuck accelerator near the Ward Two Social Club (on Bridge St. Neck, five blocks NE of my apartment) can and has dropped power downtown, along with the splintered pole he ran into.

National Grid is frustrating in its lack of communications;  they have a cute Google-based outage map on their webpage that is useful for National Grid itself, but not for its customers or residents.  Why can’t they do what the MBTA does?  I get service notices to my email and cellphone for bus routes I regularly take.  Why couldn’t National Grid have a system that had you put your zipcode and email address or cell number into its system and get notifications of outages or planned maintenance.

For that matter, National Grid already generates internal reports on power incidents.  Why not let us see them in a daily or weekly summary?  (“There was a 2-second drop on one phase at the Canal St. Substation, affecting some customers in South Salem…”)

This is not a problem for SATV to solve as much as it is the city’s.  Downtown Salem loses power several times a year;  we just go off the air, but the many other businesses lose money.  Big money.

Unfortunately, no neighborhood association will ever solicit for improved electrical distribution or refurbished substations.

After 25 years in IT and 15 years working on IBM PC’s, one comes to think he can fix anything.  Sometimes that’s not true.  I have worked on servers and rack-mount machines that are different enough from a stereotypical PC tower, and while I’ve had difficulties unique to each of them, I can manage each of them with patience and meticulousness.

This Inscriber CG machine is something else again.

SATV bought the Inscriber several years ago when we renovated our control room and designed an all-digital (SDI) interface for the video, switcher and graphics units.  As an experienced graphics operator and designer, it was up to me and our executive director to pick out a CG unit.

For reasons too lengthy to get into here, we picked an Inscriber INCACG-2U machine.  It wasn’t cheap;  the base hardware was $6,000 but the license for the features we needed was $20,000!  Not great.  But the studio it is installed in is used for some very high-profile productions in Salem so we needed professional gear (again, long story.)

The Inscriber has been working so far for 5 years.  Then a problem developed:  The Inscriber has two RAID controllers, one is SCSI and drives a 79G RAID 1 array that is the Windows partition, and the other controller, an onboard nVidia SATA interface, drives a 300G RAID 0 array for the content drive, where the graphics files are stored.

One day the Inscriber couldn’t see the RAID 0 array, nor our graphics files.  We’d backed up our CG files regularly, since RAID 0 is not resilient to put it mildly, so nothing was lost, but it was (of course) discovered before a show.  I’d thought one of the SATA drives dropped out but when I rebooted the Inscriber I found something much simpler.

The CMOS battery was dead and so would lose its configuration.  We contributed to the problem, inadvertently, since we shut off the control room equipment when we aren’t using it and the Inscriber is never on standby AC for any time.  So, I think, change the battery.  Every machine in the world seems to use those quarter-sized CR2032’s, so just pop it out and put in the new.

The Inscriber is based on the Tyan 2895 motherboard.  Here’s where the battery is supposed to be:

This is taken from the Tyan motherboard manual.  I’ve rotated the board to match how it is oriented in the case.  Card slots are on the left rear of the machine when looking from the front.  The battery itself is not in the most convenient place to start with;  I and most other techs much prefer it to be in the front where there are fewer components in the way.

This is what I found when I pulled our machine out of the rack and popped the top cover:

This is unreal.  Most PC technicians would run away and not come back.  I’m not sure what our contract engineer, Dennis, would do if he saw this blog.  The SATA RAID 0 array is front and center.  The CPU is under the cover in the middle.  One of the SCSI RAID 1 drives is in the rear—I could not find the second drive!  The right rear holds the dual redundant power supply (which has two plugs that must be connected to AC or else an alarm will sound.  Not that we discovered that.)

The card cage is at the left rear and has the SDI interface card, an ATI video card and a professional audio card.

Under it is our battery.

Try as I might, I could not remove the card cage without a prybar.  Since one can’t experiment like that on a $20K machine, I gave up, only after taking the cover off several times over a few days and trying to find the magic screws that would let me remove the cards without damaging them and change the battery.

Another view of the card cage, with the connectors on the SDI card partially removed.  Notice the ATI card below.  And the hot-melt glue holding what appears to be the ATI’s S-Video output.  When I saw that I just gave up.

I contacted Harris Broadcast, which owns Inscriber.  Their technical support could not give me instructions, or had no instructions for changing the battery.  All they could tell me is what I already knew, that it was based on the Tyan 2895 board.

I understand that a machine like this cannot be built like “Joe’s Whitebox Servers”.  The broadcast market is specialized.  Many broadcast plants are built more like server farms with dual power supplies and the kind of redundance one expects from a machines like these often used for 24/7 broadcast graphics and newsroom operations.

SATV has always been in a hard place where we can’t use consumer gear because it doesn’t hold up or doesn’t do what we want but we can’t afford the broadcast gear that does do what we want.  That is life in our industry.

We will probably work around this for the life of the machine by reconfiguring our power distribution so that the Inscriber is powered from standby at all times.

Still I wish I could change that battery.  I can’t fix everything.

And that hurts.

Over the last week, I have completed my share of maintenance at SATV.  While everyone else was helping to paint the hallways, I was installing our new $300 powerstrip.  Now, in many ways, the real work of 2010 begins.

These projects are looming:

• Completing migration from three-ring reservation binders to our computer-based reservation DB
• Planning our migration from Vista (which will be oh-so-brief at SATV) to Seven, to be done by President’s Day
• Comply with new Massachusetts regulations on personal information by March 1st.
• New firewall/NAT appliance to supersede our cranky cable modem.
• New network switch to replace one in Cablecast which is totally full (8 ports used out of 8).
• Implement an image-based backup scheme for our workstations.
• Figure out how to provide city news and announcements to blind and visually-impaired users.
• Automate and report everything that is not already automatable and reportable.

Amazon.com has a new BFF in me, as seen above!

Except for Windows Internals, I haven’t opened any of those books.

I’m overwhelmed.

While I have loved IT for 30 years, I am mentally exhausted.  I have ADD and have had it my whole life.  Even though I have and use all the organizational coping strategies available (Outlook, Word, my PDA and Spiceworks), I can do 4 hours of mental work on a computer—and feel like I’ve done 20!

The past six months of database migration have killed me.  I’m almost not sure I want to open the books and get going for 2010.