As mentioned in my last post, there are a lot of old, interesting and scary idioms that RSTS/E programmers had to employ if they were to write any kind of game on the system. Of course, the term “graphics” has to be stretched quite a lot with a text-mode VT100, but when you’re in 1981 and can’t afford an Atari, one does what one must.
The screenshot above shows some of the graphic handler routines in Hac-Man. You’ll note this looks somewhat like BASIC, only with some extensions. (This describes virtually every dialect of BASIC with “Extended” in the title…) RSTS/E BASIC has what is known as EXTEND mode. This allows for longer variable names and multiline statements and comments.
I’m using the DEF FN functionality of BASIC to wrap some code that displays VT100 escape sequences to the user’s terminal. FNAD$ is a function that performs direct cursor addressing. It is not obvious from my snippet of code, but RSTS/E is a 7-bit ASCII system and one had to do some tricks to get the escape code to pass to the terminal without the system intercepting it. Earlier on, there is a line in my code that defines the escape character (which would normally be decimal 27).
910 ESCH$=CHR$(155) ! Default escape character
Code 155 is simply code 27 with the high bit set. RSTS ignores it and it is echoed back to the terminal.
To make this game work, we also need to turn on the alternate character set. That’s in the function FNLD$. Here’s what the program listing looks like normally:
The GFL$ array holds our playfield, which I will describe in a later post. Doesn’t look like anything.
Here is the listing in special character mode:
The playfield is being murdered by line wrap, but you can see blocks and dots there.
An aside on editing: RSTS/E had an older revision of TECO, but with screen editor capabilities via the TECO program VTEDIT. (EDT also had a screen editor, but back in the day I much preferred VTEDIT; also, today, EDT’s screen mode doesn’t seem to work in Tera Term.) Such an editor was a virtual necessity when developing in any language on the 11/60. This playfield was first composed in TECO as a plain text file, and then copied, and edited again to add code around the text.
The next few posts will explain more about my graphics and my input logic. Then, I’ll be almost ready to move Hac-Man and his ghosts around the screen!
As I write this, it’s been a year and a quarter since SATV hatched our idea for internet broadcasting. We still had a few things to pick up.
Over the spring, I bought a GPS module for myself, for experimentation. With some open-source software, I found that it would make an excellent, cheap time reference. For less than $200, we could keep our facility timekeeping—our on-air timekeeping—to within 1/1000’s of one second. Sal agreed, and we got a GPS module of our own. I’m planning a guide on cheap GPS timing in a future series of posts.
We’ve long been worried about our server storage. We now have several terabytes of video at SATV. We need a backup and data recovery plan for off-site. While we figure out that, we got our first SAN, a Drobo B800i. It’s loaded up with 14 TB worth of storage. We plan on using it to backup both our video servers, plus our SBS machine and the backup DC it is paired with.
I’m well aware that we need something more for offsite. I plan on talking with our city’s IT director (who, in an irony, was once my boss—I interned for her the summer after I graduated from Salem State College (now University)) to see what resources she may be able to bear on our problem. She was the one who first asked about airing government meetings over the internet!
As a member of the Salem Commission on Disabilities, I’ve known for a long time how important TV coverage is to us as we conduct our business. Government transparency has been cited so often that the expression is clichéd now. But I have seen first hand how important it is and when Sal wanted to implement video-on-demand I jumped in with both hands and feet.
This whole project has been a labor of love for me, and while I want to give Sal Russo the credit for our overall leadership, I’ve worked in the background to give us the IT experiences and resources that we all accept without a second’s thought. Now, a thought about the future.
SATV receives its funding through a contract with our cable provider, Comcast, and a franchise agreement that our city has negotiated every 10 years for nearly 30 years. In Massachusetts, at any given time, one of our neighboring communities will either be going through the franchise negotiations, completing the franchise negotiations or just starting discussion.
When I started this project a year and a quarter ago, Sal and I had been discussing the last of our payments from Comcast in our current contract period, and we decided on the strategy and spending that I’ve just described in the past three posts.
We are about to think about our future as an organization. It’ll be a challenging and intensive process; I was involved in our planning for the current agreement and it was a stressful time. With our economic and political situation, I have not enjoyed thinking much about the future.
But it is here nonetheless.
I’m already planning replacements for several big-budget items, including our impossible-to-fix Inscriber. I am well aware of Microsoft’s roadmap, and the fact that Windows XP will be completely out of support in 2014. Not long ago, we needed two huge road cases to broadcast in the field. They’re obsolete now. We’ are thinking of a suitcase and a laptop. If that. Years ago I could not hope to have my own personal camcorder, but today I have a camera I can hold in my hand that shoots excellent HD video. It cost me just $100.
Those are the realities that SATV and I have to adjust to. I normally love working with technology—it really has been a labor of love for me—but it could be difficult and dispiriting for a while.
But we can’t argue with the results so far. I hope we can be just as relevant tomorrow as we are today.
In my last post I talked about the new video-on-demand system we got at SATV. Recapping, this is a rack-mount Windows Server 2008R2 system that runs IIS and Windows Media Services in a custom system that will handle all the programming playback and ingest needs for a small public-access cable facility. It also transcodes programming for on-demand playback through the built in WMS.
This past May we introduced this capability to the community of Salem, Massachusetts. How was it received?
It was all WIN!
This past spring, Salem lost our former mayor Sam Zoll. He was only our mayor for a few years in the early 1970’s before being appointed to the bench, but he was a much-beloved individual with the foresight to stage-manage his own memorial service. It was filmed and broadcast on SATV, but on a Friday afternoon not long afterwards, we had our VOD server transcode the file and we published the link on our web site.
In the screenshot above is a capture of my internal reporting tool (using PowerShell and Log Analyzer) that runs once a month and gives us stats on what people are watching. It shows that Zoll’s memorial service was accessed over 270 times, and had a total viewing time of over 26 hours!
It was only overshadowed by Salem Now, which featured a very popular Salem band, The Extras Band, giving an excellent performance that was easily in the realm of American Bandstand of my youth. That show had over 500 accesses!
In a time where organizations like SATV’s must take advantage of new social media, our new system was very fortuitous indeed!
Now that we had the system working, I needed to make it as reliable as could be. We had been having glitches with the network in our cablecast plant for quite some time. We had an 8-port gigabit switch and an 8-port Cat 6 patch panel, both of which were overloaded.
We purchased a Dell 2824 24 port managed gigabit switch to replace the old switch, but I found a problem with our patch panel:
When I first designed and built our Cat 6 network a few years back, I used a Leviton 12-port panel in this and a few other rooms in the facility. This panel was designed to be oriented just as you see it. Note that the RJ45 tab is on the side as are the contacts. Note the broken contacts on jack #6.
The jack was stressed and was destined to fail. $150 later—and a reorientation—that problem was fixed:
And this is our network switch:
We still weren’t done. Following a very visible power failure in our facility last year, Sal and I were determined to not have that happen again, or at least, we wanted to ride out any power transients in a much smoother way than before.
We purchased two Eaton 5130 1500 VA UPS units, each with an SNMP/web network card and an external battery. We can access these directly from the network without needing a PC to control them. We use Spiceworks to manage our IT and that app has UPS support for our units. A happy day!
We redistributed our older Powerware 5125 units to power our server, and perhaps more importantly, part of our control room from which our studio shows are produced. We have a total of 4 managed Eaton/Powerware units, 1 unmanaged APC and one unmanaged TrippLite. During a summer in downtown Salem that saw National Grid trucks on the street every day, and a 5-hour blackout at my own apartment building one Sunday afternoon, this new hardware is most welcome.
We were finally getting closer to perfection.
In my final post, I talk about the last few things we’ve bought, but more importantly. I talk about our future and what it means.
I haven’t posted for a long time. During the past year at SATV, we have made a number of improvements to our IT that have given us new capabilities to serve the city of Salem and its communities.
Earlier this year, we got a new video server from Tightrope Media Systems; We have been running their media system for years and have been running their SX-4 video server for several years. It has two channels of encoder input and four channels of encoder output; three of the four channels normally feed each of our three cable channels, 3 for public access, 15 for education and 16 for government. The extra playback channel is used as a spare. The two input encoders are used to capture analog video for playback.
Our plant was originally built to ingest (input for broadcasting) VHS, DV and DVD video content, but after several years almost all of our on-air content comes from MPEG-2 files that are served by our SX-4. DVD’s are transcoded and demultiplexed with a utility provided by TRMS, so our DVD players have been getting less and less use.
Over the past few years, the Internet has continued to encompass more and more everyday activities as time goes on. Many public access TV facilities have been offering their content online. Also, the City of Salem has wanted and needed to have government meeting coverage available online and on-demand.
Last year, I worked with Sal Russo and the staff to plan the implementation of a VOD (video-on-demand) system at SATV. The new VOD server would require a number of changes to our network configuration and an upgrade to our level of service from Comcast, from which we purchase Internet and phone service.
The first challenge I needed to confront was our firewall. Once upon a time we had Microsoft SBS 2003 which included ISA Server. It was an excellent firewall and I still miss it. However, most of the SBS community didn’t share my enthusiasm for this product so Microsoft elected to remove ISA. After we migrated to SBS 2008, we had no firewall. We were left with the NAT features that were in our cable modem, an SMC 8014 used by Comcast.
This was barely tolerable at the time we installed it because we did not have very many incoming connections; our schedule web page accessed our SX-4 directly and that was it, not counting the occasional VPN connection from home. A VOD service would make us serve a lot more connections and much more traffic. That was why we needed to move up to (and pay for) the next tier of service from Comcast.
And we needed a firewall. I chose a Zyxel USG firewall appliance.
What I liked about this firewall were the seven ports that could be configured to be in multiple zones, with forwarding rules for each. I used this to put our public WiFi access points on their own separate network, routed via a VLAN on our core managed switch. This was a very nice side benefit that worked great.
This past February we got the VOD installed, and about six weeks later, we started playing programming on-demand over the internet.
How did it go?
That’s for my next installment.
This past spring, we had our annual rite, the Annual Meeting. This is a board function required by law where we explain our finances, and our past year, and our future plans. Kevin Walker has a few kind words for me in this video.
The mistake with the cable was forgotten very quickly. Ten years ago, I wrote a strategic plan for SATV in which I explained that IT would be a very important part of our success in serving the community of Salem. It was received politely, but it would only be taken seriously by our new, and current executive director, Sal Russo. When he was hired in 2002, he and I would work to realize many elements of my plan.
Then, we were in the final years of our contract with Comcast and we had limited funds; we also had to tally up our assets and figure out what needed to be improved.
It was a long list and SATV was a tired place in many ways. A caretaker director and an unaware board had made life very discouraging; in 2002 you could be forgiven for thinking our best years were behind us. Our first executive director had unknowingly contributed to this decline when he was first hired; in a very technological business like ours, he had no knowledge of technology, not even superficially.
While our first director, Bob Miot, was second to none in making the contacts SATV needed to thrive in its early life, the technology was neglected to our detriment. Members like myself who had expertise were not really encouraged to be involved in that area.
Very soon after Sal came on board, I got the money to run new phone wiring for DSL. Six months after that I dusted off my plans for our new network and in early 2004, we had our new Category 6 gigabit network, which has been nearly unchanged to this day. (In 2006, we renovated our studio space so we performed another round of network improvements.)
More gradually, in recent years, our server technology has been updated; we had SBS 2000 on our first Dell , followed by SBS 2003 and now, as of a year ago, SBS 2008. We installed a video server in 2008 and our network is busy with Mac editing workstations and a public WiFi network.
We have modern Dell PC’s running Windows 7, and a new VoIP PBX that replaced a very elderly Panasonic key system. We are as progressive as our budget will allow us to be.
Gone are the days when I didn’t know something was down until I came in or staff would call. We have an environmental monitor, our SuperGoose, and numerous notification systems that go out to my cell phone.
We now have a modern IT-centric system that sits in the background and does its work for us, helping our government connect with its community, our community connect with their government and our citizens connecting with each other, over video and now online.
Now, as it was 10 years ago, we are nearing the end of another contract cycle with Comcast. The monies we get from them will be lesser and our budget tighter.
Now, as then, I have been asked to write another strategic plan for SATV. Using what we have learned over the years and what I have learned over 10 years, I am once again responsible for helping SATV navigate the next few years in IT.
We’ve seen many changes, too many to list. Ten years ago, I was on a tour of WHDH-TV, Channel 7. We were shown two large robotic tape machines.
They were each used to cue and play commercials, one acting to backup the other.
It is safe to say those tape machines are gone now.
Five years ago, the first video servers came into use at high-end broadcast facilities. We’ll not get these for a while, we thought.
Nearly two years ago we got our first video server and it has forced a change in how we deal with video, so much of a change that I have had to dedicate a section of our plan to recommend policies on how long a member’s video needs to be in our servers, for example.
It has changed backup procedures; five years ago we were on DDS, then DLT tape. Then external hard drives just as you would find at Best Buy. How do we backup a terabyte of video?
Can we afford to? Afford not to?
What more will we see over a few short years?
Whatever happens, I have been given a mandate and an endorsement to continue, perhaps for another ten years.
More errors, more mistakes, but also more triumphs, and successes.
And, one hopes, the continuation of SATV’s mission.
All one can ask for.
UPDATE: Phil Elder has some kind words.
There have been ups and downs over the years at SATV, not a few have been down to my own mistakes. This is one of them.
This is or was a wiring harness used in our video server. Because PC cases don’t have a lot of room for connectors, particularly not for BNC video connectors, it is very common in the broadcast and professional audio fields to have cable snakes—a series of BNC or XLR pigtails that go into a DB connector where it can be DB-15 (like the old IBM Joystick connectors) or DB-25’s (serial).
This is one I broke. I was sliding out the video server for cleaning—I had wanted to check the machines for dust ingestion, which was a problem when we renovated our studio area a few years ago. The slide rail got stuck and I pulled a little harder.
It snagged the cable without my realizing it and broke off several cables in the snake which we only found out about later on. Fortunately, our program director, Dave Gauthier, had been able to work around the problem.
I offered to pay for the cable, but was declined. "You do more good than bad”, I was told.
Another amusing picture:
When we got our new Dell server, it opened, as most Dell tower servers do, with a key inserted into the lock on its bezel. The power button and case release button and screws are all under the bezel so you must unlock it and take it off from time to time. This key was in the bezel while it was propped up against a cabinet while I worked on the server. As you may know from the history of that machine, it had spent a lot of time being open.
I knocked the bezel down and it landed flat on the floor on the “good” side. The key snapped off in the lock. Lucky I had another key and a pair of pliers to remove the stub. Keeping honest people honest is all this does.
We use Spiceworks to manage our IT assets and I’m a regular in the community there. There was a thread going a little while ago, “What is Your Most Recent IT Screwup? Be Honest”, and I contributed two posts I will reproduce here:
Network/Systems Administrator at Salem Access Television
My worst mistakes are in the home. You know, you’re more likely to trip on a cat toy and fall down the stairs and die at home, that sort of thing.
I have an SBS box at home with backups. Good thing too. I’ve done these:
1) Changed security inheritance in c:\Windows in such a way that the next step was to get out the restore disk and the backup drive–or break down crying. I no longer change security on Windows directories, ever. (and I did have that backup!)
2) Torn apart my new server motherboard five times on a dead system indication, not realizing the BIOS has a blank screen for the first 30 seconds and there was nothing wrong with the board.
3) In the days of hard sectored floppies, inadvertently inserted a CP/M distribution disk as the "destination" in an old NorthStar machine and realized it only after hearing clickclickclick. (Fortunately I had a copy of that disk itself so I copied it back over…)
3A) Fired up a very, very loud daisywheel printer on that NorthStar for testing–during a staff meeting in that same room! It would have been a good way to drown out boss ranting had I thought of it…
4) Flashed the firmware on a managed switch through the serial port and didn’t take into account that 9600 baud is not fast to transfer an 8 meg image. Switched over to backup switch very very quickly and quietly. Configured TFTP the next day.
5) Rebooted video server in the middle of a program during Patch Tuesday. Embarrassing. We run from the satellite in the mornings for exactly this reason.
6) Ignored a call on the cell during afternoon sleepytime thinking it was a wrong number. Nope. My boss was on the line and he never calls me just because, and it WAS something I needed to have acted on 15 minutes ago.
Oh does that bite! I was on the beta for SBS 2008 and every time a refresh came out I would save a PST as I put all my personal mail on Exchange. I migrated the mailstore twice in the course of the beta.
The RTM comes out and I have a copy.
I say, bleep it, I won’t bother with a PST, I migrated the store twice how hard could it be?
Could not migrate the mailstore from my beta machine for love nor money.
I go and buy some program to read OST’s. It doesn’t get all my mail but just enough to triage it and get the important stuff. I try yet another OST reader to get one piece of mail that has an important voucher code in it, the one email I most need.
Ain’t going to do that again…
There you have it! Those are my mistakes and I don’t expect to stop making any soon, even though one never wakes up in the morning with the intent to screw up. But, there have been good moments, next.
The end of May marked a milestone quietly passed: I have been managing the IT at Salem Access Television since the spring of 2000.
I had been at SATV as a member since 1994, and served on its board as one of the three member representatives from 1999 through 2002. I had learned, on my own and with the help of SATV, video production, editing, and my personal favorite skill, broadcast graphics design.
But I was never involved with our Macs (we were a Mac shop when we opened), nor our Amigas (which we used for TV production and graphics). I had to be involved with the Amigas out of necessity since our graphics machine was very very flaky. I had learned AmigaOS, Broadcast Titler, SCALA (which was, and is still, a superb digital presentation/signage program) and the Video Toaster, the legendary video production machine, but this out of necessity.
The executive director at the time was not so much interested in my help, or even aware of it, being very nontechnical and more of a social networker than anything. I never bothered to be involved. The only reason I paid attention to computers at all was to deal with the cranky Amigas; despite the mystique around these nice machines, they were still computers all the same and I applied my computer skills no differently to them than to IBM’s or DEC’s. There wasn’t anyone else in the building who could help when the Amiga CG went down before a show so…
In 1998 SATV moved to a Windows network and Small Business Server 4.0 (Back Office Server 4.0, Small Business Edition, was the name for it.) I didn’t know it at the time, but the consultant’s expertise or lack of would make me more involved with our network whether I knew it or not.
When I served on the board in 1999, we were dealing with the financial repercussions of the new network—our director wasn’t really happy with the process nor the consultant and there were some disputed bills.
The staff wasn’t happy either: Machines would crash and the network would often stall.
We hired a new executive director after our first director, the nontechnical one, felt burned out and wanted to do something else. After this happened, I spent a little more time on staff machines, essentially diagnosing slowdowns from crapware that people would install and finding out just how flaky the client machines were. Many of them had fans that failed quickly (a staffer told me once, “the machine sounds like it’s in pain!”) and networking problems that were all too often solved by, removing the NIC and the drivers and reinstalling.
These clients were all Windows 95 machines. The consultant didn’t want Windows 98, “it was unproven”. (Given our good experience with Win98 machines later on, I was livid remembering this but that is off the point…)
We had used a third party NAT program to access our (then dialup) internet connection and it was flaky. The network continued being intermittent—it was a 10 M shared Ethernet hub. (Long-time Ethernet experts may know where this is going but hold those thoughts for the moment.)
Sometime in May 2000, I don’t know the exact date, our executive director, Jen Casco, had had enough. I was babysitting a graphics project, or fixing the CG machine, can’t recall which, when she asked me that afternoon if I would look at the server. “If you’re comfortable doing that. It’s driving us crazy.”
Me: “OK, give me the password…”
It took me two years to get all the problems out and get us a good system.
The network problem, I much later found out, was due to the same junky NICs in the clients; if you are that Ethernet expert, you probably guessed our problem was due to a duplex mismatch, which can and does affect everything on a hub.
The server had never been backed up. It was built with a Travan backup drive, an abomination in itself. It had never seen a service pack. It had never seen any routine checkups. So much of my time over that first year was spent learning about our particular network and configuration.
Over that year, these matters were slowly resolved. NT 4.0 Service Pack 6a was applied and when the upgrade to SBS 4.5 was published, I applied that. I borrowed tools and parts from home to fix things. I lent out, for a time, a 10/100 Ethernet switch that I had used to connect computers in my home network. I even managed to fix the noisy fans by finding surplus CPU cooling units that would fit the (then-slotted) Pentium II processors in the workstations.
Eventually, I had to replace that Frankenstein server our consultant had built, in one final insult. Over the year that I managed it, it would bluescreen continually. That’s probably where my interest in Windows internals came from. It was not a magic fix that I could do, but rather a bad motherboard.
A cheap workstation motherboard.
Not a server board.
I would be lying if I said I never used a client board as a server, because I have. But for a paying customer? This consultant had the brass to suggest he had built and sold us a “Mercedes Network”!
It was nothing like that. There’s really no such thing as “Mercedes networks”, I told the board, only larger and smaller, simpler and more complex depending on the needs of the facility.
The network that guy left us wasn’t even that good—just five runs into the Engineering room, where my network core sits now.
(Years later, I would rip out that old network literally with my bare hands, and watch coils of cheap cable fall out of the ceiling, not even secured…)
The end of my year’s work (really year and a half) came when Jen listened to my request to refurbish the server with a new motherboard, and instead authorized the purchase of a new Dell server, the first of three Dell machines we’ve owned.
That is when I knew I had been a success. (I also knew I loved this sort of work more than I do board work—no disrespect intended!)
The month of May at SATV has been …
[BEEEEEEEEEEEEEEEEEEEEP] [Sound of UPS alarms] [Sound of cell phone message notifications] [Sound of cell phone ring] [Repeated]
We’ve had several power interruptions in Downtown Salem in the past month, and several others earlier in the year, one of which happened in the middle of an update to the Exchange server. We lost half a day’s email on that one.
This was one of our live shows:
I was working for that show at the time and I was just about to hit a key on our Inscriber to start the show introduction when the power hit, or didn’t.
A week later, I was at SATV again around 6 PM, processing the video for the Salem Commission on Disabilities meeting on which I sit. Darkness. Alarms. My cell phone message tone sounded over and over. I was in a dark room, which was not fun but at least I had the light of my laptop to see by. This was a longer outage, lasting about 25 minutes.
SATV doesn’t have backup power building-wide. We have a UPS in the server room, and two more in Cablecasting. These worked as well as can be expected during a power failure. We used to have a Comcast-managed demarc equipment room in the furthest corner of our facility, but this has been literally ripped out and replaced by 6U worth of fiber optic equipment in our Cablecast racks.
A notable gap is our phone system; the PBX itself is protected but the individual phones are not—they stop working during a power incident. It’s a very sad reminder of the days when all phones and even some PBX’s were provided by Ma Bell and powered by the central office and stayed up no matter what. I have investigated PoE (Power Over Ethernet) switches—SATV is planning a refresh of our network hardware—but there are few affordable options to us.
We are too small to have generators—we are in leased space and the only place they could go is the roof, if our landlord would even allow it; as well, it would not be cheap. (I have never been approached about running a Home Depot generator in the building. Fortunate, since I’d then have to explain about enclosed spaces, fire regulations and CO.)
We thought about using a spare UPS we have to power the control room; Our control room rack power is supplied by two power cords with NEMA 5-20 plugs. Our UPS doesn’t have compatible sockets, in fact none of ours do. Unless we want to completely rewire the rack, we can’t use our UPS..
In talking with Sal, we’re reluctant to get higher-capacity UPS units, since the costs rise very, very quickly when you go beyond the usual 1500 VA units, the biggest one can buy for regular office power. UPS’s are not made to power everything for a long time, but just to carry over brief interruptions and keep the loads up until the generators start and power is switched in.
If our facility disappeared and I was asked to design a new one from scratch, I would have definitely put the server space close to the cablecast space and close to the Comcast & Verizon demarcs. And configured for 208V three-phase in those areas. And had a locked door. And working HVAC. But very few of us have been fortunate enough to get that blank slate. Not me, either.
It’s just about impossible to predict what will happen next and how National Grid will deal with it. Our power situation—Downtown Salem in general—reminds me of Salem 35 years ago when we had a very old water and sewer system. Then, water mains would rupture seemingly every month. One May evening in the late-70’s, Salem’s water main downtown let go.
It was in the train tunnel under Riley Plaza. There was, at one point, several feet of water in spots on street level. Never mind the tunnel itself, which was completely submerged. I note with cheer that virtually all of Salem’s telecom cabling ran through this point connecting with the New England Telephone (now Verizon) CO that was and still is just nearby.
DSL service—my DSL service—now runs through that tunnel. We don’t have a map of Comcast’s cable junctions, but we do know their major cable vault downtown, near Summer & Norman Sts., was prone to flooding, too.
What surprises do our aging electrical systems have for us? I already know that a car with a stuck accelerator near the Ward Two Social Club (on Bridge St. Neck, five blocks NE of my apartment) can and has dropped power downtown, along with the splintered pole he ran into.
National Grid is frustrating in its lack of communications; they have a cute Google-based outage map on their webpage that is useful for National Grid itself, but not for its customers or residents. Why can’t they do what the MBTA does? I get service notices to my email and cellphone for bus routes I regularly take. Why couldn’t National Grid have a system that had you put your zipcode and email address or cell number into its system and get notifications of outages or planned maintenance.
For that matter, National Grid already generates internal reports on power incidents. Why not let us see them in a daily or weekly summary? (“There was a 2-second drop on one phase at the Canal St. Substation, affecting some customers in South Salem…”)
This is not a problem for SATV to solve as much as it is the city’s. Downtown Salem loses power several times a year; we just go off the air, but the many other businesses lose money. Big money.
Unfortunately, no neighborhood association will ever solicit for improved electrical distribution or refurbished substations.
After 25 years in IT and 15 years working on IBM PC’s, one comes to think he can fix anything. Sometimes that’s not true. I have worked on servers and rack-mount machines that are different enough from a stereotypical PC tower, and while I’ve had difficulties unique to each of them, I can manage each of them with patience and meticulousness.
This Inscriber CG machine is something else again.
SATV bought the Inscriber several years ago when we renovated our control room and designed an all-digital (SDI) interface for the video, switcher and graphics units. As an experienced graphics operator and designer, it was up to me and our executive director to pick out a CG unit.
For reasons too lengthy to get into here, we picked an Inscriber INCACG-2U machine. It wasn’t cheap; the base hardware was $6,000 but the license for the features we needed was $20,000! Not great. But the studio it is installed in is used for some very high-profile productions in Salem so we needed professional gear (again, long story.)
The Inscriber has been working so far for 5 years. Then a problem developed: The Inscriber has two RAID controllers, one is SCSI and drives a 79G RAID 1 array that is the Windows partition, and the other controller, an onboard nVidia SATA interface, drives a 300G RAID 0 array for the content drive, where the graphics files are stored.
One day the Inscriber couldn’t see the RAID 0 array, nor our graphics files. We’d backed up our CG files regularly, since RAID 0 is not resilient to put it mildly, so nothing was lost, but it was (of course) discovered before a show. I’d thought one of the SATA drives dropped out but when I rebooted the Inscriber I found something much simpler.
The CMOS battery was dead and so would lose its configuration. We contributed to the problem, inadvertently, since we shut off the control room equipment when we aren’t using it and the Inscriber is never on standby AC for any time. So, I think, change the battery. Every machine in the world seems to use those quarter-sized CR2032’s, so just pop it out and put in the new.
The Inscriber is based on the Tyan 2895 motherboard. Here’s where the battery is supposed to be:
This is taken from the Tyan motherboard manual. I’ve rotated the board to match how it is oriented in the case. Card slots are on the left rear of the machine when looking from the front. The battery itself is not in the most convenient place to start with; I and most other techs much prefer it to be in the front where there are fewer components in the way.
This is what I found when I pulled our machine out of the rack and popped the top cover:
This is unreal. Most PC technicians would run away and not come back. I’m not sure what our contract engineer, Dennis, would do if he saw this blog. The SATA RAID 0 array is front and center. The CPU is under the cover in the middle. One of the SCSI RAID 1 drives is in the rear—I could not find the second drive! The right rear holds the dual redundant power supply (which has two plugs that must be connected to AC or else an alarm will sound. Not that we discovered that.)
The card cage is at the left rear and has the SDI interface card, an ATI video card and a professional audio card.
Under it is our battery.
Try as I might, I could not remove the card cage without a prybar. Since one can’t experiment like that on a $20K machine, I gave up, only after taking the cover off several times over a few days and trying to find the magic screws that would let me remove the cards without damaging them and change the battery.
Another view of the card cage, with the connectors on the SDI card partially removed. Notice the ATI card below. And the hot-melt glue holding what appears to be the ATI’s S-Video output. When I saw that I just gave up.
I contacted Harris Broadcast, which owns Inscriber. Their technical support could not give me instructions, or had no instructions for changing the battery. All they could tell me is what I already knew, that it was based on the Tyan 2895 board.
I understand that a machine like this cannot be built like “Joe’s Whitebox Servers”. The broadcast market is specialized. Many broadcast plants are built more like server farms with dual power supplies and the kind of redundance one expects from a machines like these often used for 24/7 broadcast graphics and newsroom operations.
SATV has always been in a hard place where we can’t use consumer gear because it doesn’t hold up or doesn’t do what we want but we can’t afford the broadcast gear that does do what we want. That is life in our industry.
We will probably work around this for the life of the machine by reconfiguring our power distribution so that the Inscriber is powered from standby at all times.
Still I wish I could change that battery. I can’t fix everything.
And that hurts.
Over the last week, I have completed my share of maintenance at SATV. While everyone else was helping to paint the hallways, I was installing our new $300 powerstrip. Now, in many ways, the real work of 2010 begins.
These projects are looming:
- Completing migration from three-ring reservation binders to our computer-based reservation DB
- Planning our migration from Vista (which will be oh-so-brief at SATV) to Seven, to be done by President’s Day
- Comply with new Massachusetts regulations on personal information by March 1st.
- New firewall/NAT appliance to supersede our cranky cable modem.
- New network switch to replace one in Cablecast which is totally full (8 ports used out of 8).
- Implement an image-based backup scheme for our workstations.
- Figure out how to provide city news and announcements to blind and visually-impaired users.
- Automate and report everything that is not already automatable and reportable.
Amazon.com has a new BFF in me, as seen above!
Except for Windows Internals, I haven’t opened any of those books.
While I have loved IT for 30 years, I am mentally exhausted. I have ADD and have had it my whole life. Even though I have and use all the organizational coping strategies available (Outlook, Word, my PDA and Spiceworks), I can do 4 hours of mental work on a computer—and feel like I’ve done 20!
The past six months of database migration have killed me. I’m almost not sure I want to open the books and get going for 2010.