The All Nighter

Posted on by

The lab was dark except for a pool of light around a blue and white workbench, the kind with an instrument shelf toward the back and 110-volt outlets along the edges.

The rest of the lab was buzzing as various electronics and in-development projects sang songs of cooling fans, blinking their LEDs to the rhythm of an unseen conductor.

In the darkness, it was hard to make out the 7-foot-high racks, similar to the racks of today but a bit wider as was the fashion of telephone companies in the mid–80s. Each rack held some massive technology being produced by the small Georgia company I worked for at the time. The collection of 23" wide metal chassis of various heights all sat open faced, their vertical cards and colorful LEDs exposed.

Collectively the equipment represented a Digital Electronic Cross-Connect Switch, a magic system able to juggle over 170,000 phone calls. It was the first DS3/DS1 electronic digital cross-connect switch in the world.

I sat on a stool in the only lit corner. In front of me was a new Wyse WY–60 terminal, which could display 24 lines of 80 green characters and had a separate keyboard. A listing of my system, bound in plastic comb, lay open on the bench. It was 3/4" thick, and represented a recent version of the control system for the cross-connect switch.

Planted on the stool next to me was Jim Kemp, director of R&D and the man who’d conducted my interview and offered me the gig. That night he served as a living requirements document.

The company designed a large cross-connect switch with custom semiconductors, a massive switching backplane, and five racks of DS3 and DS1 interface shelves. In the center of a control cable nexus was a computer system on which they planned to run Unix and eventually add custom control software.

Not that they knew anything about Unix. All they really knew, or thought they knew, was that you could click together small programs with things called “pipes.” I think Jim knew what he didn’t know and was looking to hire the right person to work it all out.

As Jim began his search for a new employee, I was working for a zipper business turned dictaphone company turned who-knows-what, called Harris-Lanier. In the hodgepodge of Lanier Business Products, there happened to be special purpose computer software called a “word processor.” Before Microsoft Word there were typewriters. The bridge between those two was equipment dedicated to word processing, equipment that I worked on.

I joined Lanier because they were known to experiment with Unix and had already hired some friends. Things didn’t work out, and after a year several of us looked for another gig. When my buddy turned an offer down, the company asked if he knew anyone who was looking. “Sure,” he said. “Brantley.”

Soon I was at Digital Transmission Systems, sitting in Jim Kemp’s office in front of a 16”x11” diagram of the massive system drawn on the new Macintosh computers used by the engineering department. In the center was a 4’ by 24”, 18 layer switch matrix backplane that boggled the mind. It was as big as a door and thick as a plank. Later the manufacturer would send along a photo of the massive thing supporting the weight of a proud designer. It was impressive.

There were seventeen switch cards in the matrix, each with sixteen DTS switching integrated circuits. These chips, designed by the same engineer who designed the backplane, could grab any of the 127 DS3s. Each of those could run data at 45 M-bits and pull any of the 28 DS1s in a DS3 out and recombine them back into a DS3 output.

Like all phone company equipment, the DXC had to be redundant and fail-safe but also needed to be quick to fix. There were no active components (transistors) on anything that couldn’t slide into the front of the chassis. Which is why there were seventeen matrix cards. Relays could switch out any single bad one and replace it with the seventeenth standby card. This is called 1-N protection, N being 16 in this case.

In the bottom of the first rack was the computer chassis. That’s what I’d worked on for the past year at Digital Transmission Systems. In it was a VME backplane and several Motorola VME boards. Instead of a motherboard with expansion slots, this system used a passive backplane with a CPU card, some cards for main memory, interface cards, and a separate chassis for disks. It was the kind of equipment that Sun Microsystems used to get their start.

This CPU card was the MVME134, a 68020 processor, a 68851 memory management unit, and a gigantic four megabytes of dynamic ram! Only a few years before, a $100,000 DEC VAX, a minicomputer the size of two refrigerators, maxed out at eight megabytes. Moore’s Law was in full swing in the mid–80s.

I ported Unix to that system. Using some compiler work from MIT and my Seventh Edition source code, I wrote the assembler parts, memory management code, and device drivers. The system was running in short order. We did that a lot in those days.

At first I used a Motorola tower computer running their version of Unix, but I quickly moved all my development, lock stock and barrel, to the switch itself. Usually, I just had a small VME chassis with all the same cards that went into the product and ran the system on that. But then I was tagged to do the control software as well.

You may be wondering what Jim and I were doing in a gloomy lab so late at night. We were adding the failover for the matrix which sensed a failed switching card and brought in the seventeenth card to restore phone traffic. The work was urgent.

Like most projects, it took longer and cost more than the company had originally expected. The sales of their T1/E1 monitoring product helped, but money was tight for the small company.

A few days before, the middle-aged, slightly heavy CFO, caught sight of a Georgia Power truck pulling around the building. He dropped his BIC accountant fine point and made a hasty dash for the door to intercept the lineman. After some discussion, the truck left, with the power to the building still flowing. Juggling which bills get paid when is a time-honored tradition in the land of companies building new technologies.

Jim and I knew that the next day the first customer for this switch, a railroad company that had buried fiber optics over their right of ways, would arrive with a check. There was a catch: he would only give us the check after he’d seen the traffic tolerate the failover of a matrix card.

We promised that it would and worked on it diligently. I logged 4,000 hours in the lab that year, and that didn’t include the 70-minute commute each way. I don’t recommend it, and I would never keep such an extreme schedule again. By some miracle, I’m lucky enough to still be married to the same wonderful woman that put up with me back then.

Even with all our hard work, time got the best of us. I was learning about the realities of software development. I’d yet to figure out that people couldn’t tightly plan software. All you can do is pad the hell out of it and hope you’d only have to redo the timeline once or twice as the project continued.

It was new to the management of the company too. They developed electronics, which had a design phase too, but that part of the schedule was small compared with the hours of schematic capture, PCB layout, debugging, acceptance testing, and the rest. Electronics are a carefully orchestrated activity, and they had it down to a science. It wasn’t the first time a hardware company had to learn that software is harder to pin down.

So there we were, pulling an all-nighter.

On my Wyse terminal, I typed away using the “ed” line editor. I would add some code, compile, and test. When I hit a design choice about how the DXC would function, I would ask Jim how he wanted it to work. He would stop and think. I could tell he was picturing the operation of the switch, every gate and amp, every relay and circuit. He would ponder for maybe 30 seconds, sometimes more, and then give me an answer. It always made perfect sense. We’d both spent a lot of time thinking about the giant box.

Edit, compile, test. Edit, compile, test. Over and over, each time getting close to the goal. The editor and compiler were running on the switch itself. The byproduct of using our own switch hardware to develop switch software was that I didn’t have to update or load anything. The giant piece of equipment was merely a big peripheral to my Unix system. We just ran the control code fresh each time.

The control system ran my Unix. When we started the project we used a new disk technology called SASI, short for Shugart Associates Systems Interconnect. It wasn’t long before the name changed to SCSI. In the past, disk drives used removable disk packs. The less expensive, more compact 5.25” drives were now sealed.

It occurred to me that I could make a carrier for the disk that would mount on the drive, which would let me plug and unplug disks. We incorporated this into the control chassis and it worked like a charm. As far as I know, it was the first hot-plug disk carriers. This was in the 80s.

As the night wore one, the pre-dawn hours of the morning anyway, the trips to the rack were more and more frequent. Pull the card. See what happened on my screen. Push the card back in. Look again.

Soon I was seeing the card disappear and reappears as my code sent and received messages through the RS–485 network to the microprocessor in the switch cards. Before long we were “failing” any of the matrix cards and getting the protection card switched online in its place.

We yawned as we walked into the bright, early morning sun. A trip down the road to the local diner was a welcome change from the past fourteen hours. By the time we got back to the office, people had started arriving at work, all fresh and ready for a new day. We were met in the foyer by the CFO. His face wore a question as he stared us down.

“Well?!’ he asked.

“Of course it works!” I answered.

I’ve not been hugged by a chief financial officer before or since.

The customer arrived. We demoed. He liked. As he drove his rental car away, the CFO held the check, smiling. For a while, at least, the cash had a bit of flow. It was time to start my seventy-minute commute back home.

About the Author

Brantley CoileInventor, coder, and entrepreneur, Brantley Coile invented Stateful packet inspection, network address translation, and Web load balancing used in the Cisco LocalDirector. He went on to create the Coraid line of storage appliances, a product he continues to improve today.

Cheaper than the Cloud

Coraid Etherdrive SRX™ software s also easier than the cloud. It’s faster too. It builds out like the cloud, yet provides the security of on-premise data. Let me explain both how and why.

Learn How

←Previous | Blog Archive | Next →