Looking Inside a Server

I’ve been peeping inside electronics boxes since Dwight Eisenhower was in the White House. As a tiny child, I was amazed and mystified by all the bits and pieces. There was a sort of aesthetic about it all. In a darken room, the warm red glow of the vacuum tubes on top of the metal box in the radio created a kind of modern ambience, inviting curiosity about how it all worked. Inattentiveness on the part of the assigned adult was sufficient once to allow me to pull out the metal box, with the tubes glowing, from its pretty wood case that normally surrounded it, and peep underneath, where my Father had, only a few days before, resoldered something, resurrecting the tube type radio that was old even in the 1960s.

Looking underneath, the radio was a mess, not like the top at all, which was a garden laid out to some master plan. The underside was more like a hodgepodge of spans of wires with bulges of some sort in the middle of each wire, running from a lug on one tube socket to different lug on a different socket, large waxy cylinders on wires, and mysterious flat wound wire wrapped around a paper tube with some sort of metal slug in it. Mostly, I learned that to perform such an operation with the radio still plugged into the wall was fraught with danger, and, I also accidentally learned it was best to touch one thing at a time and with only one hand. Luckily, I didn’t hold onto the metal chassis with one hand while I poked around with the other.

I’ve been peeking under the metal skirts ever since. I rarely get anything that can come apart without taking it apart. That was especially true of computers when I found them at the university, back when Jimmy Carter was president.

In the quiet of the middle of the wee hours of the dark morning, as a student worker at the University of Georgia, trusted with a keycard to the machine room with the little Cyber 18 minicomputer, I slip into the dark and flip on the lights. I know how to boot it, so I also know how to turn it off. Swing a metal side panel, twenty inches by thirty-six inches, swing out from the cabinet spring catch at the bottom, and the panel lifts away exposing a number of chassis inside.

A flip of the double power switch and the brutish fan sounds come to the sudden halt. Two quarter turn fastener turns laters, the metal plate covering the large anodized gold box containing the central processing unit, memory, disk, tape, and terminal controllers lifts off, exposing the edges of twenty-six printed circuit boards. Each board was eighteen inches tall, and twelve inches deep. An extraction tool, velcroed to the back of the cover, is inserted into the hole near the lower end of a board and pry. Out comes the first board.

The board slides out, riding on a smooth plastic board guide, all twelve by eighteen inches of it, and it’s covered with 100 small black plastic caterpillars, each with ten legs to the inch. Numbers like 7474, 7403, and 7400 appear on them, written in sliver print on each of the little plastic houses, like some carefully laid out subdivision of orderly dwellings, each with some sort of code on its roof, maybe for airplane pilots to see.

Each board makes its appearance in turn, emerging from the dark of the large cabinet. This minicomputer was built into a table with metal side panels going all the way to the floor; there is nowhere for anyone to sit at this table. The sides are the gold tan color of Star Trek’s officer’s uniforms, with rectangular vent holes backed with screen wire, also in the tanish gold color used in the famous starship. Not surprising given the time this computer was designed.

All the boards look very much alike, very orderly, each with its ten rows of ten plastic dual in-line packages (DIP). The writing on the edge of the board, etched in the same copper as the two layers of traces on each side were coated in with solder from the fountain wave of hot molten solder in the wave solder machine, give a better hint to the function of each board in the computer than looking at the constellation of DIPs.

Several say “32 K,” a whole eighteen by thirteen inch board with a hundred DIPs just to have thirty-two thousand sixteen-bit words. Two boards read “Micromemory,” a mystery to me. The word “micro” to me means the new kind of computer chip recently invented, but I see nothing on this board that looks like the forty pin DIP of a 1970’s microprocessor. The board also says “2 K,” which certainly seems “micro” compared to the 32 K of the other boards. I see a board that says “control 1” and another that declares itself to be “control 2.” Nowhere do I see the large things like those in the brand new Apple ][. It’s all a mystery.

All back together, side panel to the desk high computer back on, the fans whirl, forcing cooling air over all those boards I have just looked at. The console queries for the date and time, something that early minicomputer had to be told, and it’s all back running its operating system, Interactive Terminal-Oriented System version 2.0.

I think about those days, years ago, as I wander down the halls at Coraid a few days ago, stopping to get a cup of coffee, waiting for the long running test program to finish scanning a file system for mistakes. I walk into the lab next to the machine room. On the workbench is one of our EtherDrive Media Arrays.

I open the silver metal lid to the box, and try to imagine that I’m that green youth in my salad days, with long hair, ragged shirts, and an ignorance only exceeded by my curiosity. What would it have been like to see the insides of this computer if I knew as little as I did then, on that night in the 1970s?

The first things I would’ve seen on this computer with its single board, smaller than even one of the Cyber 18 boards, are the two large aluminum heat sinks four inches square and standing four inches off the board. What a monster. It’s hard to believe this hollow brick of aluminum’s job is to only cool the processor underneath. It’s size seems too large for such a mundane task. But I wouldn’t have known that underneath it was the large gray ceramic square. We call it the processor, but in reality it’s what used to be on the entire motherboard. We still call it the processor because we’ve always called the biggest, most expensive chip on the board the processor. And the CPU is part of it still.

Obviously, there are two processors because there are two heat sinks.

The next things that catche my eyes are the smallish printed circuit boards rising vertically up from the mother board. They gather around the heat sinks as if they were some kind of silicon life form worshiping at the feet of the processor in some kind of electronic silicon religion. In fact, they are cosy with the processor because the shorter the traces on the motherboard between memory and the processor, the faster the memory can run. Since different uses for these systems need differing amounts of memory, these vertical modules can be plugged into their small slots as desired. They are very tiny forms of the larger boards of the 1970s. They are like the 32 K word board in the Little Cyber.

The next physical features I notice are the PCIe slots, with their slick black plastic sockets, larger than the DDR slots, their wider groove inviting add-on board with sufficient mass that they need to be bolted down with the usual, poorly machined cheap triangular chromed screws. The slots are seemingly simple, a simplicity that belies the high technology of their design, carefully crafted to be reliable even given the speed of the signals that travel through them.

These PCIe boards are, in one way, mere descendants of those early minicomputer slots that the 18” x 12” Cyber 18 boards went into. PCIe was begotten by PCI. PCI was begotten by EISA. EISA was begotten by ISA. And ISA was begotten by the Apple II slot. The slots are dutifully lined up at the rear of the box, where the edge of their perpendicular boards can stick their connectors out of the back, alongside all the motherboard connectors for the USB, video, serial, and network connectors. The PCIe slots are how we add new things to make our system unique, to custom configure the system for our particular needs. In the Coraid EtherDrive Media Arrays, in the past this has included SAS Bus Adaptors.

Near the opposite end from the PCIe slots, the front of the motherboard, is a good sized, yet low, aluminum heat sink. Hiding almost unnoticed under the cover of shinny metal is the PCH, the Peripheral Control Hub.

The first hint as to what the PCH does comes from the components gathered around its heat sink. It is a motley crew of sockets, small ICs, connectors, a battery, and even a buzzer, ready to ring out the short and long beeps to explain that one forgot to install memory in the board, or that some other terrible thing has happened preventing the system from booting. Some of the USB ports may come from here, as well as the SATA ports. It’s this chip that interfaces with the flash containing the BIOS, and is loaded into the processor from here, long before the DDR sticks could function. They are all connected to the PCH chip, which itself is connected to the processors using a high speed bus such as the DMI (Direct Media Interface). DMI is cut from the same cloth as PCIe, with its lanes of balanced pair signals. Running at eight billion transfers each second, it can pump almost four billion bytes in those transfers.

The rest of the board, seemingly most of its real estate, is taken up with a repeating pattern of pairs of small rectifying electrolytic capacitors, ceramic blocks containing transformers, a small switching IC, and a small array of truly tiny passive components, very tiny capacitors and resistors. Each of these small villages, gathered around the cities of processor, DDR sockets, PCIe slots, and so on, convert the direct current voltage coming from the external power supply into a smaller voltage used by the nearby city. They do it very efficiently. And there’re a lot of them.

Near the processor, one of the voltage converters passes so much current that it needs a narrow yet high heat sink to avoid unsoldering itself from the board. But most of them, while hot to the touch, need no such accommodation. Everything seems to need a different voltage. The processors alone need several, each getting its own set of converters. Memory gets one voltage, PCIe needs 12 and 3.3 volts. DDR4 needs 1.2 V and maybe a second 2.5 V. The big chips, with a lot of pins, these days need 1.5 V, and can pull 100 amps of it for the larger dies.

I slip the lid back, sip my coffee, and walk back to my desk to check the progress of checking all those file system block tags. I can’t really unlearn the more than forty years since I pulled all the boards out of the minicomputer in the dead of night, so the exercise of looking at one of our boxes doesn’t really work. (One of those boards adorn our hall, the “control 1” board.) How hard it must be today, after all seventy-nine years of computer history, of ideas built on ideas, for today’s young and curious to see the story arc clearly. Or at all. I’m glad I arrived only about thirty years into that history.

I hope to use what I’ve learned to help those curious. And I hope together we use what we’ve learned to help all our users.