Most Coraid customers are happy using our storage appliances without digging into the technology or the design philosophy that lies beneath. Some people, however, seem to be interested in our unusual outlook, so I thought I’d talk about one of those oddities. At Coraid, we build almost all the software we use.
We do this because we strive for the highest quality, and our idea of quality has to do with simplicity, clarity, and generality. To get what we want, we need control of each layer in our systems, right down to the operating system and compilers.
Some might call this “reinventing the wheel,” but I’ve learned to dislike that phrase. It’s usually hurled at me in a meeting by someone who doesn’t actually mean “reinvent.” To reinvent the wheel, a person wouldn’t have a concept of “wheel” and have an “ah ha!” moment when they notice some round thing rolling. The first time that happened was between 6,100 and 5,400 BC in Syria.
No, the verbal rock thrower means “re-engineer the wheel.” Because I won’t settle for readily available software, they accuse me of wasting time. If only they would take a moment to look around, they would see the world is full of re-engineered wheels, all of which seem to be needed in their form.
When Henry Ford built his own power-plant to produce T-Models at the River Rouge Plant, it was far from a waste of money. Thanks to his time as chief engineer at the Detroit Edison Electric Company, he knew both what the plant required and exactly how to implement it. Turning steam into electricity was easy.
He also created his own ironworks and steel mill. The River Rouge Plant became an amazing, all-in-one facility. At one end, workers would bring in barges of ore excavated from Ford’s magnetite mines and finished Model T’s came out the other end. The plant’s smelters were fueled by coal from Ford’s coal mines. Even the railroad used to transfer materials was owned by Ford.
Sure, he could have bought coal from other people’s mines or used preexisting utilities and infrastructure, but he didn’t because he knew that if he took the time to re-engineer these things, he could make a better product all while keeping the cost down. In the 1910s, the T-Model was far ahead of other car companies. Automakers were mostly taking off-the-shelf parts and bolting them together.
These off-the-shelf manufacturers soon disappeared because they were unable to compete with the freedom Henry Ford’s re-engineering provided. Only his company could control both the price and quality of the cars.
In our own dimensions, the same is true of Coraid.
There is a great deal of technology that sprung from the minds of people far more brilliant than myself. Technologies such as Unix have not only been great solutions to existing problems but were able to adapt to new and changing problems over the course of their use.
Over time, though, these once elegant and clear technologies become larger and larger. They become cluttered with solutions to more and more problems.
Let’s continue with Unix as an example. In the early 70s, I was fortunate enough to be on a small team tasked to restore First Edition Unix from the source code found in an old technical memo written in 1972. The elegance of it all was astounding.
By May of 1975, Sixth Edition Unix was entirely written in C, a long way from its roots in assembly language. The Seventh Edition followed in 1978, an edition I’ve used successfully for two different embedded systems.
Seventh Edition Unix has all of 16,000 lines of C in its kernel. I’ve no idea the number of lines in Solaris, the last version of Unix, but it has got to be in the millions. The Linux kernel, a Unix clone, has passed the 15,000,000 line mark.
These versions of Unix became so massive because they accumulated solutions to all the problems that existed over the 42 years since Sixth Edition. Most of these amassed solutions are for problems that don’t exist. My favorite example is paging.
All modern operating systems “must” have paging, the ability to swap out 4K chunks of a running program to the disk when needed. This was a real requirement in the late 80s when a machine with 4 MB was considered a somewhat large machine. The first machine I used had all of 256 KB, and the first mainframe I used had memory that costs $1 a byte. That’s not a misprint. They paid $4M for 4MB of memory.
But today’s processors are so fast that if your laptop has to swap pages out to disk, it slows down so much that you think it has failed. (In fact, some systems have started to compress the pages and keep them in memory instead of swapping them out, a “solution” that adds even more complexity.)
So at Coraid, we search the past for technology, ideas, and source code, attempting to find the earliest versions of their history. We then bring the ideas into the 21st century. If the only version of a technology we can find is so cluttered that it becomes a digital archeological dig, we avoid it. Luckily, technology is all pretty well recorded in academic papers and vendor documents.
I call this processes “re-evolving,” and I’ve had good results with it. The software that emerges is tight, elegant, and usually easier to get right and keep working than the accumulations of changes created on other systems like barnacles on a ship that has never been scraped.
Re-evolving does have a downside, well two actually. First, it requires hard work. I can’t just simply plug in this or that and expect a product to come together. Second, the Coraid team and I have to actually understand how the technology works. Fortunately, we love this stuff and love to learn about the original, unmutilated ideas. The hard work becomes fun. Why on earth would we pay someone else to get to do it?