The Joys of Ethernet

Well! What would someone who invented ATA-over-Ethernet (AoE) say? On top of that, I called my products EtherDrive SAN. Kind of get the picture? I love Ethernet.

This is a modern photo of some crazy person (not me) playing with 10Base5. That’s 10 as in 10 Mb and 5 as in 500 meters, or how long a yellow snake you could hide above the acoustic tiles and still work. And Mb is no mistake. “M” as in mega, and “b” as in bit. We’re talking pretty slow by today’s standard. And it had everyone on the network on it. One wire. Every server, going by your MAU box in the ceiling. A bit of coding and you’re watching everything going on in the building.

Today, mostly I use either Wifi, which is no cable at all, or a 10 Gb blue fiber cable to a break out box in my office connected to the machine down the hall. (I like 10 billion bits per second better.) A long way from climbing into the ceiling with a drill, looking for the mark on the yellow snake so I can drill at the right “node” to insert my vampire tap.   

It’s Halloween as I type this, so vampire tap seems apropos. Maybe that’s why I thought of writing about this in the first place. But it was magic, this yellow snake. Then we got hubs with our unshielded twisted pair, doing dog runs to the phone closet and the mostly empty relay rack with the Synoptics hub. How funny it looked at first. The RJ45 connectors looked like swollen phone jacks, with their eight connectors. And what did the “isolate” LED mean?

It was still 10 Megabit, and I could still put my Ethernet Medium Access Controller into promiscuous mode and listen to all the connections. If one kept state, they could monitor multiple byte streams all at the same time. That was back in 1991. In fact, even after I returned from California for the first time, I used hubs. It made developing protocols easier. Even after we sold the PIX to Cisco, I continued to use slow 10 Mb hubs, which worked very well with the hyperefficient Plan 9 system. It wasn’t until in the 2000s that I moved into switches and moved up to 100, then 1 Gb, then 10.  Totally different creatures, switches. Hubs were just operational amplifiers. (That sounds a bit pretentious. We call them Op Amps at home.) So every thing saw everything. But a switch, well, that’s really just a very special computer with a large number of ports, storing incoming frames, and sending them out again in the appropriate direction. No snooping on that. Unless the switch does it as a favor. I use Andy’s switches these days.    

But during all that time, the humble Ethernet frame, that paragon of simplicity and elegance, has remained the same. 48 bits for destination. 48 bits for source. 16 for a type to demux on, and a bunch of data bits for you to stuff other headers and data into. 20 bytes of IP header, and then another 20 bytes of TCP header, then the every complex iSCSI header, then your CDB and your data. That’s if your unfortunate enough to run iSCSI.

Or the simple 36 bytes for AoE header. The destination and source and type are all still there and included in those 36 bytes. 14 bytes of Ethernet header. 10 bytes of AoE common header. 12 bytes of ATA header. That’s it.

Simple, when you know how. Simple when you’ve seen the whole story from the beginning. And you have the intestinal fortitude to say no to IP and TCP. Most developers have bosses who will tell them they can’t invent a protocol right on the wires. That’s just not done. There’ll be meetings, where the CTO, then the CEO, then the board, all will explain to you that you have to use TCP. You’ve to be innovative by doing things exactly the way they’ve been done before.

Or you can be truly counter-cultural, craft-oriented, and do the best thing from an engineering point of view, from a customer beneift point of view. This is best done by avoiding working in a company where your engineering decisions are limited by the insecurity management, by their not wanting to do anything different. I’m glad that’s not me.

Not that counter-culture means flaky or arbitrary, or undependable. What we’re doing at New Coraid now is being done solely to make sure those who bought Coraid EtherDrive products since 2004 are being well looked after. And I really like giving people the Unix of Storage Area Networking (SAN). And at a great price. We still support systems first sold in 2007!

Anyway, I really love Ethernet. I still find it a joy to think about and create with.