Bad hosts, or:
how I learned to stop worrying and love the overlay network

Concluding my notes on Web3, I wrote that

Ethereum should inspire anyone inter­ested in the future(s) of the internet, because it proves, power­fully, that new protocols are still possible.

and, accepting my own admonition, I have been tinkering on a new protocol — simple and, shall we say, “hypermedia-ish”—that I hope will be inter­esting to others. More on that later this year.

Today, I want to document a real­iza­tion that struck along the way.

The modern internet has rendered the figure of “an indi­vidual hosting a small internet service, all on their own” as antique as a blacksmith; this, I sorta knew. But where before I might have called it a matter of ergonomics or motivation, I now under­stand that it’s deeper in the network.

Critics of central­iza­tion will eagerly describe the centripetal forces sucking everyone and every­thing into the big platforms. Those forces are real! However, there ARE centrifugal forces — political, moral, creative, artistic — pushing back out toward the edges … it’s just that they are cancelled almost totally by NAT.

It’s not impos­sible to host a small internet service from a computer in your home, but it takes some fairly intense tinkering and maintenance, so it remains the pastime of … the fairly intense. Brittle router admin is no match for the sleek, it-just-works seduction of big platforms and/or cloud data centers.

Honestly, though, I am starting to think 50% of the ease and power of central­iza­tion is just a stable, public IPv4 address.

This is all coming out of my own experience, my own thought experiments, as I sketch out protocols and apps, “ways of relating” across a network. Every time I muse about something decentral­ized, I bump up against this barrier: a person connected to the internet from home cannot host a small service of their own.

There are workarounds for NAT, ubiq­ui­tous hacks, but they all require central­ized intermediaries. Think of video chat. While we’re chatting, the video is flowing straight from my computer to yours — in a sense, we are each hosting a small internet service for each other! But we can’t initiate that connec­tion ourselves. It requires a third host, one with a public IPv4 address. That host “punches a hole” through our one-way NAT mirrors and ties us together.

The connec­tion is ephemeral. Our video chat ends, and my Wi-Fi router’s heart flutters, and you are lost to me again.

The workarounds are fine as far as they go, but NAT tricks can’t get us the one thing we really want, the foun­da­tional internet thing: the ability to simply listen for connec­tions. Therefore, whole classes of possible services and rela­tion­ships don’t exist; a whole alternate internet history.

As home internet users, we can only speak and request, not listen and serve.

Computers with the ability to listen on the internet are called “hosts”, and there’s an inter­esting etymo­log­ical impli­ca­tion there. Today, as home internet users, we are not hosts; and perhaps we are missing out, therefore, on a degree of etiquette, and conviviality, and satisfaction.

I find it melan­choly: all these powerful computers, my laptop and yours, not to mention the servers in my office, your super­com­puter smartphone — they could be doing interesting things together, shuttling data around in inter­esting ways. Back when most (of the rela­tively few) internet users could host freely, none of them had a gigabit connec­tion at home; now, many internet users have bandwidth and processor cycles to spare, but we can’t host. Technological irony.

I truly did not have an opinion on IPv6 before December 2021. Suddenly, I am a wild-eyed evangelist, because this scheme, which works and is widely supported but not widely deployed, supports an internet in which every computer can be a host, if it wants to be.

For my part, I think an IPv6 internet would be almost unimag­in­ably fertile and productive. New kinds of apps and services would bloom like flowers in a dry meadow after the rain.

Ohhh welllll

The rise of “overlay networks” is a natural response to the limi­ta­tions and frus­tra­tions of the public IPv4 internet, and I think their accel­er­a­tion in the last few years deserves more attention.

These are illusory networks estab­lished on top of the public internet. Often, you install a small program on your computer, and it allows the rest of your appli­ca­tions to “see” a group of other computers as if they were close at hand on your local network, even if they’re actually far away, deep in NAT hell, unreach­able by normal means.

Here, I’ll make it more concrete:

For years, I was vexed by the task of reliably reaching the two servers in my office from anywhere outside, e.g. from home. The office doesn’t have any fancy network hardware, just a Wi-Fi router, and the best I could manage was a brittle port-forwarding scheme that never worked for more than a couple of months at a time.

Then I discovered ZeroTier, which allows you to create and manage these overlay networks. I installed it on my laptop, my old iMac, and my two servers, as well as an EC2 instance, and ever since, I have hopped between them with ease, no matter where I am. Each computer has a stable IP address in the 10.0.0.0/8 range, reserved for private networks. It’s fabulous. They are all hosts again.

There’s a similar service called Tailscale, along with Slack’s somewhat more robotic Nebula, and plenty more to come, I’m sure.

I am the only inhab­i­tant of my ZeroTier network, and I get the sense a lot of people use the service this way, but both ZeroTier and Tailscale allow you to create overlay networks with many users — hundreds or more. In those cases, the networks become little mini-internets for your coworkers, your group of friends, your pirate armada, whatever.

ZeroTier’s founder Adam Ierymenko sometimes calls the service a “planetary data center”, insisting that it ought to feel like every computer IN THE WORLD is in the same room, regard­less of its actual location or network disposition. What a vision.

Along the same lines, one of Tailscale’s founders, David Crawshaw, has a blog post, actually quite moving, titled Remembering the LAN. He writes:

The LAN was a magical place to learn about computers. Besides the physical aspect of assembling and disassem­bling machines, I could safely do things unthink­able on the modern internet: permission-less file sharing, exper­i­mental servers with no security, shared software where any one machine could easily bring down the network by typing in an innocuous command. Even when I did bring down the network the impact never left the building. I knew who I had to apologize to.

These new (old) “networks within networks” are, in 2022, both useful and evocative, and I think they open some VERY inter­esting space for work on peer-to-peer protocols and “ways of relating”—I keep writing that, I know it’s weird — appropriate for mini-internets of only 50 or 100 people.

I mean, you know I love computing at this scale.

But, as cool and promising as the overlay networks are, I am not willing to sacrifice “public” entirely, because what is the internet, if not an open invitation? And a suggestion, recurring, that you might not already know all the people you want to know. (If you’ve found your way to this newsletter, this web page, then you under­stand what I mean.)

Again, I feel the resonance of the word “host”, and again, I think about etiquette, and conviviality, and satisfaction.

There’s also the peer-to-peer browser called Beaker, a power­fully centrifugal project; the idea is, or was, that your browser might host a website as easily as it navigates to one. Very 1993! (Thanks to David for prompting this addition.)

Its successor/inheritor/whatever, a protocol called Hypercore, nudges in the same direction. That melan­choly feeling again: when you look at the code, you find it writhing with NAT contortions. So much time and energy and creativity, just to “get back to zero”, the ability to listen on the internet.

I mocked something up using Hypercore’s great imple­men­ta­tion of a peer-to-peer DHT swarm, the same technique (PDF) used to coordinate, among other things, the nodes of the Ethereum network. (There are rela­tively few Ethereum nodes — on the order of thousands.) It’s a clever model, but/and, I found it really “heavy”, and, call me greedy or unreasonable, but I just want the simplicity of

socket.listen

which, when it works on the public internet, has decentral­iza­tion built in.

My interest in all of this is a bit odd-angled, because I don’t actually care about decentral­iza­tion that much; or, I think it’s interesting, but I think a lot of things are inter­esting, and I’m willing to weigh them against each other, make compromises. TONS of compromises.

What I’m really inter­ested in — what I dream about — is the oppor­tu­nity to play with new protocols without taking on, perforce, the burden of infra­struc­ture. If we cast our gaze back to the early days of the World Wide Web, we find all these people coming up with cool new ideas for HTTP and HTML, writing new server software and new browsers with new features … and very conspic­uously not “operating the web”. They didn’t have that power or that burden. The web was … out there.

Pretty good deal if you can get it.

Defining a protocol is challenging enough; program­ming a client appli­ca­tion more chal­lenging still; but backing up the database? I am just not inter­ested in that kind of work, that level of stress.

This, too, is part of Web3’s appeal: you can invent new things without running the infra­struc­ture that supports them. Of course, it trades that stress for a new kind, the madness of immutable code, and charges you for every pulse of the virtual machine … so, for me, it’s a pass.

I have no great conclu­sion to offer, and I’m sure most of this is old news to those of you who have tangled with peer-to-peer protocols before. I guess the surprise, for me — the thing I felt an urge to share — was the real­iza­tion that at least some of this tendency towards central­iza­tion is inherent in the archi­tec­ture of the internet itself. The trapdoor of IPv4 exhaus­tion delivered us to this place, stuck behind NAT, pounding on the glass. It’s a huge bummer!

I should add, as a coda, that ZeroTier has an inter­esting offering, a set of public networks that anyone can join. Each only allows access to a partic­ular range of ports — you can choose them — via IPv6, which works for everyone, because it’s virtu­al­ized through ZeroTier’s software; it’s “fake IPv6”, I guess. These networks are a lovely affordance, and there’s a world in which the instruc­tion manual for my new protocol begins with: “Step 1. Download ZeroTier and join public network X.” Voila, the NAT problem goes away; it’s 1993 again; we can all see each other.

The thing that stops me from doing this, for now, is the lingering sense that, come on, there must be a way to offer something inter­esting that is not woven into the infra­struc­ture and protocol of one partic­ular company.

But who knows! I might get over it!

February 2022, Oakland