It has long been a goal of mine to operate a server closet, annoucing a meager allocation of IP addresses, allowing me to host a mail server away from prying eyes, and an web server away from prying hands (ie, for free). Today, that dream is finally a reality! Let me tell you a story, friend.
About 5 years ago, I was looking at the list of IPv4 /8s, because I was bored. I noticed that the entire space of 220.127.116.11/8 was allocated towards “amateur radio”. Ham radio had piqued my interest before, for its ability to maintain vast communication nets even in the event of some force majeure which might cause the Internet to fail, and also for how a meager broadcast in the shortwave bands could be heard the world over. Its relation to the Internet, however, was hitherto unknown to me.
Clicking a few links, I came to understand that 44/8 was given to radio amateurs via the portal.ampr.org website. It had been allocated in the ‘70s by Hank Magnuski, an early internet entrepreneur who, like the other /8 allocatees such as Ford, GE, and the Prudential insurance company, had the foresight to grab some IPs while they were cheap. These days, while the rest of the world slaps together haphazard solutions like carrier-grade NAT to deal with the threat of IPv4 exhaustion, AMPRnet has the capacity to hand out /24s to any amateur who asks.
Anyway, after finally getting my Ham license, I sent an application in for a 44net allocation. Choosing between connecting via radio, tunnel, or direct announcement (ie, BGP), I picked the latter, having no idea what I was getting into.
Being a total networking noob, I knew that I had to peer somewhere and announce my IPs. That was about the extent of my knowledge, not including what “peer” and “announce” really meant. I knew that there was an IXP here in Portland, so I started there.
Hi, I have a /24 allocation in progress from 18.104.22.168/8, the class A network for amateur radio. I would like to BGP annouce this allocation locally, to learn about networking and the like, but my ISP has told me they do not offer peering for residential customers. Can an amateur peer at NWAX? I have a small router machine (PC Engines APU2C4) that needs only a single 1000BASE-T connection. Thanks, Simon McFarlane KI7KOC
Oh yeah, I first asked my ISP at home if they did this, but I just got some weird looks and a link to their business offerings page.
Incredibly, I got emails back from the president and vice president of NWAX. It turns out the president is also a ham, which was a pleasant surprise.
After some back and forth, and a bit of research on the part of the vice president, he gave me a crash course on how the internet works:
Simon, While you could announce the netblock into NWAX, it wouldn't benefit anyone but the directly attached members (and some of their peers). It wouldn't announce the block to the global Internet routing table, which I believe, is what you're looking to do. You'll probably want to look at an ISP first, and then once you have that connection and have your BGP sessions figured out, we can look at joining you to NWAX to keep your traffic local. Sincerely, Cory
At this point, I knew I needed to find some transit. I emailed 5 local ISPs, and got only one response: a quote from Hurricane Electric for $300/month, which was a little bit outside my budget.
At this point, I posed the question to Brian Kantor, the 44net head honcho, who told me this:
ISPs that will announce a block are rare. You might contact the HamWan group in Seattle; I believe they will announce a block and pipe it to you over a VPN if you are willing to go that route. - Brian
Making a quick visit to the HamWAN website, I hopped onto their IRC channel and explained my situation. It turned out they did indeed provide transit to 44net allocatees for free, via their Open Peering Policy program.
HamWAN pipes you into their network via IPSec, which I had tried to use before for my own purposes with little success. Sure enough, after much toiling with EO_ from HamWAN, we failed to get OpenBSD's isakmpd to interoperate with HamWAN's Mikrotik-powered endpoints. After a couple late nights of frustration, I broke down and ponied up the $50 for a Mikrotik “RouterBOARD”, which Just Werked.
My allocation was for a /24 (22.214.171.124/24), the minimum for a BGP block (I found out later this was to prevent the global routing table from getting too big). I chose to divide this internally into 126.96.36.199/25 and 188.8.131.52/25. The former would be for my home network and would largely be firewalled, and the latter would be for servers open to the world. I did this because I didn't want an attacker to be able to use a vulnerable server of mine to pivot into my home network.
With my IP space running and ready to go, I set about moving my servers from The Cloud into The Closet. I was already a fan of OpenBSD because of its incredible security track record and uncontested documentation, so I knew from the get-go that I wanted to keep using it for my servers wherever possible. I prepared a few PC Engines APU2C4s, which I love because they draw very little power, are totally silent, and fit just about anywhere. First to go was my mail server: I tar'd up the config files from my VPS, rsync'd down the Maildirs, and switched the DNS. ez pz. I then did the same for my web server a couple days later.
It didn't take long for me to realize the limitations of dinky MIPS routers. I pay for gigabit, but I only got around 250 Mbit/s before the CPU reached 100%. At the advice of HamWAN people (smart bunch), I threw together an AMD APU build to serve as a VM host, inside which I would run Mikrotik RouterOS. So, I ponied up again the $45 for an x86 RouterOS license.
There was one missing piece of the puzzle still, and that was…
Since most of my email comes from or goes to Gmail (or
Google Apps G Suite),
most of my email comes from or goes to IPv6 enabled servers. One thing that ate
at me was that if HamWAN's tunnel goes down, I wouldn't be able to send or recieve
email, and that is not good. If I could get IPv6 from somebody else, that would
allow me to keep most of my email coming and going until they fixed it.
I signed up for Hurricane Electric's tunnelbroker.net, a service which provides IPv6 tunnels for free, with a huge amount of PoPs around the world. Importantly (for me), it also provides /48s, so I could do the same isolation between my home and public networks as I did for my v4 /24 (as most of the time, you can't split an IPv6 network smaller than /64).
It worked with only one small hiccup: macvtap does not pass IPv6 traffic to VMs properly if the interface is not in promiscuous mode. Save yourself the headache and complaints from your girlfriend who is unhappy because the internet is down, and enable promiscuous mode on any NICs you want to macvtap before you even boot the VM.
I now have a firewalled home network consisting of globally routable 44net IPs, and SLAAC-based IPv6 for everything, as well as an open-to-the-world network with static IPv6. I pull about 750 Mbit/s up and down on speedtests, which is good enough for me.
My DNS search domain is desu.ne.jp, under which all my hosts live. Its super nice being able to just do “ssh katagiri.desu.ne.jp” from anywhere, and just have it work. Inside the network of course, I can just do “ssh katagiri”.
Cable management will be my next weekend project.