Search for a command to run...
No comments yet. Be the first to comment.
Why your reactive state fails structured clone, and the fix I shipped.
I built a recipe keeper called Recipe Jar. You paste a recipe link, it gives you a clean card with just the ingredients and steps, and you can save as many recipes as you want. No account, no ads, wor
My Ubuntu server kept freezing. No warning. No error. Just completely stuck. Screen frozen, keyboard dead, only option was to hold the power button and force restart. This happened randomly. Sometimes
Let me tell you about this week(Obv. mine). I pressed the power button on our office’s Ubuntu server. Fans started spinning. CPU light turned on. Everything sounded normal. But the monitor? Completely black. Nothing. Not even the BIOS screen showed u...
Every team has one complaint that never becomes a proper task. Someone says it, everyone nods, nothing happens. One of our complaints was "the VPN is slow sometimes, or the VPN feels too slow sometimes, etc."
As a sole DevOps, I nodded along for years too. We all had the same theory: the VPN ran on a tiny t2.micro from 2018 (which also ran our monitoring, because of course it did), so the server must be too small. Case closed. If I'd fixed it blind, I would've bought a bigger instance and felt very productive about it.
Then one day I actually looked, and... yeah. This post is the story of how "tune the server" turned into "replace the whole VPN", what I found along the way, and what the new setup honestly doesn't fix.
First thing I did was pull two weeks of metrics, fully expecting to catch the little instance drowning.
Nope. CPU credits full the entire time, never throttled once. CPU at 2 percent. Network peaking at 8 Mbps on a NIC that can do way more. I kept scrolling thinking the bad part was further down. It wasn't.
So a bigger server would have changed exactly nothing, and I came embarrassingly close to buying one anyway. Pull the metrics before you spend money. Everyone says this. Now I say it too.
Most of our team is in Nepal. The server was in Ireland. That round trip takes about 250 ms.
And OpenVPN 2.3 ships with a 64 KB buffer that nobody, me included, had ever touched or even heard of. I only found it in some old forum threads while trying to work out why every tuning guide was obsessed with buffers. Turns out the speed of a single connection is just buffer size divided by round trip time. That's the formula I didn't know while being, officially, the VPN guy:
64 KB / 0.25 s ≈ 2 Mbit/s
Years of "VPN is slow sometimes", and it was one default value times distance. The "sometimes" was evening congestion piling on top.
I raised the buffer, and speed jumped about 4x the same afternoon. For a moment I felt like a hero.
Then I went through the config, and the feeling you know... went away😃.
Some highlights from that config:
Compression was on. Does nothing for encrypted traffic, and it's literally a known attack vector (VORACLE).
The cipher was AES-128-CBC, because our OpenVPN was too old for anything newer.
One route had the wrong netmask, so every laptop was shoving a huge IP range through this poor little server for no reason at all.
All of it on Ubuntu 16.04, unsupported since 2021, running an OpenVPN binary built in 2017.
I'd love to blame whoever wrote that config. Problem is, the server's old enough that some of those lines are probably mine.
Anyway, notice what these all have in common: they're settings. OpenVPN gives you dozens of them, and ours sat wrong for most of a decade without anyone noticing, because wrong settings don't explode. They just quietly tax every connection while all the dashboards stay green.
And the tuning created a new problem nobody asked for: now I was the only person who knew where those settings were.
I won't pretend I chose WireGuard out of deep expertise. I chose it because everything I read kept pointing the same way, and I went in half expecting to find the catch.
What I found instead kept mapping onto the exact mess I'd just cleaned up. There's no cipher choice (it's ChaCha20-Poly1305, period), so nobody can leave a twelve year old cipher in there. No compression option, so no VORACLE. No buffer setting at all, so the bug that started this entire story can't even be written. The whole server config is around ten lines, and half of those are keys.
At some point it stopped feeling like evaluating a VPN and started feeling like reading a list of my own mistakes, each one crossed out. That sold me harder than any benchmark could have.
Was it a risky bet? I checked. It's about 4,000 lines of code sitting inside the Linux kernel (OpenVPN plus OpenSSL is hundreds of thousands), and both Tailscale and Cloudflare WARP are built on it. Half the industry placed this bet before me.
Oh, and one thing I didn't expect to care about: switch from wifi to mobile data and the tunnel just keeps going. OpenVPN drops and reconnects every time. If your team works from home on shaky internet, that alone might be worth it.
Adding someone to the old VPN was a five step certificate ritual on the server, documented in a note that existed only on my machine.
While migrating I counted what that ritual had produced over the years: 41 certificates. Including test, test123, mfa-test, one just called dev, and my own name spelled three different ways 🪦. Nobody could say which of the 41 were still alive. And since revoking one meant regenerating a revocation list and reloading the server, nobody had ever revoked anything. Sit with that one for a second.
For the new server I wrote a small script called wg-peer:
wg-peer add name creates the user and prints the config, with a QR code for phones.
wg-peer remove name cuts access immediately.
wg-peer list shows every user and when they last connected.
So "who can still get into our network" is now a command instead of a guess. Sometimes I run it just because I can.
Now the part most migration posts skip.
Distance is still distance. Nepal to Europe is around 250 ms on any VPN ever made.
A bad ISP night isn't a VPN problem either. During one late test, the plain internet with no VPN at all was dropping 7 percent of packets to Europe. At that point every tunnel feels broken, and the VPN takes the blame for the pipe under it. Test without the VPN first, and test at more than one time of day. Ask me how I know (or see below).
There's no SSO or MFA built in. WireGuard checks keys, not people. Our old VPN had neither, so we lost nothing, and if we ever need identity, Tailscale and Headscale add it on top of this same protocol.
And it's UDP only. OpenVPN's trick of falling back to TCP port 443 to sneak through hotel wifi doesn't exist here. If your team travels a lot through hostile networks, keep that in mind.
Nobody complains about buffer sizes. People complain when they can't reach the staging database. So the migration had exactly one rule: the old VPN doesn't change until the last person has left it.
The new server ran in parallel on its own client IP range, and the entire rollback plan was "reconnect the old client". Cost: nothing. Result: nobody locked out, which is the only migration metric anyone remembers anyway.
Two decisions I'd repeat anywhere:
Give the VPN its own tiny network. Ours got a small dedicated VPC, which sounded like ceremony until I remembered where the old VPN lived: on the same network as the company blog. Meaning every firewall rule that trusted "the VPN" also trusted a public WordPress site. We were one hacked blog away from handing out every VPN permission we had.
And generate the firewall changes, don't guess them. A script read every rule that trusted the old VPN's range (72 rules across two AWS accounts) and spat out Terraform mirroring each one for the new range. Reviewable as a plain diff. Guessing firewall rules is how you lock someone out on a Friday.
The macOS WireGuard app can't see hidden folders. I'd saved the first client config in ~/.ssh, which felt like the proper place for it. The app's file picker can't open dot folders, so I pasted the config by hand... and cut the routes line short without noticing. Result: a tunnel that connects perfectly and reaches two of the seven networks. I debugged the server, the firewall, and the peering before suspecting the one component that had been operated by hand (me).
Terraform died mid-apply on shaky wifi. Three resources existed in AWS but not in the state file, so the next apply crashed into its own leftovers. The fix is boring and worth memorizing: import the orphans, delete the duplicates, plan again, then apply. A half-applied change isn't an if, it's a when.
My first speed test nearly fooled me. 127 Kbit/s through the brand new tunnel. Looked like a disaster, and I was about two minutes from opening a bug report against WireGuard itself. It was 9pm in Kathmandu, and the plain internet was dropping 7 percent of packets on its own. Next morning: fast. The pipe, not the tunnel.
I ran a VPN for years without knowing the one formula that governed its speed. Not my proudest discovery. But I've stopped thinking it's unusual, because nobody knows their whole stack. The interesting part is what you do on the day you find out.
What I took from this one: measure before you blame, because the suspect everyone agrees on was innocent. Read your defaults, because they're decisions someone else made years ago, in a different world, for a different network. And when you pick tools, count the settings, because every setting is a place where a future person can be wrong for eight years without anyone noticing.
A VPN should be boring. Ours finally is.
And,
Thanks for reading till the very end!