Posted on November 10, 2009
Filed Under English
The countless readers of my weblog (seriously, I don’t count them) will have noticed that my webserver was offline for over fourteen hours today. When I arrived at work this morning, I found that out myself.
The first thing I do when I get in (well, the second, who can start the day without coffee?) is fire up my laptop and tunnel into my network at home to have access to my email and my files. But that just didn’t pan out this time.
So I sat there scratching my head and drinking another cup of coffee. Then I connected to the very few machines that have any access to my server (a FreeBSD router/firewall/gateway actually, plus some other servers behind it) at home, and they couldn’t reach it either.
Now, I hate it when that happens. Not knowing what the problem is, and having to have doubts and dark thoughts about that all day long. Network interface dead? Firewall hiccup? DSL crash? Total crash? Disks kaputt? RAM kaputt? Do I have to buy a new server and start all over again?
Ok, I have excellent and recent off-site backups, and getting a new server online is nothing more than building the OS, building the ports, putting config files and databases back, and that’s really it. But it’s still more aggravation than I care for at any given time.
If I wanted aggravation, I’d run Linux, for crying out loud!
Anyway, when I got home again, the first thing I did (I skipped coffee) was look at the server. It was sitting there, humming along nicely, there was network activity, there was disk activity, and nothing burned down.
So I logged on, and everything looked totally normal. Traffic was coming in and going out, and … wait a minute .. traffic was actually going out and coming in, not the other way around. So SYN/ACKs came in, but no SYNs. I checked the firewall ruleset. Unaltered. I checked the rules in memory (pfctl -s rules, etc.). All fine.
Then I checked the log files, looking specifically at what had happened around 6 AM, when the last sign of ‘all OK’ had shown up on an external monitoring server. There I saw that the server had been unable to connect out from 5:59 to 6:03 … aha … roughly the time it takes my DSL modem to reboot. And that DSL modem has a default setting for firewalling: allow nothing in, and everything out, plus stateful return traffic back in. Exactly what I was seeing.
But my DSL modem was beaten into submission by me a long time ago, with a configuration that looks like this in human-readable form:
Give the friggin’ IP address to my server, and (pretty please, with sugar on top) get your filthy hands of my soddin’ traffic, I will do all of the firewalling myself. Just sit there, blink your lights, and look pretty. Don’t bother me again. Save.
To make sure, I connected to an external server and pinged my firewall from the outside. Nothing showed up in a tcpdump of pflog, and nothing even showed up in a tcpdump of my external interface. DSL modem for sure then ..
I still have no idea how that DSL modem managed to take itself offline for four minutes and then come back with its built-in firewall enabled, but a simple power cycle returned everything back to normal.
I’m afraid to check my voicemail now, because there are bound to be countless messages from people who couldn’t read my weblog …
Seriously, I don’t count them!