Adventures in Network Plumbing

I used to be such a good nerd.

Then I had two kids.

Somehow, between a wife and two boys, all of the time I used to spend pouring myself into web design, picking up a new programming language, waxing poetic about the latest nerd-rage-inducing article, or tinkering with some piece of hardware, has been replaced with other, far more valuable things like fixing my house, taking a son out for a donut, going apple picking, reading a bedtime story, bouncing the baby, etc.

There's no question whatsoever that my priorities are right in that regard, but when the technological infrastructure I set up when we first bought the house starts to break down, I have to fall back on admittedly somewhat atrophied muscles to fix them. This was the case last month when the state of our home network drove my wife to declare that I should not send her email during the day because she couldn't read it anyway.

Low WAF1.

I had, of course, noticed some problems with our home computer as well. But, in my defense, I rarely sat at it, and when I work from home, my MacBook Pro has had no problems with connectivity. Well, the wired network was flaky, but wireless was fine, so I guess… I guess I was just putting off figuring it out.

The Network

My home network is quite simple. I have a Verizon FIOS (Actiontec v1) router which has a 4-port 10/100 switch and an 802.11b/g access point. The access point is disabled, and one of the four ports has a Linksys WRT610n 802.11n router with 4-port 10/100/1000 switch connected to it. The Linksys has DHCP disabled and a static IP assigned outside of the Actiontec's DHCP range. It's an access point and switch. It serves up both 2.4Ghz and 5Ghz 802.11n SSIDs using WPA2. Good enough, and it's been working for 4ish years.2

There are three machines connected to the Linksys' wired switch:

  1. A basement PC that is mostly a backup host
  2. Our primary family computer in the living room (Mid 2010 Mac mini)
  3. My work MacBook Pro, when I'm home and using my desk

All other devices (iPhones, iPads, MBP when anywhere else, wireless printers, guest computers, etc.) use the 802.11n network. The signal strength is sufficient in the house, but anywhere outside gets nothing due to a nice, solid concrete/brick foundation and walls.

The Symptoms

About 3 months ago, I noticed that the mini was taking a long time to render web pages. I figured it was a browser thing, because it was sporadic. I cleaned the machine to the best of my ability, trimmed down browser plugins, made my wife stop leaving 8 tabs open, etc. It seemed a little better, but something was still off. I switched it over to the wireless network, and things improved dramatically. I chalked it up to wiring or something and went back about my other family business.3

Then it came back, while on wireless, but again, sporadic. I started using via speedtest-cli and noticed that I was getting about 3Mb down and 1Mb up4 on both the wired and wireless networks. From my MBP, I got 50Mb down and 30Mb up when on wireless. It had to be the mini. I was running OSX 10.7.5, had never done a wipe of the drive since upgrading it from 10.6, and who knows what sorts of cruft had been building up over the 3 years since we bought it.

Then I started seeing the same thing on the wired network for my MBP, which was fishy.

Perhaps I had two problems? Our wiring was bad / the switch was bad, and the mini was just in bad shape.

Rather than back everything up, format and reinstall, I bought a fresh 500Gb 7200rpm drive, installed Mavericks from my MBP and swapped the drives in the mini.5 Suddenly, it was back to normal, even wired. Crisis averted.6 I mentioned to Cait that she should watch some youtube videos to see for herself, but she quickly forgot.

Two days later it all came back.7

Time to dust off the nerd hat

Not knowing why this was all happening had finally started bothering me enough to feel the need to cheat on my evening time with my wife to improve things. I started by isolating the problem:

  • With my MBP I proved that the wired network was always bad by using a brand new, known-good Cat6 cable 2 feet from the Linksys, and then also wiring into the Actiontec, also 2 feet away.
  • I un-plugged the Linksys and re-enabled the Actiontec wireless to show that I still had good connectivity wirelessly on the MBP, which had always been the case.
  • I validated it wasn't a MacOS problem by showing that my basement PC had the same speedtest results.8
  • I turned off all wireless networks and all wireless devices and showed the same poor wired results
  • I rebooted the Actiontec just in case I was asked to do that by a FIOS rep.

Based on all of this, the only conclusion I had was that the Actiontec router itself was having trouble. That, or I was somehow being throttled based on MAC address only, but that sounded fishy. And supposedly Verizon doesn't do that, so yeah, router.

Putting my hopes in Verizon Support

I have never felt like Verizon was all that great with customer service. In terms of cellular service, I've never really had the need to interact with them, and when I did I often felt like they didn't much care about me or my issues, just getting me out of their sight. I figured that if I called FIOS support, they'd need me to repeat everything I did already, but I had no choice left.

Thankfully, I got on the phone with someone who spoke my language. I re-iterated the problem, as well as what I had done to troubleshoot it for myself. The tech asked me if I had tried a few other things which I had forgotten to mention (I had), and then said “Well, you have a V1 router. It could just be broken. I'll send you a new one next day air.”

Service. Win.

He walked me through a manual release of the current router's IP address and said to skip the install CD (it's Windows-only anyway) and just wait 5 minutes after the IP release to turn on the new one. The router was there the next day, I did all that he suggested and we're back to our normal blazing speeds. The new router has 802.11n and a 10/100/1000 switch, eliminating the need to use the Linksys. It's coverage is incredible. I was going to root the Linksys and turn it into a repeater, but there's no need to do so. I can use wi-fi in my garage now, which had been impossible before.


The title of this post is a hat tip to Ian Beyer who used to have a blog by the same name. I'm sure if he and I lived closer to one another he would have had this figured out in minutes.

  1. Wife Acceptance Factor. Hat tip to @shanselman.

  2. We used to use the Actiontec’s wireless network, but when Circuit City was going out of business a few years ago, I picked up the Linksys router during the firesale so that we could get a speed boost. The gigabit switch was worth having almost more than the 802.11n coverage. It’s running the latest firmware that LinksysCisco makes because, at the time, DD-WRT and OpenWRT did not support it, though they both do now.

  3. See, this is what I’m talking about. I never would have let this slide back in the day.

  4. We have the FIOS Quantum 50/25 package, so this was notably unacceptable. Again, 3/1 is pretty fast for the days of my youth, but it was relatively high-latency as well, so it wasn’t like things were only slow, they were also wicked delayed.

  5. The guides over at iFixIt are incredible.

  6. Never, ever think that anything is so simple. Ever.

  7. Cait was pretty annoyed that she never opened youtube in those two days.

Jan 2nd, 2014