ISP outage: My personal disaster recovery

wifi city

My wife had to work in the middle of the night last night and it did not go well. She started around 3 a.m. only to discover that there was no internet access. The lights on our cable modem looked normal enough, but that was a scam.

Her laptop is Ethernet connected to our router. She tried to get online by connecting to a hotspot from her cellphone, but that failed. I don't know why, I was sleeping. Her work was time sensitive, so she woke me up.

As might be expected of a Defensive Computing blogger, I was reasonably prepared for an ISP outage. Our router, a Pepwave Surf SOHO, has three inputs. When the cable modem goes off-line, we can feed the router from either a Wi-Fi network or a 4G/LTE antenna connected to the USB port on the router. 

When traveling, the Wi-Fi in a hotel could be input to the router. At home, we use a hotspot from a cellphone. Since this is rare, I usually have Wi-Fi input disabled. Peplink, the company that makes the Surf SOHO, refers to Wi-Fi input as "Wi-Fi as WAN" (to techies, WAN means internet).

I have never used the Surf SOHO with a 4G/LTE antenna.

As long as the router can get online, all the devices in our home don't know or care which source the router is using. The screen shot below shows how the router is normally configured, with Time Warner Cable as the only available input. 


The normal state of affairs on a Pepwave Surf SOHO router

The Surf SOHO's flexibility is limited to one internet source at a time. That said, it can automatically switch from a non-working connection to a backup and then back again. As shown above, each input source is given a priority and the router uses the highest priority source that's available. 

The Event Log of the router showed that Time Warner had gone off-line at 1:41AM.


The router's Event Log 

The first thing I did was to enable Wi-Fi as an input source and set it as the first choice. This is done by simply dragging the row labeled "" from the bottom of the list to the top. You can see this in the Event Log above. At 3:37AM, the message "WAN: Priority changed" shows that Wi-Fi is now preferred for Internet access over Time Warner Cable.

Then, after confirming that the hotspot on a cellphone was functional, I tried to logon to it from the router. This starts by clicking the gray "Wireless Networks" button (shown below) which then shows a list of detected Wi-Fi networks. The logon process is simple and self-explanatory. Good thing too, considering I was half asleep.

The router connected to the cellphone hotspot, but three seconds later, disconnected. I have no idea why. I tried again, and this time it connected and stayed connected.

At this point all was well, and my wife did her work while connected through the cellphone. 

The screen shot below shows the Surf SOHO connected to the Wi-Fi network from a cellphone. The screen shot was taken a while later however, which is why the Time Warner cable connection is in "Standby" status rather than an error status. By the time I made this screen shot, Time Warner was back online.


Using Wi-Fi for internet access, with Time Warner on standby

If something had gone wrong with the cellphone hotspot, there was another cellphone in the house, from a different cellular provider, that could have stepped in. I made the decision to use multiple cellphone providers 15 years ago, with fallback in mind.

Both cellphones had already been tested with the router. This prior testing reminded me to plug the cellphone into an electric outlet. No doubt serving up a hotspot drains the battery faster than usual.

The event log shows the first hotspot connection was made at 3:41:21 and the second at 3:41:53. The black boxes in the screenshot are the SSID of the cellphone's hotspot network. It also shows, that not ten minutes later, Time Warner came back online. Doesn't that figure? 

By 4:44 a.m. my wife's work was done and she turned off the cellphone hotspot. Falling back to Time Warner was done automatically by the router, and is shown in the event log. The green boxes in the event log screenshot are the public IP address assigned to the router by Time Warner.

For the most part, switching Internet providers is transparent to devices connected to the Surf SOHO router. But, a VPN running on my wife's computer was not happy with the change in ISP and disconnected. It was a simple matter however, to re-connect the VPN. In our prior experience, a different VPN did not disconnect.

After turning off the hotspot, the router keeps trying to connect to it as shown below.


After falling back to Time Warner, but with Wi-Fi still an available option

It could stay like this for a long time, I was told by Peplink that scanning doesn't consume much CPU horsepower. But, since the hotspot was gone, I opted to disable the use of Wi-Fi for internet access. This too is shown in the event log, from 5 to 5:01 a.m.


Despite having rehearsed this, mistakes are inevitable, especially in the middle of the night. 

My first mistake was to connect to the router via Wi-Fi. It should go without saying, that it's best to connect via Ethernet when making Wi-Fi changes to a router. What can I say, I was sleepy.

While connected through the cellphone, I tried to monitor the router from an Android tablet and made two mistakes.

The tablet normally gets online using the Surf SOHO equivalent of a guest Wi-Fi network. The network totally isolates the devices using it and restricts them from accessing the router. I had locked myself out.

After realizing this, and connecting to a non-guest Wi-Fi network, I still couldn't access the router. The tablet uses an always-on VPN that, of course, was on. Again, sleepy.


With a bit of planning, I could have both slept through the night and speeded up the fail-over to the smartphone hotspot.

One option would have been to enable Wi-Fi as input (I normally keep it disabled to avoid constant scanning) and set it as the secondary option. Had I done so, when Time Warner failed, all my wife would have had to do was start the hotspot and the Surf SOHO would have automatically connected to it. 

Internet outages are rare, but this is probably what I'll do the next time my wife has to work in the middle of the night. 

An even more pro-active approach would be to set Wi-Fi as the secondary input (Priority 2 in the router interface) and connect the router to a cellphone hotspot before going to bed. 

If Time Warner fails in this configuration, the fallback to the hotspot would be automatic. And, the reverse should be too -- turning off the hotspot should automatically send the router back to Time Warner. 

If Time Warner doesn't fail, this setup would have no effect on our 4G/LTE data caps. The Surf SOHO does not load balance, when the primary ISP is alive and well, no data is sent to the secondary ISP. 

The best solution, using two ISPs in parallel and load balancing between them, is significantly more expensive.  

If you work from home, and have time-sensitive assignments, what's your fallback plan for an ISP outage? 

This story, "ISP outage: My personal disaster recovery" was originally published by Computerworld.

ITWorld DealPost: The best in tech deals and discounts.