During our scheduled maintenance window on Thursday night, August 28th, we will be making some changes to our BGP configuration in an effort to better balance our traffic among our upstream connections. This should have minimal to no impact on network uptime, but there is the possibility that persistent connections (such as VPN tunnels) will reset as routes change.

The maintenance window for this action will start at 23:00 PDT on Thursday, August 28th, and conclude by 01:00 PDT Friday, August 29th.

UPDATE: 3:34 PM PDT The failed switch card has been replaced. The circuit was back up at approximately 1:30pm. After waiting 2 hours to ensure that the circuit is stable we have now turned the link back up and are running on all of our upstreams.

UPDATE: 11:35 AM PDT ETA for repair tech is 12:30pm

At 10:17 am PDT we began to see an issue with one of our upstream providers. The issue is being caused by a problem with our circuit between Digital Forest and the Westin Building. Our upstream is currently investigating, however, they have not been able to give us an ETA for resolution.

In order to minimize the effect of this on our customers we have shut down the link to the upstream affected at 10:39 am PDT. All traffic is currently being routed through our other upstreams.

During our scheduled maintenance window on Thursday night, August 21st, we will be making some changes to our BGP configuration in an effort to better balance our traffic among our upstream connections. This should have minimal to no impact on network uptime, but there is the possibility that persistent connections (such as VPN tunnels) will reset as routes change.

The maintenance window for this action will start at 23:00 PDT on Thursday, August 21st, and conclude by 01:00 PDT Friday, August 22nd.

This past weekend digital.forest experienced a network outage two hours and fifty minutes in duration, from 05:29 until 08:19 AM Saturday morning. Preliminary investigation revealed that the cause of the event was a partially-failed supervisor card in one of our two core network devices. This partial failure first created a loop, then a network storm. The loop occurred in the meshed wiring between devices used to facilitate redundancy. In this case the failure was not complete, so the redundant network path became a loop. The loop caused the network storm, as devices started responding to traffic coming back to themselves, from themselves. Once the source of the loop was discovered and removed from the network things returned to normal in about 20 minutes.

The partially-failed device remains off-line and will be replaced very soon. We are currently performing some forensics upon the failed device to ascertain exactly what lead to its failure and what can be done to prevent a reoccurrence.