AcceptIO services experienced partial and intermittent outages during most of the day on Friday, 28 December 2012. The outage was due to problems in our data center and the upstream network providers.
Full service was restored at about 10pm (Pacific time), with a further brief maintenance outage planned for the early hours of 29 December 2012.
Apologies all around.
Here are excerpts of a couple of messages we received from our data center operator:
28 Dec, 8pm PST:
As many of you know, WORLDLINK has been experiencing intermittent Internet Traffic delays throughout the day. These delays have caused periodic latency and in some cases short outages. Our technicians are diligently working with our Fiber Network providers and upstream network providers to identify and resolve the problem.
28 Dec, 10pm PST:
The network issue has been identified and the work-around have been put in place. One of the upstream providers has identified an issue they need to rectify. An emergency scheduled outage is planned for 12:15am on December 29th with a duration of less than 10 minutes. Again, thank you for your patience during this time period. We were unable to communicate as the networking issue affected both our phone system and email.
29 Dec, 11am PST:
Everything is currently up and functioning normally. At this point our staff is still investigating and getting everything sorted out, as to the full cause. We have opened a ticket with Cisco as we believe this was caused by a bug in the software and we are waiting for further input from them before we have all the pieces for a final conclusion.
We will update you in a couple days when we have collected all the data and have finished our investigation.
11 January 2013:
We would like to give you an update to the network service outage that occurred on December 28th.
The outage was cause by a bug in Cisco's REP protocol in one of our fiber circuits provided by upstream providers. The bug blocked multicast packets dropping EIGRP and OSPF hello packets across all our links. Cisco is still doing further testing to see what actually triggered the bug and working on a solution that can be applied across all Cisco networks.
To prevent this in the future, we have instituted several routing changes so that we are not as susceptible to upstream provider issues. We've also arranged our transit links to connect to both our core and distribution layers which would allow us to bypass our Ethernet links and bring each site up as it's own autonomous system. Additionally, within the next 30 days, our non-Westin fiber link will be fully activated and functioning. This link will have the ability to reroute traffic in a direction that is independent of the Westin point-of-presence.
For questions or comments about this outage, please contact This email address is being protected from spambots. You need JavaScript enabled to view it..