On Friday March 8, Cybera experienced an outage that impacted several of our members.
On Friday at 4:50 pm we began maintenance work on the DWDM in Edmonton to reset a backup power supply. This work was successfully completed by 5:00 pm. Shortly after (at 5:10 pm), we started receiving alerts notifying us that the DWDM was down. The main and backup power supplies failed, impacting members in Edmonton without regional redundancy. Peering, IBG (Internet Buying Group), VFS (Virtual Firewall Service), and shared hardware firewall services became unavailable to them. This outage continued until we were able to find and install a working power supply at 7:45 pm.
Once the DWDM came back online, the Peering and IBG services were restored for all members. Unfortunately, the DWDM outage triggered another failure. This impacted the affected firewalls, and spread to include the previously working VFS firewalls across both regions. After several hours of debugging, we determined there was a Layer 2 spanning tree loop issue. By 2:00 am on Saturday, the issue was resolved for many of our members.
We did not realize that, at that time, the Calgary region was experiencing similar issues. Later Saturday morning, some of our VFS members in Calgary let us know they were having issues (thank-you!). A review of the symptoms showed the same problems that Edmonton had experienced, and we were able to similarly resolve the issue by 10:00 am Saturday morning.
Two schools, one in Edmonton and one in Calgary, continued to have firewall issues. Our team worked with them throughout the day to debug, and were able to resolve their issues by late afternoon.
We are compiling a list of lessons learned which we will be reviewing next week. Once we’ve determined our next steps and how we will improve our procedures going forward, we will share that information with our members.
Category: Network failure
Start Time: 2019-03-08 05:06pm Local time.
End Time: 2019-03-08 07:45pm Local time.
Status update:
Service via Edmonton DWDM is restored. It was found out that the DWDM power supply failed and replacement is done.
Category: Network failure
Start Time: 2019-03-08 05:06 Local time.
End Time: No ETA
Status update:
We are experiencing issue with our DWDM equipment in Edmonton and some customer connections which don't have redundancy via Calgary are affected. We will send status update as we get more information.
—
Category: Google Services
Start Time: 2017-03-17 8:00 am Local time. (Friday)
End Time: 2017-03-17 4:00 pm Local times.
Anticipated outage duration: No Outage
During the time frame indicated above, CYBERA will be adding a redundant connection to Google Services at YYCIX ( Calgary) Internet Exchange. During the above outage window we do not expect any outage for Google Services, but if you experience any problem related to Google, please contact us via phone or email provided here.
Please contact the Cybera Network Operations Centre at network@cybera.ca or (403) 536-0880.
Category: Calgary Supernet Maintenance
Start Time: 2017-03-11 18:00 Local time. (Saturday)
End Time: 2017-03-11 18:30
Anticipated outage duration: 10 minutes
During the time frame indicated above, SHAW will be performing maintenance on the Cybera to Supernet connection in Calgary. The outage duration should be less than the total duration of the reserved maintenance window and could include several small durations outages while maintenance is being done. Members connected via Edmonton PoP will not be affected.
Contact: Please contact the Cybera Network Operations Center at NOC@cybera.ca or (403) 536-0880 when network problems are experienced outside of a maintenance window. All outages, scheduled or not are posted here:
Category: EDMONTON router maintenance
Start Time: 2017-03-08 21:30 Local time. (Wednesday evening)
End Time: 2017-03-08 21:45
Anticipated Outage Duration: 15 minutes or less
During the time frame indicated above, Cybera will be performing maintenance on one of its virtual core router in Edmonton. The outage duration should be less than the total duration of the reserved maintenance window and could include several small durations outages while maintenance is being done.
Contact: Please contact the Cybera Network Operations Center or (403) 536-0880 when network problems are experienced outside of a maintenance window. All outages, scheduled or not are posted here.
Category: Calgary Supernet Maintenance
AXIA Change Ticket # CA0001322541
Start Time: 2017-02-08 00:01 Local time. (Wednesday)
End Time: 2017-02-08 06:00
Anticipated Outage Duration: 5 minutes
During the time frame indicated above, Axia/Supernet will be performing maintenance at the Calgary PoP. The outage duration should be less than the total duration of the reserved maintenance window and could include several small durations outages while maintenance is being done.
Contact: Please contact the Cybera Network Operations Center at NOC@cybera.ca or (403) 536-0880 when network problems are experienced outside of a maintenance window.
Category: EDMONTON Supernet Maintenance
AXIA Change Ticket # CA0001322911
Start Time: 2017-02-15 00:01 Local time. (Wednesday)
End Time: 2017-02-15 06:00
Anticipated Outage Duration: 5 minutes
During the time frame indicated above, Axia/Supernet will be performing maintenance at the Edmonton PoP. The outage duration should be less than the total duration of the reserved maintenance window and could include several small durations outages while maintenance is being done.
Contact: Please contact the Cybera Network Operations Center at NOC@cybera.ca or (403) 536-0880 when network problems are experienced outside of a maintenance window.
Thanks for joining Revere, our home brewed notification system.
This page informs Cybera members of any network issues, problems, or simple notices such as maintenances. When real network issues occur, this page will be updated regularly with pertinent information.
Please add anyone within your organisation that should receive Revere network notifications messages.
Expert Tip: You can bookmark this page and just link to it next time you receive a revere message.