We expected several 5 minute service breaks during scheduled window for a notified selection of customers as we upgraded the routing engines, however the vendor equipment software upgrade did not go as expected leading to longer outage periods than expected.
Additionally our DNS servers became unreachable for all NZ customers which was not expected as we have DNS servers in Hamilton, Auckland and Sydney and should be fully redundant. This caused two DNS outages between 3am and 4:45am for all NZ customers. The first DNS outage was about 7 minutes and the second about 20 minutes.
The software upgrade was completed and does not need to be rescheduled.
We will be doing an incident report which will be made available on request when our post-mortem is complete.
