Addressing all (vh) clients

Thank you for your patience in awaiting this postmortem of last weeks incident.

Overview:

On the 7th April 2020 approximately 1:30 pm (13:00) we started seeing intermittent random spurts of traffic latency across our public network.

The issue was presenting as slow traffic pings for approximately 6-10 min at a time disrupting connections to Data Center traffic inbound / outbound.

At the time of the issue our team had presented a list of possible causes and started to action steps to troubleshoot each possibility.

As the day progressed the latency interruption became less disruptive as occurrences where spread further apart eventually stopping at approximately 9:30 pm (21:30).

Unfortunately we where unable to pinpoint the root cause of the issue at the time due to the inconsistent nature of the events but had setup additional monitoring and continuity plan in the event the issue returned.

Follow Up Event:

5 days had passed without re-occurrence until Sunday April 12th approximately 5pm (17:00), with previous monitoring already in place and more preparation we where able to determine the root cause was a DDOS DNS amplified attack targeting a clients VM which had firewalls turned off at the time.

This DDOS attack caused a slowdown in network traffic and was outbound from (vh) public facing services.

Service Improvements

To minimize further disruptions our team has performed and put in place the following:

• Additional bandwidth throttling measures for direct to public traffic and public facing services
• Additional triggers and alarms have been created to monitor client public traffic throughput hitting maximum allowed levels i.e. 70%, 80%, 90% of allowed throughput will alert our team and pinpoint the service abusing shared bandwidth.
• Direct to Public and vCloud traffic have been split into two separate networks to minimise disruption across public, private and hybrid network spaces.

If you have any questions on the postmortem please feel free to open a support ticket in the (vh) cp platform https://control.velocityhost.com.au and use "April 7th postmortem" in the subject field.

Enjoy the rest of the Easter long weekend and thanks again for your patience.

Kind Regards
(vh) Admin Team