Thank you to everyone for your patience on this issue review.

Time: 12am

Date: 23.02.2021

What we have determined by reviewing logs and timelines that the root cause of this issue stemmed from a hard crash that forced the cPanel server offline at 12am for a short time.

Service was restored shortly after with no indication of an issue.

The hard crash along with automated update scripts running in the background at the time created the database corruption when the cPanel server crashed.

The issue didn't immediately present to our team due to it's intermittent nature, even with monitoring sensors checking uptime of services the issue went unnoticed until reported by a client at approximately at 6am and then verified by our team shortly after.

Our team worked with cPanel to try and identify the root cause of the issue and how to resolve it in a timely manor.

Due to this being a shared server we needed to keep data integrity at the forefront of our efforts as rolling back to a previous backup would mean hours of new data could be lost, our goal was to make sure we covered every angle to safeguard the most recent transactions to minimise any data loss.

With this goal in mind it meant working longer and digging deeper to determine the correct fix even if it meant a longer period of unstable site access in the intrum.

Part of this issue was that not every site was effected as we had first thought, sites that where unaffected had a valid entry in the index, only sites that had their entries corrupted experienced the intermittent up/ down database connectivity.

Shout out to our team who worked tirelessly to fix this issue as quickly as time permitted, @Felix big thanks for your great comms and troubleshooting.

If you have any further questions on the postmortem please feel free to open a support ticket with “cPanel-vh4 intermittent MySQL Database drops” in the subject field and one of our team will get back to you ASAP.

thanks for your patience in this matter.

Yours Sincerely

(vh) Admin Team