The King of the Undead

Timeline of the outage: (times in UTC)
Starting at 02:03 on Saturday, requests started intermittently returning error 502, and many of the requests that were served successfully were significantly slower than normal At 05:59, most of the services on the server crashed. All subsequent requests were served by Cloudflare with error code 502 At about 11:00, I came online and attempted to diagnose the problem. Due to the previous failures I was unable to access the server, so all I was able to do was route traffic to an offline page At 16:35, with extra help, the server was forcibly restarted At 16:43 the server seemed to have come up successfully, so we enabled traffic and monitored the status By 16:53 it was clear that performance very poor and a significant fraction of the requests were resulting in error 502 (which means in this case that the server was already processing too many requests, so where were no workers available), so we disabled traffic again to investigate the situation further At 17:36, there was nothing clearly wrong so we tried enabling traffic again At 17:39 the performance was significantly regressing again so the site was turned offline again At 18:00 we ordered a new server At 18:35 the server was ready to be set up with all of the forum-related things, and for the data to be migrated to it At 21:28, the new server was fully set up and the forum was turned back online  
What were the symptoms?
IOWait accounted for the majority of CPU time, but IO utilisation was relatively normal In the syslog we were seeing a number of IO timeouts for the primary drive Prior to rebooting, the limited errors that we could see indicated that there had been disk corruption  
What was the root cause?
Although we aren't 100% sure, we think it's fairly likely that one of the two RAID 1 disks that form the primary disk had failed, and the poor performance was a consequence of trying to rebuild the array.
 
Why replace the server rather than the disk?
There were already plans to replace the server, this failure just accelerated them. We were also of the opinion that having the disk replaced and getting the array rebuilt would likely not end up being faster, especially as we did not have sufficient information to pinpoint the failure.