So, My Website Melted!

It literally melted.

I've a bunch of personal websites, one of which is a busy enough motorsport forum as well as a photography websites and a bunch of other sites for family and friends (some of which actually get traffic)

To host these I've a dedicated server running Citrix XenServer with something like 11 guest Linux servers (CentOS 7)

These were provisioned as 2 MySQL database servers (primary / replication) and individual web servers to partition the various projects.

Early last Wednesday morning (just after midnight) the external monitoring company I use (NodePing) reported that one, then two, then more of the sites were down / unaccessible.

This has happened before and when I couldn't access the server through XenCentre I did what I usually do, request a reboot and go back to sleep and deal with it in the morning.

8am and I'm up as usual, and get a message from a friend and I find out.....

https://www.reuters.com/article/us-f...-idUSKBN2B20NU

So, yes, my websites have physically melted.

Time to activate the DR plan.

Backups:

The MySQL replication server takes nightly consistent MySQL dumps of the data and stores a week worth on disk. Great, that server has melted too.

However, a (Old!) Raspberry Pi sits in my attic, with an iSCSI connection to a TerraMaster NAS and at 3am it uses Dirvish to do rsync backups from the production web servers and the MySQL replication server.

Restoration:

Assume that OVH aren't going to be in any position to provide infrastructure.

1. Ensure the backups don't get over-written, stop all cron jobs that deal with backups and copy the 200+Gb of data from the Backup Pi to local disk on laptop.
2. As my domain registration provider is also my DNS provider I configure the primary domain to redirect to the relevant Facebook page where I can issue status updates.
3. Commission a VPS from a totally different provider. I picked one that I happen to use for domain registration.
4. Provision a small "web hosting" instance from the same provider.
5. As the web hosting is effectively instantly available create a "Yes, we're offline, the server melted, follow updates on FB" page.
6. Configure the VPS with the relevant version of MySQL
7. Upload the MySQL dump of the forum data and start running it in to MySQL (I know this will take time...)
8. Start copying the forum code, static and attachment files up to it for the main website (as it's 30Gb I know it's going to take a while).
9. Identify which sites are really in use and which don't need to be restored at the moment.
10. Upload and run in the MySQL dumps for a site (Wordpress sites with relatively small amounts of data)
11. Upload the site code.
12. Change the DNS for the site.
13. Request SSL certs from Letsencrypt.
14. Change the MySQL connection details on the site and make it "live".
15. Repeat steps 10-14 as necessary.
16. Same steps regarding SSL for the main site.
17. Wait for the file rsync and MySQL imports to finish.
18. Configure database connection.
19. Check site functionality (.htaccess deny connections from anywhere except my house)
20. Adjust for things that are broken under PHP version.
21. Make site public.

Total downtime for the main site, around 23 hours from when the problem started, and 14 from when I started the DR plan. Most of that was waiting for the files to upload and the database to import. Other sites were available around 3 hours after initiating DR.

I've a few take-aways from the experience, main one being I'll likely have another external internet based backup, as that'll likely be faster to copy over than via the house link.

The moral of this tale.

Ensure you have valid backups and a DR plan, and TEST YOUR BACKUPS regularly!