Where did everything go?!
Important Update 2022-12-04 02:00-ish UTC: The old server is back online so I'm taking a fresh copy of everything and moving it to the new server. So no data is lost but there will be a couple more hours of downtime while it copies across and gets updated for the new database server (MySQL 5.7 -> MariaDB 10.9).
The code fixes are in place and posting and commenting work, so we should be reasonably stable as soon as that finishes.
Database has now been resynced and indexes are rebuilding (necessary to bring the tables into the new database version. Any files uploaded recently have also been copied.
We've been having trouble with the server for some time, and I arranged a new server and was set to start the migration today.
Unfortunately, the datacenter hosting the old server is doing some (unannounced - or at least unannounced to me) scheduled maintenance that is taking much, much longer than expected and right now I can't get to the old server to migrate things to the new server.
I'm currently loading up the last full off-site backup (three weeks old, sigh) and will bring that up if the maintenance takes three weeks itself, like it did with the datacenter fire last year.
It will take a little while to set up, as we've jumped forward three full versions of Linux and I have to manually tweak a bunch of stuff, but the databases are already copied from the backup.
That's the worst case, but tech support have been telling me the server will be back online in about an hour for the past six hours, so I'm starting to expect a worst case scenario here.
Update 2022-12-03 12:00-ish UTC: Main files are synced from backup. Remaining files (archives, logs and so on) are copying now. I need to get a few hours sleep before I tackle bringing sites back online.
We did have automated daily offsite backups - but that backup server turns out to have a corrupt ZFS volume and I can't get the backups back. So we have to go back to the most recent manual offsite backup from November 10. Tech support are still working on bringing the original server back online but I don't have an ETA.
Update 2022-12-03 13:00-ish UTC: Networking problems on the planned new server. Fortunately I have yet another server. And now two copies of that manual backup.
Update 2022-12-03 14:00-ish UTC: Running a data check and index repair on the database backup on the second new server. All looks good so far. I really do need to get some sleep now. Maybe when I wake up the server will be back. If not, I'll continue getting the backup server up and running.
Update 2022-12-03 21:00-ish UTC: Database on the new server looks good. Running through the code changes needed to bring the blogs back up on a much more current release of Linux, Python, and all the associated libraries. The code is in a running state
but not properly serving websites yet and serving websites internally, but not public yet.
Update 2022-12-03 22:00-ish UTC: All user-uploaded files are restored (again, up to three weeks ago) and file serving is working internally. I won't enable file uploads at first, I'll probably do that tomorrow. Now I just need to set up a proxy server so the sites are visible and do some more testing.
Update 2022-12-03 23:00-ish UTC: Configure a basic proxy for the front end. Database updates to allow a merge of the missing data are running now (testing on a snapshot). Waiting for DNS to propagate (~10 minutes) and then my own site should be loading and I can test further. ETA for full operations 3-4 hours.
Had some problems with automatic SSL certificates. Getting a wildcard certificate now to resolve that.
Update 2022-12-04 00:00-ish UTC: SSL issues are sorted out. My own site - three weeks old - is loading fine with images intact. Continuing testing.
ETA 2 hours.
Update 2022-12-04 01:00-ish UTC: Posting and commenting are working, you can log in as usual, sessions are working. I'm going to start firing up more worker processes and get this show on the road.
ETA 1 hour.
We will be back on line today. I'm going to adjust the database sequences so that if/when we do get the old server back I can easily merge the missing posts since the manual backup.