Hi everyone so I have good news. The outage on LGBTQIA.is has been resolved and the site is almost fully operational again. This post describes the recovery progress / steps.
So far I’ve been able to reinstall Mastodon, and restore the postgresql database, regenerate feeds, and reconnect Wasabi S3 for image/video uploads. Additionally I can message accounts on other instances without issue (federation works again).
Somethings are temporarily broken
Unfortunately due to a misconfiguration in the backups system the
.env.production file was lost. As a result certain data is permanently lost as I no longer have the encryption keys to decrypt it. This means that you will need to re-enable two-factor-auth. I have cleared two-factor-auth data from the database. This approach worked for my account and has been applied to all accounts in the database. Additionally I need to generate a new set of SMTP Credentials to fix account confirmation and password reset emails. Finally status lengths are limited to 500 until I can reapply my changes to the limit.
Things will get better over the next week
Unfortunately everything won’t immediately be back to normal and a lot of debugging and fixing is still going to occur at random intervals. Please expect temporary outages as I make configuration/code changes. I’m working to quickly resolve the situation and will do my best to get things back to normal ASAP. Meanwhile enjoy the instance and know that your data remains safe and secure.
Still having issues?
If you are still having issues accessing your account for some reason please send an email to
email@example.com and I will get back to you as soon as possible to assist in recovering your account.