PSA: We're back, here's a post-mortem! (beehaw.org)

submitted 1 year ago* (last edited 1 year ago) by Lionir@beehaw.org to c/support@beehaw.org

87 comments fedilink hide all child comments

Hi Beeple!

Here's a vague version of events :

11PM EST: Lemmy.world got hacked
12:20AM EST: Blahaj.zone got hacked
12:25AM EST: I shut down the server
12:30AM EST: I make announcements to tell people about this
12:45AM EST: I have an idea of what the problem is but there is no fix
2:20AM EST: I go to sleep
8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied
11:40AM EST: We start testing things to make sure are working And well, now here we are.

If you have issues logging in or using an app:

Log out if you somehow are still logged in
Clear all cache, site data, etc.
Hard refresh Beehaw using CTRL+F5
Log back in.

If you still have issues, write to us at support@beehaw.org

To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.

Oh yeah, in case, you haven't, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw

you are viewing a single comment's thread
view the rest of the comments

[-] frogman@beehaw.org 57 points 1 year ago

shutting down the server early was best. the nature of open source software is what allows these incidents to be mitigated as quickly as they are. thanks a lot to you guys, and to all of the team at Lemmy who worked to resolve this.

heroes <3