this post was submitted on 25 Mar 2025
43 points (97.8% liked)

The Back End

74 readers
1 users here now

This community is used for less sociable site announcements.
Things like scheduled downtime, information about upgrades, or asking technical questions you'd like to know about how things are done (within reason).

Photo credit

founded 1 year ago
MODERATORS
GreatAlbatross@feddit.uk
43
Post-mortem: 25/03/2025 unplanned outage (feddit.uk)
submitted 1 week ago by GreatAlbatross@feddit.uk to c/backend@feddit.uk
22 comments fedilink hide all child comments
 

This was very much a me mistake, so apologies from me.

The long story short, is that we were migrating the domain between accounts.

As the previous DNS records would be wiped during transfer, we made backups, and started the process.
This morning, the "approve transfer" button was lit.
A sensible person would have gone "perfect, lets get this scheduled".
Instead, I just pressed the button.

Unfortunately, when you transfer, and are keeping continuity, you should always make sure the receiving account is ready to re-apply the settings.
I didn't, and the receiving account owner was happily asleep. 100% my bad!

All sorted now (hopefully).
There might be some weirdness while DNS updates propagate through the various caches.

I think the lessons learned are pretty obvious:

  • Consider how noticeable downtime will be, and schedule with this in mind.
  • Check that your counterpart is online when transferring a domain.
  • Always schedule a maintenance window, even if you get excited when the button activates.
top 22 comments
sorted by: hot top controversial new old
[–] Hossenfeffer@feddit.uk 14 points 6 days ago

Over many years in a huge variety of tech roles, I'd like to add an additional rule:

  • Always schedule an update or re-configuration on a Friday afternoon, preferably during a bank holiday weekend.
[–] Zip2@feddit.uk 13 points 6 days ago

We’ve all done similar! Thanks for all that you do.

[–] Nighed@feddit.uk 9 points 1 week ago

Learning 4: it's always DNS

[–] wewbull@feddit.uk 5 points 6 days ago

Thanks for the transparency. It's appreciated even though you're doing this of your own volition. I'm grateful for your efforts regardless.

[–] Duckxon@feddit.uk 3 points 6 days ago (2 children)

By the way how can we support financially whoever is hosting/taking car of the server?

[–] blackn1ght@feddit.uk 3 points 6 days ago (1 children)

People can donate via OpenCollective 😊

[–] Duckxon@feddit.uk 2 points 6 days ago

Thanks!

[–] Emperor@feddit.uk 2 points 6 days ago

You already have the link but it is also in the sidebar.

[–] BeardedGingerWonder@feddit.uk 4 points 1 week ago* (last edited 6 days ago) (1 children)

I will not stand for this! Good job I'm sitting down, eh 😋

Edit: Genuine question - what are you guys like for coverage in general, I kind of saw it fall apart earlier and was thinking "I hope they're not sitting at work worrying about Lemmy and not being able to do anything about it"

[–] Emperor@feddit.uk 3 points 6 days ago (1 children)

It wasn't really a coverage issue. Once the sender hit the big button labeled "are you really sure about this?" only the recipient could sort it out.

[–] BeardedGingerWonder@feddit.uk 1 points 6 days ago (1 children)

Sorry, I meant that more as a general side question that had occurred to me earlier when the issue was happening. You guys do a fantastic job, thank you.

[–] GreatAlbatross@feddit.uk 3 points 6 days ago (1 children)

We also tend not to fret too much.
We have backups, and while we try to be professional and manage downtime, I think people are pretty understanding.

[–] BeardedGingerWonder@feddit.uk 2 points 6 days ago

Good! Genuinely appreciate the effort you all put in to bring us this place!

[–] chickenf622@sh.itjust.works 3 points 6 days ago

Always love a post-mortem write up. We all make mistakes, and sharing this info helps others avoid the same mistakes. Thanks for sharing and recognizing you dun goofed.

[–] flamingos@feddit.uk 2 points 6 days ago (2 children)

Semi-related, but I'm now the proud owner of durglesmurfgolb.co.uk (the domain register wouldn't let me create an account without buying something), any suggestions on what to do with it.

[–] GreatAlbatross@feddit.uk 3 points 6 days ago

I think we just found our new status address.

[–] Emperor@feddit.uk 2 points 6 days ago

So you bagged that before I could!

[–] Flax_vert@feddit.uk 2 points 6 days ago

I demand a full inquiry!!!! Heads will roll!!! I actually had to do something other than leech off of a lemmy instance!!!

[–] sh3llcmdr@feddit.uk 8 points 1 week ago* (last edited 1 week ago) (1 children)

I've got lots of stories of "should not have done that now" in IT and I would never publicly admit to any of them. Fair play for getting it all sorted. And respect due (Stouffer the cat style) for falling on your sword!

[–] BeardedGingerWonder@feddit.uk 2 points 1 week ago

I remember my first release, it taught me "better" is not the same as "right"

[–] a_wild_mimic_appears@lemmy.dbzer0.com 7 points 1 week ago* (last edited 6 days ago) (1 children)

Those nice, green, welcoming looking buttons are to blame! Who doesn't like to push GO-Buttons?

[–] i_am_not_a_robot@feddit.uk 3 points 1 week ago

And red buttons which are clearly labelled "DO NOT PRESS"!