this post was submitted on 25 Mar 2025
43 points (97.8% liked)

The Back End

74 readers
1 users here now

This community is used for less sociable site announcements.
Things like scheduled downtime, information about upgrades, or asking technical questions you'd like to know about how things are done (within reason).

Photo credit

founded 1 year ago
MODERATORS
GreatAlbatross@feddit.uk
43
Post-mortem: 25/03/2025 unplanned outage (feddit.uk)
submitted 2 weeks ago by GreatAlbatross@feddit.uk to c/backend@feddit.uk
22 comments fedilink hide all child comments
 

This was very much a me mistake, so apologies from me.

The long story short, is that we were migrating the domain between accounts.

As the previous DNS records would be wiped during transfer, we made backups, and started the process.
This morning, the "approve transfer" button was lit.
A sensible person would have gone "perfect, lets get this scheduled".
Instead, I just pressed the button.

Unfortunately, when you transfer, and are keeping continuity, you should always make sure the receiving account is ready to re-apply the settings.
I didn't, and the receiving account owner was happily asleep. 100% my bad!

All sorted now (hopefully).
There might be some weirdness while DNS updates propagate through the various caches.

I think the lessons learned are pretty obvious:

  • Consider how noticeable downtime will be, and schedule with this in mind.
  • Check that your counterpart is online when transferring a domain.
  • Always schedule a maintenance window, even if you get excited when the button activates.
top 22 comments
sorted by: hot top controversial new old
[–] Hossenfeffer@feddit.uk 14 points 2 weeks ago

Over many years in a huge variety of tech roles, I'd like to add an additional rule:

  • Always schedule an update or re-configuration on a Friday afternoon, preferably during a bank holiday weekend.
[–] Zip2@feddit.uk 13 points 2 weeks ago

We’ve all done similar! Thanks for all that you do.

[–] Nighed@feddit.uk 9 points 2 weeks ago

Learning 4: it's always DNS

[–] sh3llcmdr@feddit.uk 8 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

I've got lots of stories of "should not have done that now" in IT and I would never publicly admit to any of them. Fair play for getting it all sorted. And respect due (Stouffer the cat style) for falling on your sword!

[–] BeardedGingerWonder@feddit.uk 2 points 2 weeks ago

I remember my first release, it taught me "better" is not the same as "right"

[–] a_wild_mimic_appears@lemmy.dbzer0.com 7 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

Those nice, green, welcoming looking buttons are to blame! Who doesn't like to push GO-Buttons?

[–] i_am_not_a_robot@feddit.uk 3 points 2 weeks ago

And red buttons which are clearly labelled "DO NOT PRESS"!

[–] wewbull@feddit.uk 5 points 2 weeks ago

Thanks for the transparency. It's appreciated even though you're doing this of your own volition. I'm grateful for your efforts regardless.

[–] BeardedGingerWonder@feddit.uk 4 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

I will not stand for this! Good job I'm sitting down, eh 😋

Edit: Genuine question - what are you guys like for coverage in general, I kind of saw it fall apart earlier and was thinking "I hope they're not sitting at work worrying about Lemmy and not being able to do anything about it"

[–] Emperor@feddit.uk 3 points 2 weeks ago (1 children)

It wasn't really a coverage issue. Once the sender hit the big button labeled "are you really sure about this?" only the recipient could sort it out.

[–] BeardedGingerWonder@feddit.uk 1 points 2 weeks ago (1 children)

Sorry, I meant that more as a general side question that had occurred to me earlier when the issue was happening. You guys do a fantastic job, thank you.

[–] GreatAlbatross@feddit.uk 3 points 2 weeks ago (1 children)

We also tend not to fret too much.
We have backups, and while we try to be professional and manage downtime, I think people are pretty understanding.

[–] BeardedGingerWonder@feddit.uk 2 points 2 weeks ago

Good! Genuinely appreciate the effort you all put in to bring us this place!

[–] chickenf622@sh.itjust.works 3 points 2 weeks ago

Always love a post-mortem write up. We all make mistakes, and sharing this info helps others avoid the same mistakes. Thanks for sharing and recognizing you dun goofed.

[–] Duckxon@feddit.uk 3 points 2 weeks ago (2 children)

By the way how can we support financially whoever is hosting/taking car of the server?

[–] blackn1ght@feddit.uk 3 points 2 weeks ago (1 children)

People can donate via OpenCollective 😊

[–] Duckxon@feddit.uk 2 points 2 weeks ago

Thanks!

[–] Emperor@feddit.uk 2 points 2 weeks ago

You already have the link but it is also in the sidebar.

[–] flamingos@feddit.uk 2 points 2 weeks ago (2 children)

Semi-related, but I'm now the proud owner of durglesmurfgolb.co.uk (the domain register wouldn't let me create an account without buying something), any suggestions on what to do with it.

[–] GreatAlbatross@feddit.uk 3 points 2 weeks ago

I think we just found our new status address.

[–] Emperor@feddit.uk 2 points 2 weeks ago

So you bagged that before I could!

[–] Flax_vert@feddit.uk 2 points 2 weeks ago

I demand a full inquiry!!!! Heads will roll!!! I actually had to do something other than leech off of a lemmy instance!!!