this post was submitted on 08 Oct 2025

46 points (100.0% liked)

Pawb.Social Announcements

510 readers

1 users here now

Official announcements from the admin team of Pawb.Social

founded 2 years ago

MODERATORS

crashdoom@pawb.social

natebluehooves@pawb.social

itsOasus@pawb.social

Doridian@pawb.social

southernwolf@pawb.social

[PSA] Pawb.Social Data Loss - 10/08/2025 (pawb.social)

submitted 1 week ago* (last edited 1 week ago) by crashdoom@pawb.social to c/pawbsocial_announcements@pawb.social

15 comments fedilink hide all child comments

EDIT, Update: We've been able to partially recover about 25% of the media uploaded to Pawb.Social.

Upon further investigation, data from pawb.social was priorly moved to the new server and was inadvertently deleted as part of the earlier incident affecting furry.engineer and pawb.fun's cache while we were preparing the new server.

furry.engineer and pawb.fun media is unaffected, but all media directly uploaded to pawb.social is unrecoverable and has been lost. A miscommunication on my part OK'd the removal without fully checking the scope of data that was being removed; This was intended to only remove test data we were using to stress test the new hardware, but affected live data as well.

A large majority of posts are externally hosted and referenced (e.g. images from media.kbin.earth, d.furaffinity.net, etc.) are NOT affected. Only posts directly uploaded via the Create Post > Image field on pawb.social are affected.

We had previously transferred pawb.social to the new hardware as a preliminary trial and I had forgot this occurred thus failing to bring it up prior to the deletion going through.

I'm looking into options to be able to in-place reupload media on behalf of posters, but as the UUIDs are randomly generated by Pict-rs (the media service used by Lemmy), I'm unsure how to facilitate this. Any advice or information would be appreciated, and can be sent to network@pawb.social.

I understand this is a serious incident and reflects poorly on us, and I can only apologize for this loss and work to prevent this happening again in the future.

EDIT: I believe this will also impact profile pictures and community images (logos and banners), so please re-upload them at your earliest convenience.___

all 16 comments

sorted by: hot top controversial new old

[–] crashdoom@pawb.social 11 points 1 week ago (2 children)

ℹ️ Thanks to the efforts of one of our moderators, @southernwolf@pawb.social, we've been able to recover some of the lost data from pawb.social that was still active in Cloudflare's regional caches.

Currently, this is about 25% of all of the known uploaded media files stored on pawb.social.

We'll be reviewing the final tally of recovered data in the morning and trying to work on getting what we can reimported into Pict-rs.

[–] southernwolf@pawb.social 6 points 1 week ago (1 children)

For those who see this before we release the final tally: As it stands right now, I've recovered ~1/3rd of media posts via Cloudflare caching. I'll have a final total when I'm truly done, but honestly that's pretty damn good to have been able to rescue all of that via cache recovery alone.

[–] southernwolf@pawb.social 6 points 1 week ago (2 children)

This is subject to change, but here is the preliminary tally:

Recovery report generated:
2025-10-09 08:58:47,515 - INFO -   - Total entries in CSV: 6697
2025-10-09 08:58:47,515 - INFO -   - Images recovered: 2201
2025-10-09 08:58:47,515 - INFO -   - Images missing: 4496
2025-10-09 08:58:47,515 - INFO -   - Recovery rate: 32.87%
2025-10-09 08:58:47,515 - INFO -   - Total size: 1501.68 MB (1.47 GB)

[–] DarrenTheFoxcoon@pawb.social 2 points 1 week ago (1 children)

Would the Pawb staff be open to using volunteers as secondary backups? A few GB seems like absolutely no problem for my own rig

[–] natebluehooves@pawb.social 3 points 1 week ago* (last edited 1 week ago)

Amusingly we do have backup infrastructure on site. 30tb pawb nas, 80tb nas for backups, and lto tape backups periodically. my problem is getting it all set up. i’ve been battling depression and trying to catch up on housework.

the issue here was doing a partial migration and allowing a couple months to pass before trying to finish, by which point I had forgotten pawb was moved.

we plan on pawb to take a lot more storage as time goes on. we are at about 1.5tb plus ~450gb database size right now.

edit: to make some sense of this: the 1.5tb includes cached media from other instances. our users don’t use a lot of storage yet.

[–] southernwolf@pawb.social 1 points 1 week ago

Please see an update for this situation here: https://pawb.social/post/33046379

[–] Kolanaki@pawb.social 16 points 1 week ago* (last edited 1 week ago)

My face was so ugly in the pic I uploaded, that the server deleted every single picture just to be sure. 😞

[–] KoboldCoterie@pawb.social 16 points 1 week ago

No worries, mate. Thanks for the heads up that it happened; you're providing a free service to us, we aren't going to complain about the occasional hiccups, especially considering how infrequent there've been any.

[–] Kovukono@pawb.social 14 points 1 week ago

You guys are running a free server with 99.9% uptime and barely any notable issues in the time I've been here. Losing some pictures in a migration over a mistake is not as serious as you think it is. You guys do amazing work, and we love you for everything you've done.

[–] alex@pawb.fun 9 points 1 week ago* (last edited 1 week ago)

@crashdoom 🫂 (no hug react on Telegram so you're getting one here)

[–] Ethanol@pawb.social 8 points 1 week ago

This is fine, you're doing a great job. Mistakes happen and re-uploading my profile picture or banner is not a hassle at all. No need to beat yourself up. Thank you all for the work you put into this instance :3

[–] Wahots@pawb.social 7 points 1 week ago

Happens to the best of us! My work just nuked a ton of data in a similar incident, so you're in good company, haha.

[–] LordAmplifier@pawb.social 6 points 1 week ago

It's a bit of a bummer to see some of the "history" of pawb disappear, but shit happens. You're providing one hell of a good service completely for free, and that's cool as heck. Thank you for the transparency and all the good work you do behind the scenes <3

[–] Dojan@pawb.social 6 points 1 week ago

Sometimes, shit happens. We've all been there, one way or another. Thanks for all your hard work.

[–] Foxfire@pawb.social 5 points 1 week ago

No worries, good to know the cause of the image issues. I'll simply edit my posts to reupload the missing data since it was all my own art anyway.