Just make a simple script (with bash or something) and go through your post/comment history via the API. Save the JSON of the posts/comments somewhere.
this post was submitted on 10 Jun 2025
20 points (100.0% liked)
New to Lemmy
785 readers
1 users here now
Did you just join Lemmy? This community is for you!
Do you want to help new users around? Then this community is for also you!
Thank you all for being here, it makes a big difference
FAQ
- You don't have to post here, but it's fun!
- You don't have to post right away, come back later if you want
- You can post as many times as you want
I don't know what to post
- Share what you're looking for, or what you want to do. We'll help you find it
- Ask questions! The Fediverse is new to most people. If you're unsure about something, others are unsure as well.
- Just smile and wave, maybe someone will wave back π
founded 2 years ago
MODERATORS
βJustββ¦
Rest of the fucking owl moment.
My coding skill is worse than my search-fu. While I can make a "hello world" script with bash, that's about my level of coding skill.
However, let's assume that I'm willing to still do it. I have this coming weekend to do the following:
- Study the Lemmy API with the aim of extracting the JSON of:
- all of my posts
- all my comments and the posts where they were made
- Create the logic:
- extracting the posts
- tracking the posts where the comments were made, and then extracting them
- Write the script.
I think the worst of it would be studying the API, but programming using Bash (or Phyton--which I am totally ignorant of, but might be better for handling the data) might also give me trouble.
I'm going to need a lot of luck if I'm going this route. I dunno if it's better than just doing it manually.
You don't need to "study" the API. The only thing you need is the /user API endpoint: https://mv-gh.github.io/lemmy_openapi_spec/#tag/User
Hmm, reading the /user API endpoint, I think I understand some words (but hardly). βThanks!
- you create a function with a page parameter and call the /user endpoint with a bearer auth header. page argument should be 1 at first call.
- using some library (or I think requests already has deserialization of JSON) you convert the JSON into some pythonic data type (e.g. dictionaries).
- exit the function when the length of posts and comments is 0.
- otherwise loop through the posts and comments arrays with a for loop
- on each iteration, serialize the dictionary to JSON and save it to some file
- increase the page counter by 1
- call the same function at the end of the function and pass in the page counter (which was increased by 1 previously). this is called recursion.
Thanks!! I think can get started with this--like this weekend.
But this makes me wonder, if it is as easy as it seems to be, why is this not already a thing?
because nobody needed such a thing until now?
I mean, as long as the bigger instances (that are also federated with lemm.ee, e.g. lemmy.world) stay, then your content is already "archived". Unless you choose to delete your account explicitly (which probably some instances will ignore anyway), your content will stay.
Good point. I am actually not so certain if my posts and comments will remain visible after lemm.ee goes, so I am thinking of doing this as a backup.
all of the comments, including any deleted ones (deleted by the user and deleted by mods/admins)
You probably will not have access to this.
Is there any other method I can do this without resorting to manual saving?
Only if you are an Admin. https://join-lemmy.org/docs/index.html
The data seems to be stored in a postgres database, and could technically be queried.
And if I have no choice other than to save each and every post and comment manually? How should I be doing it?
It depends on your need, if it is for legal purpose, it depends on your jurisdiction. Maybe web archive is enough for this case, https://archive.org/.
Otherwise, you can use the API as others mentioned. Or you can use Selenium/Scrapy/BeautifulSoup to scrape the website.
If you would rather not program something with code, look into browser add-ons that scrape websites, they are mostly visual, and you click on the things you want to save or navigate into. I am not familiar with them to recommend you something, but there are plenty of videos on how to use them.
That said, depending on the security of your instance, your ip/account might get flagged.
Talk with the Admins of your instance first and express your intentions, maybe they can help with what you need.
I'd rather not bother my admins (they're already burnt-out), and with what you just said, maybe I'm better off doing it manually--if I would do it afterall. TBH, I'm scared off by the impression that saving my own posts and comments is somehow taboo.
Thanks!
I don't think it would be taboo to save something that is public available, as you can just as simple visit those pages and print them to pdf for example.
The same goes to things you have access with your account, and it is not bound by a non-disclose agreement (you can still save, but not broadcast it - depending on the laws of your jurisdiction).
I looked into the lemm.ee default profile page, and you might have success with the tools I mentioned.
You just need to navigate these links, and make the tool open the post names.
https://lemmy.ca/u/megane_kun@lemm.ee?page=1&sort=New&view=Posts https://lemmy.ca/u/megane_kun@lemm.ee?page=1&sort=New&view=Comments
You will need to take in account navigation with "next" buttons, and some pages need to be scrolled down to load all the comments.
If you don't want to contact the Admins, and you are not pressed for time, put a delay on the tool's web requests to not overload the servers. The browser add-on might avoid that because it mimics a more natural way of navigating a website.
Yeah, I've had a think after I made my previous reply and the questionable part mentioned isn't the archiving, but the inclusion of the deleted/removed comments.
I still haven't started with the script mentioned in a different comment, but if I were to do it, I'd likely be putting a one second delay on every request.
Thanks!
-
Why? My comments at a contribution to a public discussion, saved just for me they do little for their original purpose.
-
given the nature of Lemmy, perhaps you could run your own instance that you could use to fetch every thread you've participated in, and back up that way?
In my response to someone, I already expressed concern about their long-term visibility--which is the reason why I even thought of doing this. I can't do anything about "contribution to a public discussion" if those very discussions pretty much disappear. This might be showing my lack of knowledge about what federation actually is, and if it is, it's pretty much on me.
For the second point, I already addressed it in the OP and in one of my responses. I'm pretty much not a programmer and my knowledge about it is pretty much writing a "hello world" program.
Follow-up, I peeked at your profile and I think I've found what you're intent on preserving, posting here to keep the record alive:
I remember when I first heard the rumours and I immediately thought of how sensitive one's anal linings would be to perceive Morse code via a butt plug. Then pondered upon the max possible bandwidth of buttplug-mediated information transfer. Finally, I thought about how to send back information via rythmic anal clenching.
Only then did I conclude that it's probably easier to get better at chess.
https://sh.itjust.works/comment/3662094
PS, I could be wrong, but I think your comments should continue to exist through the federated record of other instances. I don't think they purge their history whenever a server goes offline.
Got me!
I am still into that research. Yes. Research. For science!