I'm holding in the palm of my hand the sum of all human knowledge (lemmy.world)

submitted 7 months ago by droning_in_my_ears@lemmy.world to c/casualconversation@lemmy.world

14 comments fedilink hide all child comments

Well not quite but close. I'm holding a hard disk that has ALL of Wikipedia's text in 10 different languages.

Yes you can download all of Wikipedia and yes it can easily fit in a hard drive. Isn't that amazing? Text is incredibly dense compared to images and video. Around 22 GiB for English Wikipedia alone and 56 GiB for the 10 languages I downloaded.

I also have all of Wiktionary in the same hard drive. It's around 16.4 GiB.

you are viewing a single comment's thread
view the rest of the comments

[-] droning_in_my_ears@lemmy.world 9 points 7 months ago

It's only the raw text in json line files. No media and no markup. I think I downloaded a compressed dump then used wikiextractor to extract the text.