I'm holding in the palm of my hand the sum of all human knowledge (lemmy.world)

submitted 7 months ago by droning_in_my_ears@lemmy.world to c/casualconversation@lemmy.world

14 comments fedilink hide all child comments

Well not quite but close. I'm holding a hard disk that has ALL of Wikipedia's text in 10 different languages.

Yes you can download all of Wikipedia and yes it can easily fit in a hard drive. Isn't that amazing? Text is incredibly dense compared to images and video. Around 22 GiB for English Wikipedia alone and 56 GiB for the 10 languages I downloaded.

I also have all of Wiktionary in the same hard drive. It's around 16.4 GiB.

top 14 comments

sorted by: hot top controversial new old

[-] henfredemars@infosec.pub 27 points 7 months ago

It also connects you to a huge swath of humanity and the editors that brought that content to you.

[-] droning_in_my_ears@lemmy.world 19 points 7 months ago

Yeah it's pretty incredible. Wikimedia is the kind of project that almost feels like a small glimpse into a better world. What the internet could have been. It's got some problems of course but it's still a huge success.

[-] intensely_human@lemm.ee 1 points 6 months ago

Uh, wikipedia is what the internet is.

Wikipedia’s not a glimpse of a better world, it’s a glimpse of our current, existing world. Because wikipedia exists.

It’s not like that hard drive came through a portal from another universe.

[-] penquin@lemm.ee 12 points 7 months ago

You're going to be the savior of humanity after the apocalypse

[-] trolololol@lemmy.world 10 points 7 months ago

Not the sum. The summary.

[-] Masterblaster@kbin.social 7 points 7 months ago

there's still so much valuable academic information that never sees the light of day, or gets erased as the internet serpent eats its own tail.

[-] WarmSoda@lemm.ee 6 points 7 months ago

Last time I looked into downloading Wikipedia it said it was 50gb for English text and 100 with images. How'd you get it for half the space?

[-] droning_in_my_ears@lemmy.world 9 points 7 months ago

It's only the raw text in json line files. No media and no markup. I think I downloaded a compressed dump then used wikiextractor to extract the text.

[-] AbouBenAdhem@lemmy.world 2 points 7 months ago

Does it include each article’s edit history, talk page, etc?

[-] ace_garp@lemmy.world 2 points 7 months ago* (last edited 7 months ago)

The dictionaries for Aard2 are 21gb in .slob compressed format (text only).

[-] WarmSoda@lemm.ee 2 points 7 months ago

No idea what that means. But thank you for adding more info.

[-] ace_garp@lemmy.world 3 points 7 months ago

OK yes, some supporting info is: Aard2 is an offline wikipedia app, that uses small compressed data files in .slob format.

[-] intensely_human@lemm.ee 1 points 6 months ago

Slob compression is best visualized as putting a sleeping bag into a stuff sack, except it’s all your possessions and you’re stuffing them into an old Chevy Metro