Who are we?
We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.
We are one. We are legion. And we're trying really hard not to forget.
-- 5-4-3-2-1-bang from this thread
Yeah who needs proper DR. it's not like disasters ever happen.
Disaster Recovery Concepts:
My favorite is places that don't want to allocate time or resources to testing disaster recovery.
Wow. this is one of the biggest instances of IT incompetence that I've heard of in recent years. Hosting a server farm without remote backups? Sound like the London Magnetic Tape Incident.
The LMTI: One employee was sent to the other end of London with the magnetic backup tapes every day. He got money to take a taxi, but saved it and took the tube. His favourite seat was right above one of the motors, where he sat the bag with the tapes on the floor for the journey. The tapes were just stored at the destination, and not checked in any way. Guess what they learned the first time they had to rely on those tapes?
That's really funny. It's such a specific situation too. Almost like a comedy sketch
Imagine the reaction of the guy when he found out.
Perhaps contained evidence against the former PM who tried to do a military coup and this is the way of getting rid of it? Perhaps protecting co-conspirators if not the top guy.
As a sysadmin that's pretty much my worst nightmare. I really feel for those trying to pick the pieces back up again after that.
A nice clean start on a fresh install.
I feel your pain, but meanwhile I'm laughing. No backups, no fire suppression system, what the fuck are these clowns doing.
Suppressing evidence? Looks like this was a deliberate oversight to later erase data.
Probably data retention safety was overruled earlier by one person.
So, I think the designed erase worked as designed .
Hanlon's razor mate. I know it's entirely possible that it's a conspiracy, but it'd entirely possible (and more likely) that it's just incompetence and lack of communication due to inefficient and lackluster processes as well as too many levels of leadership.
Pretty big conspiracy theory for a situation where incompetence doesn't seem out of the ordinary.
SysAdmins saying "We need X, we need Y", someone who signs the cheques saying "But it works fine! Why would there be a fire? We already spent so much on those UPSs"
I don’t know a lot at this time. The cool thing is this will be discussed for years , and any who follow this will know a lot more.
I’m just saying if it looks like a duck, quacks like a duck it may not be a squirrel
What you described isn't "looks like a duck, quacks like a duck".
This is a fire in a government (publicly-funded) location in an industry that's often overlooked and underfunded... in a traditionally conservative country (with a Liberal leader for the last few months) that is part of a very top-down hierarchical structure. Which would suggest that the complaints of a systems administrator are likely to be ignored. And that's assuming that the systems administrator is competent, which isn't a guarantee in and of itself.
I'm seeing what looks like a duck, I'm hearing the quacks. The conspiracy theory is that it's a squirrel in a duck costume that learned how to quack so people would toss it bread.
Here we have a glass of water, it’s half full.
Did someone forget to fill it up, or did someone drink it.
Is it institutional incompetence or criminal conspiracy?
I don't think changing the analogy makes your argument stronger
I don’t have any argument, just my natural suspicion only.
But you likewise are thin on facts, your natural inclination is to see this as more a system issue.
Either we will never know , or one of us will be correct later
Bingo!
I don't know about deliberate, but it certainly crossed my mind.
Hmm a system that stored government documents caught fire and they have no backups? Hmm, this carries the same energy of registry offices catching fire "spontaneously" in the past.
Korean government's
South.
(Just in case anybody else isn't a clairvoyant either)
I'll be honest, I assumed it was the South because I would be honestly shocked if the North ever reported anything bad happening to itself.
The north doesn't need backups because errors do not happen there /s
There are no backup failures in Ba Sing Se
“Cloud storage” in the North is just a drawer full of usb sticks confiscated from people smuggling K-dramas in from the South.
The DPRK has their own intranet called kwangmyong, which needs its own storage and servers. Citizens use it regularly, access to the broader global internet however is more controlled and limited to specific jobs and positions as far as I know.
I figured it had to be south, as north would definitely have backups
At least they saved the Internet box. Otherwise, we'll have to bother the elders of the Internet again.
The fire reached Big Ben?
At a sea world??!!
Always have two backups in different places than the original. If not, the least you can do is have one backup copy. How does the government don't have such thing?
I have one backup but keep legacy things so that in a massive disaster I still have archives that have some of my important long term type documents. So figure one up to date backup and in a disaster I have stuff from last year.
Government people get jobs by schmoozing and making deals, not by merit or skill.
Pournelle's Law always seems relevant.
sounds like you are suggesting that in private companys jobs are distributed by merit and skills lol
No.
But it does give more options to work around or not support those entities since they don't have as much direct authority.
Ooh, ooh! I'm in that law! I'm in the (to paraphrase) "competent and devoted to the goal but unempowered" group!
This is a clear example where 'the cloud' really is a physical place.
What else would it be? What do people think the cloud is?
For a long while, I had hoped it was at least 6 physical places, with various redundancies. A few billion small-ish servers at internet network hubs.
That or the magical floating bits that go over hackers heads in the movies. Those also look like the cloud. Not very secure, but quite convenient.
Well, a properly managed cloud storage service very much should be multiple locations with redundant copies of data lol yes. So you're not wrong in that.
Too slow storage to back up? What a stupid, false reason. I assume nobody works at night. Do something else than full backups and you at least have something. A simple differential update replication would have saved them here.
Or do it by priority. Files that change often or are very important are copied fully often, maybe daily. A differential update of all files could follow daily or who knows weekly. Its the government, they should have money to add more storage, so that shouldn't be the problem. At least some strategy to manage slow speed, instead not having ANY backup its the dumbest thing I've read from governments in a while.
Always backup!!! From the fire, I mean, place your safety first
Kein Backup, kein Mitleid!
It's too bad it wasn't a rain cloud or it would havd destroyed the fire and not the other way around
So this is the "maintenance" the gov services are currently having.