452
submitted 6 months ago by Moonrise2473@feddit.it to c/technology@lemmy.ml

A week of downtime and all the servers were recovered only because the customer had a proper disaster recovery protocol and held backups somewhere else, otherwise Google deleted the backups too

Google cloud ceo says "it won't happen anymore", it's insane that there's the possibility of "instant delete everything"

you are viewing a single comment's thread
view the rest of the comments
[-] heluecht@pirati.ca 31 points 6 months ago

@Moonrise2473 Regardless of one thinks about "cloud" solutions, this is a good example, why you always should have an offsite backup.

[-] Hirom@beehaw.org 5 points 6 months ago

They had backups at multiple locations, and lost data at multiple (Google Cloud) locations because of the account deletion.

They restored from backups stored at another provider. It may have been more devastating if they relied exclusively on google for backups. So having an "offsite backup" isn't enough in some cases, that offsite location need to be at a different provider.

[-] heluecht@pirati.ca 6 points 6 months ago

@Hirom With "offsite" I mean either a different cloud provider or own hardware (if you hold your regular data at some cloud provider, like in this case).

[-] Hirom@beehaw.org 1 points 6 months ago

That would indeed be a good backup strategy, but better be specific. "Offsite" may be interpreted in different ways.

[-] Tangentism@lemmy.ml 2 points 6 months ago

It may have been more devastating if they relied exclusively on google for backups.

Which is why having any data, despite the number of backups, on a cloud provider shouldn't be seen as off-site.

Only when it is truly outside their ecosphere and cannot be touched by them should it be viewed as such.

If that company didn't have such resilience built into their backup plan, they would be toast with a derisory amount of compensation from Google.

[-] Hirom@beehaw.org 2 points 6 months ago

Having a backup at a cloud provider is fine, as long as there is at least one other backup that isn't with this provider.

Cloud provider seems to do a good job protecting against hardware failure, but can do poorly with arbitrary account bans, and sometimes have mishaps due to configuration problems.

Whereas a DIY backup solution is often more subject to hardware problems (disk failure, fire, flooding, theft, ...), but there's no risk of account problem.

A mix is fine to protect against different kind of issues.

[-] Tangentism@lemmy.ml 2 points 6 months ago* (last edited 6 months ago)

as long as there is at least one other backup that isn't with this provider.

Which is exactly what I was saying.

Any services used with a cloud provider should be treated as 1 entity, no matter how many geo-locations they claim your data is backed up to because they are a single point from which all those can be deleted.

When I was last involved in a companies backups, we had a fire safe in the basement, we had an off-site location with another fire safe & third copies would go off to another company that provided a backup storage solution so for all backups to be deleted, someone had to go right out of their way to do so. Not just a simple deletion of our account & all backups are wiped.

That company had the foresight to do something similar & it's saved them. [edited - was on the tube when I wrote this and didnt see the autocorrect had put 'comment', not 'company']

[-] Hirom@beehaw.org 2 points 6 months ago

Okay, I misinterpreted your comment.

[-] Tangentism@lemmy.ml 1 points 6 months ago

No, it's all good. We're on the same page about disaster recovery!

this post was submitted on 11 May 2024
452 points (99.1% liked)

Technology

34917 readers
115 users here now

This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.


Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.


Rules:

1: All Lemmy rules apply

2: Do not post low effort posts

3: NEVER post naziped*gore stuff

4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.

5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)

6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist

7: crypto related posts, unless essential, are disallowed

founded 5 years ago
MODERATORS