31
Docker configs destroyed after update [resolved!] (lemm.ee)
submitted 8 months ago* (last edited 8 months ago) by Player2@lemm.ee to c/selfhosted@lemmy.world
11 comments fedilink hide all child comments

Hello selfhosted community, something weird just happened to my setup while running a routine update.

I'm running docker containers on a couple Debian LXCs through Proxmox, and a regular apt-get upgrade just wiped all my configurations. Somehow it seems to have gutted my databases and deleted the compose.yml files without a trace remaining. Thankfully all my data seems to be intact as far as I can tell.

Did I royally mess something up in all of my configurations or in doing the update? This has never happened to me before. Thankfully I have a backup for the configs that's about 6 days old, but it's still extremely annoying. Any hints? Thanks

top 11 comments
sorted by: hot top controversial new old
[-] Player2@lemm.ee 9 points 8 months ago

Update in case anyone is interested: I figured out what caused the problem. When I mounted the new drive I used to store my configurations onto Proxmox, I completely forgot to make the relevant /etc/fstab entry. The drive mounted successfully so I didn't realize at the time that I had forgotten to do that step. The update I ran from apt-get included a kernel update, so I restarted the machine to complete it. Since I hadn't modified fstab, my new drive was not mounted when the system started up again. Even though the drive wasn't mounted correctly, I still somehow had access to some incomplete version of the files in its directory (no idea how that works). So no fault of Docker, LXCs or Proxmox, purely PEBKAC.

Despite getting the files back I will still work towards a more resilient system and more regular backups.

[-] mikyopii@programming.dev 8 points 8 months ago

I'm not sure. All I know is that Proxmox doesn't recommend using Docker in LXCs. It seems Proxmox has much more control over the internals of the container and can break stuff.

https://pve.proxmox.com/wiki/Linux_Container

If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.

[-] Player2@lemm.ee 2 points 8 months ago

It was just a matter of time then until something broke? Guess I'll need to do some more research on how to best manage the services I want to run. Good thing I didn't come into this hobby hoping to reduce the amount of headaches

[-] towerful@programming.dev 2 points 8 months ago

I run a debian VM for hosting docker things.
Its like 500mb extra disk space, and a small ram/cpu hit. But its been solid for me

[-] Crogdor@lemmy.world 6 points 8 months ago

You didn’t happen to change an unprivileged container to privileged, or vice versa, after creating it, right? Doing so can break filesystem permissions, which could have resulted in something like this.

[-] Player2@lemm.ee 2 points 8 months ago

Interesting, I think I did actually do that. I guess my best bet for now is to just nuke the LXCs then and move everything to a new VM. Thanks for the advice!

[-] MangoPenguin@lemmy.blahaj.zone 5 points 8 months ago

That's pretty weird, where were the files stored that went missing? Just in your home folder?

Definitely recommend setting up regular daily backups, you can use the built in proxmox backup system, or run whatever tool you want inside each VM/CT.

[-] Player2@lemm.ee 1 points 8 months ago

The files were on a mountpoint, completely separate drive. This has also been the case for all the previous times I ran an update, though I did recently move these files to a different drive mounted the same way. I got some sort of permission wrong maybe?

Will definitely set up a better backup system as soon as I can

[-] MangoPenguin@lemmy.blahaj.zone 1 points 8 months ago

Very odd, I think something else is going on not related to LXC/Proxmox. I've got LXC containers with external mounts and internal storage and haven't seen any data loss over the years they've been running.

[-] tagginator@utter.online -2 points 8 months ago

New Lemmy Post: Docker configs destroyed after update (https://lemmy.world/post/12089867)
Tagging: #SelfHosted

(Replying in the OP of this thread (NOT THIS BOT!) will appear as a comment in the lemmy discussion.)

I am a FOSS bot. Check my README: https://github.com/db0/lemmy-tagginator/blob/main/README.md

this post was submitted on 18 Feb 2024
31 points (100.0% liked)

Selfhosted

39700 readers
553 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
ruud@lemmy.world
CannaVet@lemmy.world
Loki@lemmy.world
HybridSarcasm@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz