idk how i would define difficult, but the thing i probably put the most time into figuring out thus far is LXC containers.

Or LXC, if you like not using redundant acronyms. Those containers are good shit, weird shit, but good shit nonetheless.

At some point I've installed rust implementation of the coreutils from the AUR, they worked for a long while until some ssl vulnerability were discovered and everyone had to update the library. As you can imagine, without working coreutils system were hard to use. troubleshooting were also a pain in the ass because who could blame coreutils of all things? :P

A recent one:

/var was almost full and I ran pacman -Syu and left the comp to go and make dinner. This was also at the time Plasma 6 was rolling out.

It was a big upgrade along with a new kernel. Download seemed to go smoothly, but during installation, it didn't have enough space to unpack stuff and there was no kernel available to boot. Even the "previous kernel" options didn't work.

It wasn't too hard to fix because I had learnt how to use pacman in a chroot env, but my dinner got cold by the time I was ready to eat.

I still haven't learnt the lesson though. This is the third time I am having a problem with paccache and I still haven't setup a removal daemon/cron job.

is debootstrap considered reinstalling? because i'vedebootstrapped at least 2 systems to fix botched upgrades.

Using Linux on a GTX660 without proprietary drivers. I never managed to succeed. Desktop would always freeze. Never again.

For me it was migrating my Arch install from EXT4 to ZFS. GRUB had to be configured in particular ways to get it to work with ZFS and I didn't do it properly so it wouldn't/couldn't boot.

Then I updated ZFS to a version that wasn't supported by GRUB yet so I chrooted into my installation to switch to Systemd-boot with Unified Kernel Images. Now I still can't figure out how to add a boot entry for Windows. I followed the proper steps I think but selecting the Windows entry just reloads Systemd-boot.

I was trying to setup Timeshift for system snapshots on a work computer with Ubuntu. It didn’t work for some reason so I tried to first get rid of it. After uninstalling it, I wanted to remove, what I though, were remains of TS files in /run/timeshift, but the root partition was still mounted, so I rm-rfd the whole root, luckily except for home. And the computer has BIOS password with secure boot, so talking to IT dep about what I’ve done that is…. Or is it? The /boot and the initramfs was still in place, so it was dropping me to emergency shell when trying to boot. Connecting external USB to see if I can mount it, hmm doesn’t show up. Quick search on my private computer what kernel modules are required for USB storage, modprobed couple of xhci_* and bang, was able to mount it. I saved result of ls -l /dev/disk/by-uuid on the drive and moved to my private PC, where I created VM and installed exact same Ubuntu with exact config (LVM+Luks) and after it was done I copied all of / content to the (now formatted as ext4) external drive using cp -a, then edited fstab and crypttab to put proper UUIDs there, set up hostname and user account accordingly. Then moved back to the borked laptop, copied the newly installed Ubuntu back to the root partition, rebooted and it worked perfectly on first try and continues to work. All of that roller coster in just a single hour.

My mint install won't let sound through my sound card. Drivers are there, it knows exactly the brand and model of card and shows it, it even knows when I plug/unplug stuff from it, but 0 sound, ever.

The solution?

Just plug my headphones into my new speakers that have their own DAC, anyway.

Still no idea why the card doesn't work right

Fixing Grub issues when I was first starting out.

my session manager refused to start, and I was very close to reinstalling my system.

Not fixed but there is an Arch problem that is and will always be the bane of mi existence.

For some reason when I click with the trackpad buttons the touchpad gets frozen for like a second (it's like they are recognised by the system as keyboard buttons, I have enabled that option to temporarily disable it when using keyboard).

I've checked for hours and days the libinput documentation and some synaptics libraries, even legacy ones. It is to this day the only problem that has lead me to reinstall my system but the problem remains.

It's not even like I have some niche setup, I mean, surely there must be thousands of Arch users running with a ThinkPad X1 Carbon Gen 7, and surely not every single one of them must be running it like this, right?

It has come to a point where I just gave up and got used to my system as is, but I'm sure I would be running fanfare if some day I am able to fix it.

I used to main Gentoo.

Breaking the install was more of a guarantee.

I once removed most of X by trying to remove Gnome dependencies and it lead to an interesting couple of hours but I did have a working system when I was done.

There were countless dependency bugs and broken systems but at least I learned how to use the Gentoo Forum and also a lot of how Linux works.

I kind of want to give it another go.

It's been a long time but generally network issues and reinstalling bootloaders or kernels. Fairly easy if you can chroot.

Accidentally put grub on the wrong partition on the device, which it was not happy with. Was able to copy some files over, manually boot the OS, and reconfigure grub to be in the right partition, took me about 2 hours? Then I did it again on a different machine, and speedran it lol

I managed a CentOS system where someone accidentally deleted everything from /usr, so no lib64, and no bin. I didn't have a way to get proper files at the time, so I hooked the drive up to my Arch system, made sure glibc matched, and copied yum and other tools from Arch.

Booted the system, reinstalled a whole lot of yum packages, and... the thing still worked.

That's almost equivalent to a reinstall, though. As a broke college student, I had a laptop with a loose drive, that would fall out very easily. I set it up to load a few crucial things into a ramdisk at boot, so that I could browse the web and take notes even if the drive was disconnected, and it would still load images and things. I could pull the cover off and push the drive back in place to save files, but doing that every time I had class got really tiring, so I wanted it to run a little like a live system.

Used to be messing with kernel arguments and installing/tweaking boot parameters. That was until Grub broke, I learned systemd-boot and chrooting into the system via live USB

Now if I break anything it's just a matter of "sigh, let me get the USB and type a few commands"

Hmm I have come up with a bunch of neat solutions over the years. Where to start?

One time I broke the sudoers file on a distro without a root account, thoroughly locking myself out. I used docker -v /:/chroot to get myself root access to my root filesystem where I fixed the sudoers file. Protip always use visudo

I feel seen here, I was building a Ubuntu server and messed up the firewall settings not being able to get an internet connection, hours of trying to get back to where I was I gave up and plan to just start from scratch next time.

Is there a way of taking system snapshots with Linux?

Rescuing home partition from ZFS, actually that doesn't really count since I did have to reinstall (was no longer booting), but recovering the Home partition from ZFS and to the other ext4 drive was much harder than it should've been and that's why I would never recommend people use ZFS.

Jumping from the default kernel with zfs to the xanmod kernel using a manually compiled version of zfs. I don't rememeber a whole lot but it was quite... interesting. Next would be a suddenly vanished efi partition and my f* mainboard refusing to boot ZBM.

Bonus: my currently still unfixed problem is a very weird freezing/stuttering of the whole OS and the only (useless) "lead" I have is workqueue: fill_page_cache_func hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND