Selfhosted

We have NixOS, Proxmox and TrueNAS in use.

• TrueNAS on a dedicated NAS host. It's great for that, and has been super stable. The snapshotting works great, and all the little tasks associated with a NAS are taken care of without needing to spare a thought.
• Proxmox as VMS host. You haven't mentioned it above, so I'll leave it at this: also works really well for its purpose.
• NixOS: acouple dozen NixOS VMs runnign on the Proxmox hosts. I like the separation (i.e.: one VM <-> one task/service), but it's not necessary, esp. if you plan on using a stable branch. I absolutely love NixOS, and would never run server applications on anything else ever again. The documentation thing is trueish. There's not even close to the same documentation as with e.g. Arch and the Arch Wiki, but that makes sense when you think about it: instead of hundreds of lines of documentation, you hide that complexity behind an option, e.g. graphics.nvidia.enable = true; which then becomes pretty self-explanatory, at least if you are somewhat familiar with the ecosystem already. The way I'd recommend going about documentation with nix is this:
  • go to search.nixos.org/options, search for the service you would like to host. 90% of the time, the options and descriptions shown are all you need.
  • if an option is unclear, click on its "declared in" link. You'll be taken to the module source in nixpkgs. Look at what they are doing there/the comments explaining why. Often, this resolves any ambiguity, or helps you out with your goal.
  • if that did not help, check the NixOS wiki; often, common pitfalls are documented there, together with the nix expression to fix them.
  • another great way is to search GitHub for language:nix <thing you need to do>. As a random example: I recently wasn't sure how to configuring scaling in hyprland on NixOS, but searching for an appropriate term will quickly show you how other people have solved the same problem. It's not really documentation, but the declarative nature of nix means it's easy to find TONS of working examples via a github search.
  • all else failing, ask on discourse.nixos.org. Youńll usually get useful help very quickly there.

So, what's my advice?

If you are unfamiliar with NixOS, it's probably a bit of a headache getting a NAS to run satisfactory. Truenas works so well, there isn't really a need for nix. But running your services in nix is great, totally recommend!

I run Nix on my primary box and my travel laptop. I was intimately familiar with Linux before my first NixOS install.

You'll get the primary install up pretty quickly. If you have trouble with the nix.configuration, 9 out of 10 times chatGPT can can get you more or less what you're looking for in documentation. It's probably the single shining positive use of LLMs.

Adding things to the config once you know what you're trying to add and where you're trying to add it is not difficult at all. Sometimes documentation will tell you to use certain commands that don't seem to be available It's usually that you're missing a include which are somewhat cagey.

All that said, other than partial immutability, I don't know that you're going to reap any true benefit from nix other than the challenge. It's a nas, so you're not going to be moving it between different hardware. The data itself isn't immutable or protected or backed up in any way specific. I fear you'd just be fighting nix to support, manage and maintain RAID or ZFS.

Another pretty serious complication that Nix throws at you is major updates. Individual nix channel and home manager rebuilds aren't too bad, But when you need to run that 25.11 update You've got a pretty good chance at breaking your system. And it's not like they continue to run the older updates forever, they EOL those previous versions pretty quickly. That's actually one of the things keeping me from running it on more serverses they lack of a true LTS. A new major stable comes out every 6 months and you have one month after it comes out before the old version is EOL

For my laptop and desktop that's no big deal. But not so much for servers that I don't want to dick with all the time.

Could you expound on why you've narrowed it down to those two in the first place?

I haven't tried TrueNAS, but I use NixOS for my NAS and my other servers. It's perfect for servers imo since it's essentially a Linux system builder that you can customize to add whatever you want to your system.

The downside is that you'll have to learn a weird functional programming language, especially when you get into writing service modules yourself (for services that don't have an upstream module). And it'll take some time to get familiar with it and learn how to effectively write your configuration.

How well NixOS fits your purpose really depends on what you want to do with the OS. If you're just going run a bunch of docker containers, you could manage them via Nix but its a little cumbersome.

Where NixOS really shines for small servers are the so called NixOS Options. They allow you to install tons of services on bare metal but manage all the configuration for you. E.g. open the correct firewalls ports, run a dedicated DB or cache, etc. and all those simply require you to enable them with an ... = true;.

Smaller projects might not have a NixOS Option available and some options are more and/or easier configurable than others, but if you're running just a few common services you could feasibly manage your whole server with just one native config file and no docker shenanigans.

I'd recommend checking what's available under the link above. If you wanna go the container route instead, you have the option of just using docker non-declaratively as on every other distro (but then you lose some of the benefits NixOS gives you), or you can declaratively have NixOS manage all the docker containers. There are a few ways to do and manage this so some further research will be required.

I like having truenas in a proxmox vm. through proxmox you can easily do backups of your vms and in my head it containerizes things. I've had so many problems trying to use truenas as anything but a pure nas so having it as a vm in proxmox gives me the flexibility to run other vms and containers on my machine, and proxmox can easily be set up to connect to a nas and keep vm and container data on it.

Truenas has the primary focus of being a NAS. Running apps or VMs is more of a side thing. It can do these things fine, but because it's not the primary focus it doesn't get the same level of polish as the NAS aspects.

My personal solution is to run Proxmox as the host OS with a Truenas VM. Any other services I want to run get to live in VMs or LXC containers. For example all my docker services run across 3 VMs with a docker swarm setup.

Meh,

Debian with docker and Samba. Done.

Nixos if you want to repeat it... (Wouldn't do it as my first NixOS experience)

TrueNAS if you want clicky GUI handhold.

truenas as vm on proxmox

Personally, I keep storage and compute separate, makes it easier to maintain IMO. Where I have done it on a single box, ive just gone with a regular old Debian box and dropped in the drives.

I am considering playing with rockstor on a spare beige box I have, but basically the same separation of storage and compute will apply.

Keep storage separate or run the nas as a vm on the hypervisor.

I have been using TrueNAS Scale for a while but have not used base Linux for my NAS. My opinion is if you're looking for a quick initial setup or are like me and didn't want to install ZFS yourself, TrueNAS is rather appealing, but otherwise it doesn't offer much. It has ZFS pre-installed, gives you a webUI to monitor basic things about your machine, and has fairly easy ways to setup data protection with snapshots and backups with rsync or zfs replication. In the more recent versions it even has Docker apps built-in so you can host some basic things. The downside of TrueNAS is that despite being Linux under the hood, it's a lot more locked down so doing advanced measures is more of a pain and much of their "simpler" UI-based stuff is exceedingly basic, half-featured, and lacks documentation.

The way I use TrueNAS right now is to treat the main OS as mainly untouchable. I don't try to break out of the limits placed upon it. I instead use a "Jailmaker" machine (defunct wrapper script for systemd-nspawn) for all my Docker needs. This way the main system remains more stable. If I have to re-install, then it's a simple config import and my NAS is back to how it was.

I would use the built-in VM tools or the built-in Docker tools for this, but A. they weren't implemented or weren't working when I set this up, and B. I found their setup rather... annoying. For instance, I tried to set up some apps with their previous app system and it required configuration before working and yet nowhere did anyone explain how to configure it so I was wokring blind. No one makes guides for setting up an app in the TrueNAS UI, so the extra layer of obfuscation was just a hinderance to me. Compare that to setting it up directly in Docker, there are a million guides and great documentation for everything I get stuck on. Thus, despite being the "harder" way to set it up, it was easier due to the existence of information about it.

So, looking at it objectively, what parts of TrueNAS do I even use compared to base Linux? Not much. I use the WebUI to accomplish basic tasks such as creating or modifying datasets and permissions, snapshots, SMB shares, etcetera. All the basic things are there and I use the UI for them. But ever since that initial setup I spend most of my time in the CLI adjusting my scripts and Docker config files, creating directories inside the datasets, fine-tuning permissions... I could definitely have gone for a base Linux install as long as I knew what to install for ZFS support, some manner of WebUI, and so on. TrueNAS just did all that initial setup for me, and having a more locked-down OS forced me to use safer methods of installing programs via containers and keeping my install a lot more portable which I plan to continue no matter what OS I use.

This was probably not helpful, but that's been my experience of TrueNAS for what it's worth. Whatever you do, just remember: RAID is not a backup. It is protection against drive faults, but an error in the RAID system itself or the RAID pool's data requires a separate copy of the data stored elsewhere to restore.

TLDR; TrueNAS

I run TrueNAS and am happy with it so far. Run a few containers/apps on the same hardware without any problems but haven’t tried virtual machines. TrueNAS uses ZFS for its file system, which is good for preventing silent data corruption, which I didn’t really believe in until I was sorting out some photos recently and noticed that some of them that I had copied from my old NAS were slightly corrupt, i.e. line through the photo and Immich refusing to import the photo.

So if the apps you want to host are available as containers, then you will be ok. If you want to run virtual machines, then you can, it’s just not the primary use case, look into that a bit more.

I personally ended up running my containers on a VM on top of TrueNAS to get the best of both worlds (and because back then running applications directly on TrueNAS SCALE was convoluted)

You could read this article where the author runs NixOS in VMs on top of TrueNAS.

I'd avoid NixOS personally. It is to obscure of a Linux distros to be widely supported.

TrueNAS all the way. If you want something more you could virtualize it by picking up a extra Data/SAS card and passing it though to a TrueNAS VM.