Hey all. I'm exploring the idea of building a desktop PC optimized for running LLMs locally. My two primary use cases are I'd like to be able to add local documents and then talk to my files; I'd also like to use it as a coding assistant. Lower priority use case but something I'm tangentially interested in is image generation using stable diffusion. I don't plan to do any model training, I'll leave that to the pros.
One of the decisions I'm currently working through is whether to create this as a desktop workstation (like a PC build) or as more of a homelab environment (like a "local cloud"). On one hand, I believe a desktop workstation would be easier for me to wrap my head around b/c I've built several gaming PCs, whereas I have no homelab or self hosting experience beyond running a local-only Jellyfin instance on an old laptop. On the other hand, I like the thought a separate, atomic AI hub as like a local cloud if you will, similar to how I think of the NAS as a separate thing. What I like about the separate local cloud thinking is In both cases, the AI hub and/or the NAS can be accessed from any device.
I would like to strike the right balance between budget, power efficiency, and speed. I don't need to set any land speed records, but I would also like to avoid waiting several minutes for responses. I can probably spend up to $2,000 on this project, and I'm located in the US.
My questions for those the community who've gone before me:
- Has anyone build built a desktop workstation and then wished they built it as a server?
- Is there actually much of a difference between a desktop workstation versus a homelab environment when it comes to hardware for AI tasks?
- What other questions I should be asking myself to decide which way to go?
Thanks!
I don't have experience in this in terms of setting up a server vs desktop, but I have played with AI on a desktop and in a VM.
I'd suggest trying the tech in both a desktop and a VM to see which works for you. You could set up a low powered version of your server idea, see if it works in principle and see if it gives you what you want?
I do have a non AI Raspberry pi set up and there is a lot.of benefit to an always on always accessible device so I can see the attraction. But for AI are you going to use it enough to justify such a set up? Testing it with a VM might help answer that at least in part.
I've actually gotten a second graphics card recently to give a VM it's own hardware - I'm just tinkering and won't depend on it but a kernel VM on a desktop is a viable way to do this stuff too.
With a desktop you do have flexibility if you want to use it as an actual desktop, but equally with an always on server stack you do have a lot of options beyond AI to use it. For example a media server, a next cloud set up, syncthing mirror etc.
If you have $2000 to burn I guess either will work. If you don't I'd probably pragmatically go down the desktop route myself so I can make other users of the kit or have flexibility with what I can do with it (including selling it)
Gotcha, so for AI and gaming (dual purpose), I may as well incorporate both into one rig is what you're suggesting, right?