this post was submitted on 30 Aug 2025
182 points (86.7% liked)

Selfhosted

51028 readers
2313 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud@lemmy.world
CannaVet@lemmy.world
Loki@lemmy.world
HybridSarcasm@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz
182
1U mini PC for AI? (startrek.website)
submitted 1 day ago by nagaram@startrek.website to c/selfhosted@lemmy.world
56 comments fedilink hide all child comments
 

My rack is finished for now (because I'm out of money).

Last time I posted I had some jank cables going through the rack and now we're using patch panels with color coordinated cables!

But as is tradition, I'm thinking about upgrades and I'm looking at that 1U filler panel. A mini PC with a 5060ti 16gb or maybe a 5070 12gb would be pretty sick to move my AI slop generating into my tiny rack.

I'm also thinking about the PI cluster at the top. Currently that's running a Kubernetes cluster that I'm trying to learn on. They're all PI4 4GB, so I was going to start replacing them with PI5 8/16GB. Would those be better price/performance for mostly coding tasks? Or maybe a discord bot for shitposting.

Thoughts? MiniPC recs? Wanna bully me for using AI? Please do!

top 50 comments
sorted by: hot top controversial new old
[–] Flax_vert@feddit.uk 1 points 1 hour ago* (last edited 1 hour ago)

A single raspberry pi 5 can host an entire website

[–] Flax_vert@feddit.uk 1 points 1 hour ago

How much did this cost?

[–] Korhaka@sopuli.xyz 5 points 23 hours ago

Ohh nice, I want it. Don't really know what I would use all of it for, but I want it (but don't want to pay for it).

Currently been thinking of getting an N150 mini PC. Setup proxmox and a few VMs. At the very least pihole, location to dump some backups and also got a web server for a few projects.

[–] GirthBrooks@lemmy.world 19 points 1 day ago (1 children)

Looking good! Funny I happen across this post when I’m working on mine as well. As I type this I’m playing with a little 1.5” transparent OLED that will poke out of the rack beside each pi, scrolling various info (cpu load/temp, IP, LAN traffic, node role, etc)

[–] ripcord@lemmy.world 3 points 1 day ago (1 children)

What OLED specifically and what will you be using to drive it?

[–] GirthBrooks@lemmy.world 2 points 20 hours ago

Waveshare 1.51” transparent OLED. Comes with driver board, ribbon & jumpers. If you type it in Amazon it’s the only one that pops, just make sure it says transparent. Plugs into GPIO of my Pi 5s. The Amazon listing has a user guide you can download so make sure to do that. I was having trouble figuring it out until I saw that thing. Runs off a python script but once I get it behaving like I want I’ll add it to systemd so it boots on startup.

Imma dummy so I used ChatGPT for most of it, full ..ahem.. transparency. 🤷🏻‍♂️

I’m modeling a little bracket in spaceclaim today & will probably print it in transparent PETG. I’ll post a pic when I’m done!

[–] hendrik@palaver.p3x.de 6 points 1 day ago (2 children)

Well, I always advocate for using the stuff you have. I don't think a Discord bot needs four new RasPi 5. That's likely to run on a single RasPi3. And as long as they're sitting idle, it doesn't really matter which model number they have... So go ahead and put something on your hardware, and buy new one once you've maxed out your current setup.

I'm not educated on Bazzite. Maybe tools like Distrobox or other container solutions can help running AI workloads on the gaming rig. It's likely easier to run a dedicated AI server, but I started learning about quantization, tested some models on my main computer with the help of ollama, KoboldCPP and some random Docker/Podman containers. I'm not saying this is the preferrable solution. But definitely enough to get started with AI. And you can always connect the computers within your local network, write some server applications and have them hook into ollama's API and it doesn't really matter whether that runs on your gaming pc or a server (as long as the computer in question is turned on...)

[–] Flax_vert@feddit.uk 2 points 1 hour ago

You could probably run several discord bots on a Raspberry Pi 3, provided they aren't public and popular

[–] nagaram@startrek.website 4 points 1 day ago (1 children)

Ollama and all that runs on it its just the firewall rules and opening it up to my network that's the issue.

I cannot get ufw, iptables, or anything like that running on it. So I usually just ssh into the PC and do a CLI only interaction. Which is mostly fine.

I want to use OpenWebUI so I can feed it notes and books as context, but I need the API which isn't open on my network.

[–] hendrik@palaver.p3x.de 1 points 1 hour ago* (last edited 40 minutes ago)

Thanks for the info. Some day I'll try the shiny modern distros and learn the little peculiarities. I use a weird mix of Debian, NixOS and LMDE and it's relatively straightforward to add firewall rules to those, both dynamically to nftables and to the persistent config... And I believe Debian didn't even come with firewalling out of the box... But I understand Debian might not be the best choice for gaming and there is for example some extra work involved to get the latest Nvidia drivers. Neither is it an atomic distro.

[–] umbrella@lemmy.ml 5 points 1 day ago* (last edited 1 day ago)

for your usecase, i'd get an external gpu to plug in to one of these juicy thinkstations right there. bonus for the modularity of having an actual gpu instead of relying on whatever crappy laptop gpu minipc manufacturers put in there.

you could probably virtualize a sick gaming setup with this rig too. stream it to your phone/laptop.

nice setup btw.

[–] TropicalDingdong@lemmy.world 14 points 1 day ago (1 children)

This is so pretty 😍🤩💦!!

I've been considering a micro rack to support the journey, but primarily for house old laptop chassis as I convert them into proxmox resources.

Any thoughts or comments on you choice of this rack?

[–] nagaram@startrek.website 4 points 1 day ago

Not really a lot of thought went into rack choice. I wanted something smaller and more powerful than my several optiplexs I had.

I also decided I didn't want storage to happen here anymore because I am stupid and only knew how to pass through disks for Truenas. So I had 4 truenas servers on my network and I hated it.

This was just what I wanted at a price I was good with at Like $120. There's a 3D printable version but I wasn't interested in that. I do want to 3D print racks and I want to make my own custom ones for the Pis to save space.

But this set up is way cheaper if you have a printer and some patience.

[–] thejml@sh.itjust.works 5 points 1 day ago* (last edited 1 day ago)

Honestly, If you are delving into Kubernetes, just add some more of those 1L PCs in there. I tend to find them on ebay cheaper than Pi's. Last year I snagged 4x 1L Dells with 16GB RAM for $250 shipped. I swapped some RAM around, added some new SSD's and now have 3x Kube masters, 3x Kube worker nodes and a few VMs running a Proxmox cluster across 3 of the 1L's with 32GB and a 512GbB SSD each and its been great. The other one became my wife's new desktop.

Big plus, there are so many more x86_64 containers out there compared to Pi compatible ARM ones.

[–] Diplomjodler3@lemmy.world 8 points 1 day ago (1 children)

I'm afraid I'm going to have to deduct one style point for the misalignment of the labels on the mini PCs.

[–] nagaram@startrek.website 9 points 1 day ago (1 children)

That's fair and justified. I have the label maker right now in my hands. I can fix this at any moment and yet I choose not to.

I'm man feeding orphans to the orphan crushing machine. I can stop this at any moment.

[–] Diplomjodler3@lemmy.world 6 points 1 day ago

The machine must keep running!

[–] InternetCitizen2@lemmy.world 6 points 1 day ago

NSFW

[–] ZeDoTelhado@lemmy.world 7 points 1 day ago* (last edited 1 day ago) (4 children)

I have a question about ai usage on this: how do you do this? Every time I see ai usage some sort of 4090 or 5090 is mentioned, so I am curious what kind of ai usage you can do here

[–] teslasdisciple@lemmy.ca 13 points 1 day ago (2 children)

I'm running ai on an old 1080 ti. You can run ai on almost anything, but the less memory you have the smaller (ie. dumber) your models will have to be.

As for the "how", I use Ollama and Open WebUI. It's pretty easy to set up.

[–] kata1yst@sh.itjust.works 5 points 1 day ago* (last edited 17 hours ago)

Similar setup here with a 7900xtx, works great and the 20-30b models are honestly pretty good these days. Magistral, Qwen 3 Coder, GPT-OSS are most of what I use

load more comments (1 replies)
[–] chaospatterns@lemmy.world 6 points 1 day ago

Your options are to run smaller models or wait. llama3.2:3b fits on my 1080 Ti VRAM and is sufficiently fast. Bigger models will get split between VRAM and RAM and run slower but it'll work.

Not all models are Gen AI style LLMs. I run GPU based speech to text models on my GPU too for my smart home.

[–] nagaram@startrek.website 4 points 1 day ago (2 children)

With a RTX 3060 12gb, I have been perfectly happy with the quality and speed of the responses. It's much slower than my 5060ti which I think is the sweet spot for text based LLM tasks. A larger context window provided by more vram or a web based AI is cool and useful, but I haven't found the need to do that yet in my use case.

As you may have guessed, I can't fit a 3060 in this rack. That's in a different server that houses my NAS. I have done AI on my 2018 Epyc server CPU and its just not usable. Even with 109gb of ram, not usable. Even clustered, I wouldn't try running anything on these machines. They are for docker containers and minecraft servers. Jeff Geerling probably has a video on trying to run an AI on a bunch of Raspberry Pis. I just saw his video using Ryzen AI Strix boards and that was ass compared to my 3060.

But to my use case, I am just asking AI to generate simple scripts based on manuals I feed it or some sort of writing task. I either get it to take my notes on a topic and make an outline that makes sense and I fill it in or I feed it finished writings and ask for grammatical or tone fixes. Thats fucking it and it boggles my mind that anyone is doing anything more intensive then that. I am not training anything and 12gb VRAM is plenty if I wanna feed like 10-100 pages of context. Would it be better with a 4090? Probably, but for my uses I haven't noticed a difference in quality between my local LLM and the web based stuff.

load more comments (2 replies)
load more comments (1 replies)
[–] TexasDrunk@lemmy.world 4 points 1 day ago

I didn't even know these sorts of mini racks existed. now I'm going to have to get one for all my half sized preamps if they'll fit. That would solve like half the problems with my studio room and may help bring back some of my spark for making music.

I have no recs. Just want to say I'm so excited to see this. I can probably build an audio patch panel.

[–] 6nk06@sh.itjust.works 7 points 1 day ago (2 children)

https://en.wikipedia.org/wiki/ThinkCentre because I didn't knew it existed.

[–] nagaram@startrek.website 11 points 1 day ago

These are M715q Thinkcentres with a Ryzen Pro 5 2400GE

[–] nagaram@startrek.website 6 points 1 day ago

Oh and my home office set up uses Tiny in One monitors so I configured these by plugging them into my monitor which was sick.

I'm a huge fan of this all in one idea that is upgradable.

[–] Colloidal@programming.dev 4 points 1 day ago (1 children)

You could combine both 1U fillers and install a 2U PC, which would be easier to find.

load more comments (1 replies)
[–] brucethemoose@lemmy.world 3 points 1 day ago* (last edited 1 day ago) (1 children)

If you can swing $2K, get one of the new mini PCs with an AMD 395 and 64GB+ RAM (ideally 128GB).

They're tiny, lower power, and the absolute best way to run the new MoEs like Qwen3 or GLM Air for coding. TBH they would blow a 5060 TI out of the water, as having a ~100GB VRAM pool is a total game changer.

I would kill for one on an ITX mobo with an x8 slot.

[–] princessnorah@lemmy.blahaj.zone 4 points 1 day ago (2 children)

I think the mainboard from the Framework Desktop meets your requirements: https://frame.work/au/en/products/framework-desktop-mainboard-amd-ryzen-ai-max-300-series?v=FRAFMK0002

[–] MalReynolds@piefed.social 3 points 1 day ago (1 children)

Pretty sure that's a x4 PCIe slot (admittedly PCIe 5x4, but not many video cards speak PCIe5), would totally trade a usb4 for a x8, but these laptop chips are pretty constrained lanes wise.

[–] brucethemoose@lemmy.world 4 points 1 day ago* (last edited 1 day ago) (1 children)

It's PCIe 4.0 :(

but these laptop chips are pretty constrained lanes wise

Indeed. I read Strix Halo only has 16 4.0 PCIe lanes in addition to its USB4, which is resonable given this isn't supposed to be paired with discrete graphics. But I'd happily trade an NVMe slot (still leaving one) for x8.

One of the links to a CCD could theoretically be wired to a GPU, right? Kinda like how EPYC can switch its IO between infinity fabric for 2P servers, and extra PCIe in 1P configurations. But I doubt we'll ever see such a product.

[–] MalReynolds@piefed.social 2 points 1 day ago (1 children)

It's PCIe 4.0 :(

Boo! Silly me thinking DDR5 implied PCIe5, what a shame.

Feels like they're testing the waters with Halo, hopefully a loud 'waters great, dive in' signal gets through and we get something a bit fitter for desktop use, maybe with more memory (and bandwidth) next gen. Still, gotta love the power usage, makes for one hell of a NAS / AI inference server (and inference isn't that fussy about PCIe bandwidth, hell eGPU works fine as long as the model / expert fits in VRAM.

[–] brucethemoose@lemmy.world 2 points 1 day ago* (last edited 1 day ago) (1 children)

Rumor is it’s successor is 384 bit, and after that their designs are even more modular:

https://www.techpowerup.com/340372/amds-next-gen-udna-four-die-sizes-one-potential-96-cu-flagship

Hybrid inference prompt processing actually is pretty sensitive to PCIe bandwidth, unfortunately, but again I don’t think many people intend on hanging an AMD GPU off these Strix Halo boards, lol.

[–] princessnorah@lemmy.blahaj.zone 1 points 7 hours ago (1 children)

I don't know that that is necessarily true. Having a gaming machine that can play any game and dynamically switches between a high-power draw dGPU and a genuinely capable low-power draw iGPU actually sounds amazing. That's always been possible with every laptop that has a dGPU but their associated iGPU has often been bottom of the barrel bc "why would you use it" for intensive tasks. But a "desktop" build as a lounge room gaming PC, where you can throw whatever at it and it'll run as quietly as it can, while being able to play AAAs at 4K60, sounds amazing.

[–] brucethemoose@lemmy.world 1 points 2 hours ago* (last edited 2 hours ago) (1 children)

Eh, actually that’s not what I had in mind:

  • Discrete desktop graphics idle hot. I think my 3090 uses at least 40W doing literally nothing.

  • It’s always better to run big dies slower than small dies at high clockspeeds. In other words, if you underclocked a big desktop GPU to 1/2 its peak clockspeed, it would use less than a fourth of the energy and run basically inaudible… and still be faster than the iGPU. So why keep a big iGPU around?

My use case was multitasking and compute stuff. EG game/use the discrete GPU while your IGP churns away running something. Or combine them in some workloads.

Even the 395 by itself doesn’t make a ton of sense for an HTPC because AMD slaps so much CPU on it. It’s way too expensive and makes it power thirsty. A single CCD (8 cores instead of 16) + the full integrated GPU would be perfect and lower power, but AMD inexplicably does not offer that.

Also, I’ll add that my 3090 is basically inaudible next to a TV… key is to cap its clocks, and the fans barely even spin up.

[–] princessnorah@lemmy.blahaj.zone 1 points 2 hours ago (1 children)

That's all valid for your usecase, but you were saying that you didn't think many people would use it that way at all and that's what I was saying I didn't agree with. As well, a HTPC is kind of a different use case altogether to a lounge room gaming computer. There's some overlap for sure, but if you want zero compromise gaming then you're going to want all that CPU.

[–] brucethemoose@lemmy.world 1 points 2 hours ago* (last edited 1 hour ago) (1 children)

Eh, but you’d be way better off with an X3D CPU in that scenario, which is both significantly faster in games, about as fast outside them (unless you’re dram bandwidth limited) and more power efficient (because they clock relatively low).

You’re right about the 395 being a fine HTPC machine by itself.

But I’m also saying even an older 7900, 4090 or whatever would be way lower power at the same performance as the 395's IGP, and whisper quiet in comparison. Even if cost is no object. And if that’s the case, why keep a big IGP at all? It just doesn’t make sense to pair them without some weirdly specific use case that can use both at once, or that a discrete GPU literally can’t do because it doesn’t have enough VRAM like the 395 does.

[–] princessnorah@lemmy.blahaj.zone 1 points 1 hour ago* (last edited 1 hour ago) (1 children)

Correct me if I'm wrong here, but is the 395 not leagues ahead of something like a 4090 when it comes to performance per watt? Here's a comparison graph of a 4090 against the Radeon 8060S, which is the 395's iGPU:


Source.

Now that's apparently running at the 395's default TDP of 55W so that includes the CPU power. It's also clear that a 4090 can trounce it on sheer performance when needed. But if we take a look at this next graph:


Source.

This shows that a 4090 has a third of the performance while still running at 130W, more than twice the TDP of the entire 395 APU.

Edit: This was buried in the comments under that second graph but here's the points scored per Watt on that benchmark: 130W = 66 / 180W = 85 / 220W = 92 / 270W = 84 / 330W = 74 / 420W = 59 / 460W = 55 and this clearly shows the sweet spot for a 4090 is 220W.

[–] brucethemoose@lemmy.world 2 points 1 hour ago (1 children)

Oh wow, that's awesome! I didn't know folks ran TDP tests like this, just that my old 3090 seems to have a minimum sweet spot around that same same ~200W based on my own testing, but I figured the 4000 or 5000 series might go lower. Apparently not, at least for the big die.

I also figured the 395 would draw more than 55W! That's also awesome! I suspect newer, smaller GPUs like the 9000 or 5000 series still make the value proposition questionable, but still you make an excellent point.

And for reference, I just checked, and my dGPU hovers around 30W idle with no display connected.

[–] princessnorah@lemmy.blahaj.zone 2 points 36 minutes ago (1 children)

You can boost the 395 up to 120W, which might be where Framework is pushing it too, but those benchmarks are labelled 55W and that's what AMD says is the default clock without adjustment. I'd love to see how the benchmarks compare at that higher boost but I'd imagine it's diminishing returns similar to most GPUs. I think the benefit to using it in a lounge gaming PC would be the super low power draw, but you would need to figure out a display MUX switch and I don't think that's simple with desktop cards. Maybe something with a 5090 mobile would be the go at that point, but I have no idea how that compares to the 395 and whether it's worth it.

[–] brucethemoose@lemmy.world 1 points 13 minutes ago* (last edited 9 minutes ago)

Mobile 5090 would be an underclocked, binned desktop 5080, AFAIK:

https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#GeForce_50_series

In KCD2 (a fantastic CryEngine game, a great benchmark IMO), at QHD, the APU is a hair less half as fast. For instance, 39 FPS at QHD vs 84 FPS for the mobile 5090:

https://www.notebookcheck.net/Nvidia-GeForce-RTX-5090-Laptop-Benchmarks-and-Specs.934947.0.html

https://www.notebookcheck.net/AMD-Radeon-8060S-Benchmarks-and-Specs.942049.0.html

Synthetic benchmarks between the two

But these are both presumably running at high TDP (150W for the 5090). Also, the mobile 5090 is catastrophically overpriced and inevitably tied to a weaker CPU, whereas the APU is a monster of a CPU. So make of that what you will.

load more comments (1 replies)
[–] possiblylinux127@lemmy.zip 4 points 1 day ago (1 children)

You also could pickup a powerful CPU with lots of memory bandwidth like a threadripper

[–] nagaram@startrek.website 4 points 1 day ago (1 children)

I think I'm going to have a harder time fitting a threadripper in my 10 inch rack than I am getting any GPU in there.

[–] Cocodapuf@lemmy.world 2 points 1 day ago* (last edited 22 hours ago)

I think I'm going to have a harder time fitting a threadripper in my 10 inch rack than I am getting any GPU in there.

Well, you could always use a closed loop CPU cooler. (Not necessarily that one)

With the radiator hanging out in back, this shouldn't need much height.

[–] tofu@lemmy.nocturnal.garden 4 points 1 day ago (1 children)

Since you seem to be looking for problems to solve with new hardware, do you have a NAS already? Could be tight in 1U but maybe you can figure something out.

load more comments (1 replies)
load more comments
view more: next ›