Selfhosted

I don't need to build a datacenter, i'm fine with building a rack myself in my garage.

During the last GPU mining craze, I helped build a 3-rack mining operation. Gpus are unregulated pieces of power-sucking shit from a power management perspective. You do not have the power requirements to do this on residential power, even at 300amp service.

Think of a microwave's behaviour ; yes, a 1000w microwave pulls between 700 and 900w while cooking, but the startup load is massive, almost 1800w sometimes, depending on how cheap the thing is.

GPUs also behave like this, but not at startup. They spin up load predictively, which means the hardware demands more power to get the job done, it doesn't scale down the job to save power. Multiply by 58 rx9070. Now add cooling.

You cannot do this.

the H200 has a very impressive bandwith of 4.89 TB/s, but for the same price you can get 37 TB/s spread across 58 RX 9070s, but if this actually works in practice i don't know.

Your math checks out, but only for some workloads. Other workloads scale out like shit, and then you want all your bandwidth concentrated. At some point you’ll also want to consider power draw:

• One H200 is like 1500W when including support infrastructure like networking, motherboard, CPUs, storage, etc.
• 58 consumer cards will be like 8 servers loaded with GPUs, at like 5kW each, so say 40kW in total.

Now include power and cooling over a few years and do the same calculations.

As for apples and oranges, this is why you can’t look at the marketing numbers, you need to benchmark your workload yourself.

It depends!

Exllamav2 was pretty fast on AMD, exllamav3 is getting support soon. Vllm is also fast AMD. But its not easy to setup; you basically have to be a Python dev on linux and wrestle with pip. Or get lucky with docker.

Base llama.cpp is fine, as are forks like kobold.cpp rocm. This is more doable without so much hastle.

The AMD framework desktop is a pretty good machine for large MoE models. The 7900 XTX is the next best hardware, but unfortunately AMD is not really interested in competing with Nvidia in terms of high VRAM offerings :'/. They don't want money I guess.

And there are... quirks, depending on the model.

I dunno about Intel Arc these days, but AFAIK you are stuck with their docker container or llama.cpp. And again, they don't offer a lot of VRAM for the $ either.

NPUs are mostly a nothingburger so far, only good for tiny models.

Llama.cpp Vulkan (for use on anything) is improving but still behind in terms of support.

A lot of people do offload MoE models to Threadripper or EPYC CPUs, via ik_llama.cpp, transformers or some Chinese frameworks. That's the homelab way to run big models like Qwen 235B or deepseek these days. An Nvidia GPU is still standard, but you can use a 3090 or 4090 and put more of the money in the CPU platform.

You wont find a good comparison because it literally changes by the minute. AMD updates ROCM? Better! Oh, but something broke in llama.cpp! Now its fixed an optimized 4 days later! Oh, architecture change, not it doesn't work again. And look, exl3 support!

You can literally bench it in a day and have the results be obsolete the next, pretty often.

Qwen3-235B-A22B-FP8

Good! An MoE.

Ideally its maxium context lenght of 131K but i’m willing to compromise.

I can tell you from experience all Qwen models are terrible past 32K. What's more, going over 32K, you have to run them in a special "mode" (YaRN) that degrades performance under 32K. This is particularly bad in vllm, as it does not support dynamic YaRN scaling.

Also, you lose a lot of quality with FP8/AWQ quantization unless it's native FP8 (like deepseek). Exllama and ik_llama.cpp quants are much higher quality, and their low batch performance is still quite good. Also, VLLM has no good K/V cache quantization (its FP8 destroys quality), while llama.cpp's is good, and exllama's is excellent, making it less than ideal for >16K. Its niche is more highly parallel, low context size serving.

My current setup is already: Xeon w7-3465X 128gb DDR5 2x 4090

Honestly, you should be set now. I can get 16+ t/s with high context Hunyuan 70B (which is 13B active) on a 7800 CPU/3090 GPU system with ik_llama.cpp. That rig (8 channel DDR5, and plenty of it, vs my 2 channels) should at least double that with 235B, with the right quantization, and you could speed it up by throwing in 2 more 4090s. The project is explicitly optimized for your exact rig, basically :)

It is poorly documented through. The general strategy is to keep the "core" of the LLM on the GPUs while offloading the less compute intense experts to RAM, and it takes some tinkering. There's even a project to try and calculate it automatically:

https://github.com/k-koehler/gguf-tensor-overrider

IK_llama.cpp can also use special GGUFs regular llama.cpp can't take, for faster inference in less space. I'm not sure if one for 235B is floating around huggingface, I will check.

Side note: I hope you can see why I asked. The web of engine strengths/quirks is extremely complicated, heh, and the answer could be totally different for different models.

Ah, here we go:

https://huggingface.co/ubergarm/Qwen3-235B-A22B-GGUF

Ubergarm is great. See this part in particular: https://huggingface.co/ubergarm/Qwen3-235B-A22B-GGUF#quick-start

You will need to modify the syntax for 2x GPUs. I'd recommend starting f16/f16 K/V cache with 32K (to see if that's acceptable, as then theres no dequantization compute overhead), and try not go lower than q8_0/q5_1 (as the V is more amenable to quantization).

LLMs are experimental, alpha-level technologies. Nvidia showed investors how fast their cards could compute this information. Now investors can just tell the LLM what they want, and it will spit out something that probably looks similar to what they want. But Nvidia is going to sell as many cards as possible before the bubble bursts.

ML has been sold as AI and honestly that's enough of a scam for me to call it one.

but I also don't really see end users getting scammed just venture capital and I'm ok with this.

Tflops is a generic measurement, not actual utilization, and not specific to a given type of workload. Not all workloads saturate gpu utilization equally and ai models will depend on cuda/tensor. the gen/count of your cores will be better optimized for AI workloads and better able to utilize those tflops for your task. and yes, amd uses rocm which i didn't feel i needed to specify since its a given (and years behind cuda capabilities). The point is that these things are not equal and there are major differences here alone.

I mentioned memory type since the cards you listed use different versions ( hbm vs gddr) so you can't just compare the capacity alone and expect equal performance.

And again for your specific use case of this large MoE model you'd need to solve the gpu-to-gpu communication issue (ensuring both connections + sufficient speed without getting bottlenecked)

I think you're going to need to do actual analysis of the specific set up youre proposing. Good luck

Efficiency still matters very much when self hosting. You need to consider power usage (do you have enough amps in your service to power a single GPU? probably. what about 10? probably not) and heat (it's going to make you need to run more A/C in the summer, do you have enough in your service to power an A/C and your massive amount of GPUs? not likely).

Homes are not designed for huge amounts of hardware. I think a lot of self hosters (including my past self) can forget that in their excitement of their hobby. Personally, I'm just fine not running huge models at home. I can get by with models that can run on a single GPU, and even if I had more GPUs in my server, I don't think the results (which would still contain many hallucinations) would be worth the power cost, strain on my A/C, and possible electrical overload.

😑