[-] allyg79@alien.top 1 points 10 months ago

Thank you, much appreciated :-)

[-] allyg79@alien.top 1 points 10 months ago

The feedback's been good so I'll hopefully get this available to buy soon. I think somewhere in the $800ish price range without the CMs, which hopefully is near to home use price range.

[-] allyg79@alien.top 1 points 10 months ago

The Pi shortage was definitely a nightmare. I wouldn't have bothered with this even at the start of the year as you just couldn't find the kit. It does seem to have eased now, though. Digi-Key and Farnell have had good stock levels of CM4s for the last couple of months and the CM5 can't be far away now. I've found performance of CM4s with NVMe SSD's are pretty good, certainly enough for my use cases. Blades like this aren't much use for storage servers though, not really enough storage options. I wouldn't rule out doing a purpose built storage server at some point though, my home network has a couple of big NAS boxes with 20-odd SSDs and I'd love to replace those!

[-] allyg79@alien.top 1 points 10 months ago

I think I'll probably do something like that. I'll make it available as a full prebuilt unit but I'll open source the design files for anyone that really wants to DIY or build their own spins. I've deliberately used an off-the-shelf case and PSU, and only components easily available in distribution, so that it's easy to get the parts.

[-] allyg79@alien.top 1 points 10 months ago

Yeah I'll do a proper blog post on this in the next few days and then open up the design files on a public repo. I've got a new version of the blade being manufactured now so I'll upload the design once I've got it back and made sure it works. (The current version I'm using works perfectly except that I never noticed that I connected the USB the wrong way round, so I had to bodge-wire that out on my own units!)

[-] allyg79@alien.top 1 points 10 months ago

I'll probably do both! I've only done a rough costing so far but I think it'd be somewhere around $800ish USD for a 10 blade unit (without CMs of course.) I'll also likely open source at least the schematics and firmware for if anyone fancied making their own version of it. I'll do a blog post at some point soon about the design, and another once I've thought more about sales.

[-] allyg79@alien.top 1 points 10 months ago

There's a few pictures of these in some of my other replies, and I'll do a full blog post on this in the next few days.

I'd really like to do a version of this with an on-board Ethernet switch. It'd be really nice to do all the switching on-board and just have a single 10GbE uplink to the outside world. 1GbE/10GbE switch ICs with 11+ ports are pretty expensive so I'll probably see if I can sell a few of these ones before I try that!

Haven't really thought about other expansion beyond that but definitely interested in any ideas! Do you mean making it possible to connect PCI cards to the blades?

[-] allyg79@alien.top 1 points 10 months ago

Yeah I know about that one, I looked at it when I first started thinking about using Pi's to do the server stuff I wanted, but I couldn't actually buy one then. So I built my own :-) As I mentioned on another post, there's a few differences around my focus on using this as a simple server system.

[-] allyg79@alien.top 1 points 10 months ago

Thanks, that's very kind. I've added some more detail on other replies and I think I'll do a full blog post in the next couple of days.

There are definitely parallels with the Compute Blade project but there are a few differences. My blades are a bit simpler, they don't have the TPM that the Compute Blade does, as I didn't have any real need for it. The CB also has a more dense number of blades in the 19" width. This was another design decision on my part, I quite liked the short depth case making the unit small and I wanted to make sure there was plenty of airflow for cooling (tbh I didn't need as much as I used!)

My unit is more focused on being like a traditional server unit, as that's what my use case was. Centralised power, centralised management and provisioning etc. You're correct, the Compute Blade uses PoE, and I did it through the backplane. My preference was for central management rather than per-blade, so that meant a backplane and it all flowed from there. It allows you to feed the USB and serial console into the management server which is great for provisioning and debugging. The displays are also born out of my days as a network infrastructure guy, where being able to see the server's name and IP address on the physical unit would have been a godsend when doing maintenance! So I guess the design differences between this and the Compute Blade are about my focus on more of a server use rather than general compute module.

I'd say it's probably a bit cheaper using a backplane than PoE. The PoE adds a bit to the cost of each blade which would soon multiply up, plus the additional cost of a PoE switch vs non-PoE. I'm using an off-the-shelf ATX PSU and these are made in such huge quantities that the price per watt is difficult to beat.

[-] allyg79@alien.top 1 points 10 months ago

Thanks, some more info on other replies and I'll do a proper blog write up in the next few days.

[-] allyg79@alien.top 1 points 10 months ago

Yeah, this isn't useful for many things, but as others have mentioned there are situations where it is. My original use case, the thing which prompted me to build this (other than just the fun of seeing if I could do it!) was to replace a whole load of low complexity VMs. I'm a freelance programmer and I do a bunch of hosting for both myself and some clients out of my home office. I've got a small rack setup in my attic with UPS, and have redundant fibre connections. It's obvs nowhere near datacentre quality but it works well for my purposes.

I'd previously been using VMs running on some second hand enterprise x64 kit that I bought. Whilst this works great, the electricity bill is rather higher than I'd like! When I analysed what all the VMs are doing I realised that it'd be perfectly possible to do this on a Pi. In the dim and distant past I was a network infrastructure guy, so I started looking into "proper" server Pi solutions and before I knew it I was down this rabbit hole!

It works really well for low power server applications. It's not in the same league as the big iron ARM mega-core servers (or indeed Xeon servers) for performance, but then it's nowhere near that league for price either. I haven't figured out an exact price if I was to sell it commercially, but it'd likely be in the $800 US price range without CMs. If you were to max that with 4GB PIs that'd end up around $1500, which'd give you 40 cores of pretty decent performance and 80GB of RAM. The Gigabyte and Altera servers I've seen are awesome and way more powerful than this but are several times more expensive.

[-] allyg79@alien.top 1 points 10 months ago

Thanks, that's very kind. Here's links to some more pictures. The original ones were taken by my photographer wife and these ones were taken by me on my phone, so apologies for drop in quality!

This https://imgur.com/9eqdiGn is a view of my development test unit on the bench with the cover off. I'm using an off-the-shelf 1U PSU for power as it's a nice easy way of getting 100W+ all delivered at the right voltage levels. It's also the limiting factor in the number of blades that the box will take, as it takes up a decent chunk of space.

The PSU leaves just enough space at the front for the front panel board https://imgur.com/OSK9ngE. I'm using on off-the-shelf 2.4" LCD modules for the main screen and 0.91" OLED modules for the blade displays. The management CM4 is on its own little riser board as the CM is about 10mm too big to fit horizontally in the space. To keep costs down you'll see I'm using PCI-e x1 as the card edge connectors. These are WAY cheaper than the fancy purpose built back plane connectors so do the job perfectly.

The management board, the backplane and the individual blades all have RP2040's on them for management. https://imgur.com/YpDE1Uo is a close up of this on the management board. I could probably have done it with cheaper microcontrollers, but the RP2040 isn't overly expensive, is easy to get hold of, and it's nice keeping it all in the Pi ecosystem.

The backplane's got a couple of 74HC4067 multiplexers for switching the UARTs from the blade CMs down to the management module, and four FSUSB74's to do the same for the USB interface. There's also a few 9535 I/O expanders, both because I ran out of GPIO's on a single RP2040 but also to make routing easier on the 4 layer board.

I've mentioned on another reply some plans for the software, but mainly planning to add full status info (stats from each of the blades), along with a serial console and USB provisioning.

For my original use case, I'm actually using them all as individual servers. It replaced a bunch of VMs running on some second hand enterprise kit I had. The Pi's are able to do basically as good a job for what I need but consume much less power (the CM datasheet puts the max typical at about 7W, so even allowing for extra overhead you're running 10 blades at less than 100W.)

I'll need to do a proper blog post with all this at some point soon!

1
submitted 10 months ago by allyg79@alien.top to c/homelab@selfhosted.forum

Hi,

I thought I'd post my latest project. I use a bunch of Raspberry Pi compute modules as servers and decided to build myself a custom blade server to host them. This is replacing a bunch of old Intel rack mount servers on my home network - it's a lot less power hungry! It's been through a few iterations and is now working really well. This is the server:

https://preview.redd.it/4eff1iwi5i1c1.jpg?width=5442&format=pjpg&auto=webp&s=f91eebef92053a9698f74588df2a8ef3cd29462b

It's a 2U rack mountable unit, in an off-the-shelf ABS case with some custom 3D printed parts. The server takes up to 10 of these blades:

https://preview.redd.it/zi84q19k5i1c1.jpg?width=5472&format=pjpg&auto=webp&s=7b5e757c0f054ab96a97cf4be5b1ce9f4c49ff7f

It's got gigabit Ethernet, USB-A and HDMI on the front and an NVMe SSD slot on the board, along with an SD card slot and a battery backed real time clock. There's a little OLED on the front displaying information about the blade, including the name and IP address to make it easy to identify for maintenance. There's also an RP2040 on it for management.

The blades plug in to a custom backplane which provides power and centralised management. There's an LCD front panel providing basic tools for powering on and off blades and status information, and another compute module which acts as a management web server. It can be used to upload flash images to the blades via the backplane, and provides serial console access to the blades through the web interface.

I've been using this for a while now and was wondering if other folks out there are interested in it? It would be quite quick and easy for me to turn this into a product for sale if there was a market out there for it.

Please let me know any comments or suggestions you have, any feedback is appreciated!

Alastair

view more: next ›

allyg79

joined 10 months ago