My Homelab Setup From Hardware to KVM - Part 1

TL;DR

I plan on writing a few posts about my Homelab project. This is the first one focusing on the hardware spec and networking.

Last year I decided to set up a homelab to learn technologies, to run some services for my own use and more importantly, to have fun.

There is a budget and a few high level requirements on the infrastructure side I worked on:

• Be able to run a hypervisor that supports both Linux and Windows virtual machine

• One single host. No clustering

• Use open source as long as the it’s mature enough

• Easy to manage and operate

• Cost effective

• No dedicated networking hardware

I plan on writing a few posts with respect to the HW spec I chose, the architecture decisions I made along the way and how I built the whole lab. This is the first post.

Hardware

Accessories such as case and power supply are not listed but included in the total cost.

Total Cost

The hardware cost of my homelab is around CA $1,400 when I bought them back in December 2019.

Motherboard

If you need to use ECC memory, you have to pick a motherboard that support ECC. In consumer grade motherboard market, look no further than ASUS and ASROCK. Those two are know for building consumer grade ECC-enabled motherboards.

If you need to manage your homelab hypervisor from a remote location, and be able to access it when troubleshooting something low level, e.g. when hypervisor is crashed or configuring BIOS, you have two choices: Phyical Access or IPMI.

Phyical Access

It means hook up a display, a keyboard and a mouse to your server and sit in front of it when troubleshooting.

IPMI

With IPMI, tasks such as power on/off the machine, configure BIOS settings and monitor console output all can be done from remote wia a web portal. Motherboards that support IPMI typically provide a dedicated Ethernet port for IPMI management. Learn more about IPMI here.

ASROCK has some pretty good IPMI-enabled workstartion motherboards. For AMD cpu, take a look at ASROCK X470D4U or x470d4u2-2t with 10G networking. There are also x570d4i-2t to be released in 2020.

Storage

To have flexability and speed, I need two kinds of storage. Direct Attached Storage (DAS) for hypervisor and guest OS images and Network Attached Storage (NAS) for data such as video and photos.

SSD as DAS

M2.SSD is my primary choice for DAS since it’s much faster than SATA SSD both on paper and in reality. So I bought a XPG SX8800 PRO 1TB and installed it on my server.

Unfortunately my server started freezing up randomly every a few hours to a few days. Troubleshooting proved to be one hell of an effort when my server is in the basement without a dedicated display. I’d have to take an external display from my home office down to the basement for troubleshooting. IPMI could have saved my day.

It took some time to reproduce the problem and troubleshoot the issues. Finally I was able to conclude that the controller chip that XPG SX8800 PRO uses may have compatibility issue with PRIME X470-PRO. After I replaced the M2.SSD with an old SATA SSD that I have, the server runs smoothly without any issues.

Synology NAS

I have a Synology DS416play. It’s a little home NAS with 4 bays and 1GB memory. I’ve since upgraded the memory to 8G and have it fitted with 4x10G HDD. In terms of usable space it has around 27T with one disk fault tolerance which is more than enough for me.

The downside of this NAS is it only has Gigabit Ethernet. If you think 10G is appealing, I’m totally with you. QNAP is offering a number of 10G ready NAS. Check it out here for detail.

Networking

My home network topology is rather simple. Both homlab server and NAS are connected to the internet router. The router provides basic functionalities such as IP management and port mapping.

Given my home network cables are all Cat5 supporting up to 1 Gigabit, I have to stay with 1G network and the on-board Gigabit Ethernet works well in my setup.

If you are considering 10G, you would need to make sure all critial devices on the network (e.g. router, NAS, Wifi etc.) are capable of dealing with the throughput, since any performance bottleneck can be the low watermark that undermines the overall network performance.

GPU Passthrough

The KVM hypervisor supports passing graphics card(s) onto dedicated guest OS.

On my setup, GTX1660 is the primary card for tasks such as streaming encoding/decoding whereas the GT710 is pass-through to a Win10 guest. This setup allows me to play a few not-so-demanding games on the Win10 through remote desktop.

I will discuss more on GPU passthrough in the next post.