SuperMicro VMware Home Lab

I have had a lot of interest in my new home lab, which I have finally installed. So thought it would be a good opportunity to share what I have done for all to see.

I will say this straight up I paid for everything, no gear was given to me, so i am not endorsing any particular brands or kit in this post. It all is just what worked for me to meet my requirements.

Background

Previously I had 3 white boxes, i7 with 2011 socket motherboard and 64GB RAM, these have done me nearly 3.5 years and cost me around $3200 AUD. 2 VCDX’s and a lot of Presentations later they are up for retirement.

These were great, but RAM was a killer with everything I need to run in the lab. More RAM was my number 1 requirement, but I had a big constraint which was cost, I don’t earn the big bucks so I needed to keep my costs down as much as possible.

Now it would be awesome to be able to get new kit that’s on the HCL for 6.5 but this really was not going to be possible for the resources I needed, and that’s OK. Had white boxes before so not terribly worried about that, as my experience says ESXi/vCenter will run fine.

I was really interested in the Supermicro E200 and E300 Tai has done a series of blog posts on these at https://lab-rat.com.au/2017/04/01/supermicro-vs-intel-nuc/  I know a number of people who have bought these and have had a good experience with them. My main concern was cost If you take the numbers Tai had done up below:

Product Part Number Price
SuperMicro E200-8D SYS-E200-8D $799.99
64GB ECC UDIMM RAM (4 x 16GB) TBA $329.95
1TB 2.5” SSD SanDisk X400 $299.95
128GB NVMe M.2 SSD Plextor PX-128S2G $59.99
1x Additional Case Fan FAN-0065L4 $9.95
Rack Mount Brackets SMC-MCP-290-10110-0B $44.95
$1544.74
virtuallyGhetto 2% Discount VIRTUALLYGHETTO2OFF -$30.89
TOTAL   US $1513.85

If I wanted to get the same amount of RAM as I already had It was going to cost me USD $4539 which would be roughly AUD $5922 which was too rich for my blood for a system that would give me some benefits over what I currently had, but ultimately be able to run the same amount of VM’s that my current lab supports. If I went ahead and added 128GB of RAM to each That would be an extra USD $600 per node which would bring the home lab to AUD $8271. This is not including the 10Gbe copper switch I would also need to buy.

If I wanted to stay married I had to come up with a cheaper option

Hunt for the New Home Lab Begins

Now I stumbled across a group called ITinStock who are in the UK and they sell refurbished server kit there, bonus is a number of items had free shipping to Australia. I really liked the look of the 2RU chassis that contain 4 server nodes. They had a number of different brands but the supermicro was the best value at the time. They also allowed the modification of the items they where selling so I got a quote to add 96GB RAM per node and liked what I saw, so I pulled the trigger on a SuperMicro 6027TR-HTQF.

The 6027TR-HTQF comes in many different configurations 6core, 8 core etc. I will do a similar comparison as Tai on what my specs are.

SuperMicro E200-8D Intel NUC 7th Gen SuperMicro 6027TR-HTQF (per node)
ESXi 6.5 Compatible Native install works Requires NIC drivers Native install works
CPU Type XEON D-1528 Intel i7-7567U Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
CPU Capacity 6 core 12 threads

1.9 GHz – 2.2 GHz

Dual Core

3.5 GHz – 4.0 GHz

2 x 6 core (24 threads)

2.5 GHz

RAM Type 4x DDR4 2x DDR4 SODIMM 8 x DDR3
RAM Capacity 128GB ECC RDIMM

64 GB Non-ECC UDIMM

32GB Non-ECC 512GB ECC (LRDIMM)

256GB ECC (RDIMM)

Intel Optane Ready Yes Yes No
HDD Capacity 1x 2.5” 1x 2.5” 3 x 3.5/2.5
NVMe / M.2 SATA YES YES NO
SATADOM YES NO YES
Micro SDXC NO YES NO
1Gbe Networking 2x 1Gbe 1x 1Gbe 2 x 1Gbe
10Gbe Networking 2x 10Gbe 1x Thunderbolt 40Gbps 1 x infiniband (40Gbps)
Wireless NO 802.11ac NO
IPMI YES NO YES
Power Consumption 60W 64W 120W
Rack Mounting YES  NO YES
SR-IOV Support YES NO  YES
Video Port VGA only HDMI 4k  VGA only
Noise Comparison 2x Fans Fan-less  4 x 8cm Fans(Chassis)
USB Capacity 2 Ports 4 Ports  2 ports
Price Comparison US$799 US$630  US$801 (FULLY POPULATED)

Because this is refurbished gear the cost can vary, people in the US would most likely find a better deal than I could because international shipping was huge from US sellers but cheaper units. And if your in the UK it would also most likely be cheaper too. The cost in the table above is the cost of the total purchase divided by 4, which includes:

  • Chassis
  • 4 x nodes
  • 1200w platnium PSU
  • 384GB RAM
  • 8 x 6core CPU’s

Delivered to my door was Cost of the Unit $3800 AUD + customs $400 AUD = Total $4200 AUD which is about $3205 USD. This IMO was good bang for buck, it delivered on more resources than I currently had, left 2 free dimms per node so easy to expand to 128GB per Node totaling 512GB RAM for the lab if i needed to down the track.

I was going to re use the VSAN hard disks that I already had but running hybrid being able to fit 3.5″ disks allow this benefit. nothing special 256GB SSD and 2TB Sata HDD.

All that was left now was networking and a switch. I was tempted to give Ethernet over infiniband a go, switches are cheep it was on-board and 40Gbps was more than I needed. During some reading I got the impression the x3 no longer worked with 6.5 and thought I better stick with what I know (side note the 40Gbps nic shows up in vSphere and looks fine). At the moment Copper 10Gbe switches are too expensive and the Mellanox MT26448 10Gbe sfp cards are still on the HCL and are really cheap ended up getting a bundle of 5 cards with 5 x 3m dat cables.

The switch was the next thing I needed. I really wanted to have 10GB for VSAN but also vMotion etc. I ended up going with the TPLINK T1700G-28TQ It was $399 AUD and had 4 x 10GB sfp+ slots which can be used as well as the 24 x 1GBe ports.

BOM (to my door)

Part Cost
SuperMicro 6027TR-HTQF (fully populated with the above hardware)
$4200AUD / $3205USD
5 x Mellanox Technologies MT26448 10Gbe with 5 x 3m DAT cables $200AUD /  $152USD
TPLINK T1700G-28TQ $399AUD / $304USD
Total $4799 AUD / $3659 USD

In total I have 48 core (96 Threads), 384GB RAM, 8TB of VSAN, 10Gbe networking. This was great The equivalent capacity E200/300 would be almost twice as much. Now granted the E200/300’s are newer, compact and quieter but just don’t cut it for what I need to do. If I had an apartment and needed to make sure they where quiet I might have to have gone with another solution but for me this lab was almost perfect.

As with the E200 and NUC’s vsan is running on on-board ACHI drivers, I am not trying to break any records and if I used infiniband I could have put a controller in. either way it still performs fine.

Noise

Now don’t get me wrong I know some are going to say this solution would be noisy and to be honest it is. not as loud as a Cisco 3750 but its still a 2RU rack server. this is really the only CON there is. But for me being quiet was not a requirement. I have a big arse garage and this is still in a 42 RU server rack in there. Can not hear a thing. E200 and NUC’s being basically silent this is a downside of my solution

Power

Compared to the others it is double the power, under load the system appears to pull about 120 watts which is twice than the E200 and NUC’s but it is twice the compute as well.

Conclusion

I personally I don’t think I could have done too much better with the amount of resources I now have, It ticked all my boxes for what I was wanting out of a home lab. I hope I get 3 years out of this lab too.

Cheers

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Anti SPAM BOT Question * Time limit is exhausted. Please reload CAPTCHA.