I have had a lot of interest in my new home lab, which I have finally installed. So thought it would be a good opportunity to share what I have done for all to see.
I will say this straight up I paid for everything, no gear was given to me, so i am not endorsing any particular brands or kit in this post. It all is just what worked for me to meet my requirements.
Previously I had 3 white boxes, i7 with 2011 socket motherboard and 64GB RAM, these have done me nearly 3.5 years and cost me around $3200 AUD. 2 VCDX’s and a lot of Presentations later they are up for retirement.
These were great, but RAM was a killer with everything I need to run in the lab. More RAM was my number 1 requirement, but I had a big constraint which was cost, I don’t earn the big bucks so I needed to keep my costs down as much as possible.
Now it would be awesome to be able to get new kit that’s on the HCL for 6.5 but this really was not going to be possible for the resources I needed, and that’s OK. Had white boxes before so not terribly worried about that, as my experience says ESXi/vCenter will run fine.
I was really interested in the Supermicro E200 and E300 Tai has done a series of blog posts on these at https://lab-rat.com.au/2017/04/01/supermicro-vs-intel-nuc/ I know a number of people who have bought these and have had a good experience with them. My main concern was cost If you take the numbers Tai had done up below:
|64GB ECC UDIMM RAM (4 x 16GB)||TBA||$329.95|
|1TB 2.5” SSD||SanDisk X400||$299.95|
|128GB NVMe M.2 SSD||Plextor PX-128S2G||$59.99|
|1x Additional Case Fan||FAN-0065L4||$9.95|
|Rack Mount Brackets||SMC-MCP-290-10110-0B||$44.95|
|virtuallyGhetto 2% Discount||VIRTUALLYGHETTO2OFF||-$30.89|
If I wanted to get the same amount of RAM as I already had It was going to cost me USD $4539 which would be roughly AUD $5922 which was too rich for my blood for a system that would give me some benefits over what I currently had, but ultimately be able to run the same amount of VM’s that my current lab supports. If I went ahead and added 128GB of RAM to each That would be an extra USD $600 per node which would bring the home lab to AUD $8271. This is not including the 10Gbe copper switch I would also need to buy.
If I wanted to stay married I had to come up with a cheaper option
Hunt for the New Home Lab Begins
Now I stumbled across a group called ITinStock who are in the UK and they sell refurbished server kit there, bonus is a number of items had free shipping to Australia. I really liked the look of the 2RU chassis that contain 4 server nodes. They had a number of different brands but the supermicro was the best value at the time. They also allowed the modification of the items they where selling so I got a quote to add 96GB RAM per node and liked what I saw, so I pulled the trigger on a SuperMicro 6027TR-HTQF.
The 6027TR-HTQF comes in many different configurations 6core, 8 core etc. I will do a similar comparison as Tai on what my specs are.
|SuperMicro E200-8D||Intel NUC 7th Gen||SuperMicro 6027TR-HTQF (per node)|
|ESXi 6.5 Compatible||Native install works||Requires NIC drivers||Native install works|
|CPU Type||XEON D-1528||Intel i7-7567U||Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz|
|CPU Capacity||6 core 12 threads
1.9 GHz – 2.2 GHz
3.5 GHz – 4.0 GHz
|2 x 6 core (24 threads)
|RAM Type||4x DDR4||2x DDR4 SODIMM||8 x DDR3|
|RAM Capacity||128GB ECC RDIMM
64 GB Non-ECC UDIMM
|32GB Non-ECC||512GB ECC (LRDIMM)
256GB ECC (RDIMM)
|Intel Optane Ready||Yes||Yes||No|
|HDD Capacity||1x 2.5”||1x 2.5”||3 x 3.5/2.5|
|NVMe / M.2 SATA||YES||YES||NO|
|1Gbe Networking||2x 1Gbe||1x 1Gbe||2 x 1Gbe|
|10Gbe Networking||2x 10Gbe||1x Thunderbolt 40Gbps||1 x infiniband (40Gbps)|
|Video Port||VGA only||HDMI 4k||VGA only|
|Noise Comparison||2x Fans||Fan-less||4 x 8cm Fans(Chassis)|
|USB Capacity||2 Ports||4 Ports||2 ports|
|Price Comparison||US$799||US$630||US$801 (FULLY POPULATED)|
Because this is refurbished gear the cost can vary, people in the US would most likely find a better deal than I could because international shipping was huge from US sellers but cheaper units. And if your in the UK it would also most likely be cheaper too. The cost in the table above is the cost of the total purchase divided by 4, which includes:
- 4 x nodes
- 1200w platnium PSU
- 384GB RAM
- 8 x 6core CPU’s
Delivered to my door was Cost of the Unit $3800 AUD + customs $400 AUD = Total $4200 AUD which is about $3205 USD. This IMO was good bang for buck, it delivered on more resources than I currently had, left 2 free dimms per node so easy to expand to 128GB per Node totaling 512GB RAM for the lab if i needed to down the track.
I was going to re use the VSAN hard disks that I already had but running hybrid being able to fit 3.5″ disks allow this benefit. nothing special 256GB SSD and 2TB Sata HDD.
All that was left now was networking and a switch. I was tempted to give Ethernet over infiniband a go, switches are cheep it was on-board and 40Gbps was more than I needed. During some reading I got the impression the x3 no longer worked with 6.5 and thought I better stick with what I know (side note the 40Gbps nic shows up in vSphere and looks fine). At the moment Copper 10Gbe switches are too expensive and the Mellanox MT26448 10Gbe sfp cards are still on the HCL and are really cheap ended up getting a bundle of 5 cards with 5 x 3m dat cables.
The switch was the next thing I needed. I really wanted to have 10GB for VSAN but also vMotion etc. I ended up going with the TPLINK T1700G-28TQ It was $399 AUD and had 4 x 10GB sfp+ slots which can be used as well as the 24 x 1GBe ports.
BOM (to my door)
|SuperMicro 6027TR-HTQF (fully populated with the above hardware)
||$4200AUD / $3205USD|
|5 x Mellanox Technologies MT26448 10Gbe with 5 x 3m DAT cables||$200AUD / $152USD|
|TPLINK T1700G-28TQ||$399AUD / $304USD|
|Total||$4799 AUD / $3659 USD|
In total I have 48 core (96 Threads), 384GB RAM, 8TB of VSAN, 10Gbe networking. This was great The equivalent capacity E200/300 would be almost twice as much. Now granted the E200/300’s are newer, compact and quieter but just don’t cut it for what I need to do. If I had an apartment and needed to make sure they where quiet I might have to have gone with another solution but for me this lab was almost perfect.
As with the E200 and NUC’s vsan is running on on-board ACHI drivers, I am not trying to break any records and if I used infiniband I could have put a controller in. either way it still performs fine.
Now don’t get me wrong I know some are going to say this solution would be noisy and to be honest it is. not as loud as a Cisco 3750 but its still a 2RU rack server. this is really the only CON there is. But for me being quiet was not a requirement. I have a big arse garage and this is still in a 42 RU server rack in there. Can not hear a thing. E200 and NUC’s being basically silent this is a downside of my solution
Compared to the others it is double the power, under load the system appears to pull about 120 watts which is twice than the E200 and NUC’s but it is twice the compute as well.
I personally I don’t think I could have done too much better with the amount of resources I now have, It ticked all my boxes for what I was wanting out of a home lab. I hope I get 3 years out of this lab too.