Whitebox

New whitebox for extending my home lab

For a couple of months I’m searching for an extra whitebox host for extending my home lab environment. My current lab whitebox is a Haswell based whitebox (see: link).  Here is an overview of the new lab environment:

lab environment

For the new whitebox I  had the following requirements:

  • Hardware such as NICs must be recognized by VMware ESXi
  • Use 32 GB memory or more
  • Low power consumption
  • Expandable
  • Small formfactor
  • Quiet
  • Possibility to run nested hypervisors such as VMware ESXi and Hyper-V
  • Remote Management
  • Possibility to create a VMware Cluster and use vMotion, HA, DRS and DPM with the existing Haswell host

I reviewed the following popular home lab systems:

  • Intel NUC
  • Apple Mini
  • Gigabyte BRIX

The main reason to NOT choose for one of the above systems is the only support 16 GB of memory. In November 2014 I  found a motherboard that passes all the requirements, after reading a review on a Dutch hardware website. The review was about the ASRock C2750 motherboard. After some additional research I ordered the following parts to build this whitebox:

  • ASRock C2750 motherboard
  • Kingston 4 x8GB, DDR3, PC12800, CL11, Total 32 GB memory
  • be quiet System Power supply 7 300W
  • Cooler Master Midi Tower N300 ATX

VMware ESXi boots from an USB stick and the VM’s are placed on a iSCSI target so no extra storage is needed. The above parts cost me around € 735,00.

The ASRock C2750D4I motherboard has the following specifications:

  • Mini ITX motherboard
  • CPU: Intel Avoton C2750 64-bit 8 cores processor (passive cooled)
  • Graphics: ASPEED AST2300 16 MB
  • Memory: 4 x DDR3 DIMM slots, max: 64 GB memory
  • Controller: Intel C2750: 2 SATA3, 4 x SATA2 Marvell SE9172 2 x SATA 3, Marvell SE9230 4 x SATA3. Total of 12 SATA ports.
  • NIC: Dual Intel i210 Gigabit LAN adapter
  • 1 x PCIe 2.0 x8 slot
  • Remote Management: BMC Controller with IPMI dedicated LAN adapter
  • TPD of 20 watt
ASrock IMG_3839

CPU

The Intel Avoton C2750 is an atom based processor and contains 8 cores. It is passive cooled and quiet. The Avoton processor is 64-bit and supports Intel VT-x with Extended Page tables (EPT), so it is possible to nest hypervisors such as ESXi and Hyper-V. The Atom processor with 8 cores gives enough CPU performance for my lab environment.

Memory

The motherboard contains 4 memory banks with a maximum of 64 GB DDR3 memory (4 x 16 GB). I choose to use 4 x 8 GB Kingston DDR3, PC12800, CL11 dimms because of the price. 16 GB modules are to expensive on the moment. The motherboard has 32 GB of memory.

NICs

The ASRock C2750D4I system contains a dual Intel i210 Gigabit LAN adapter. The Intel i210 adapters are standard recognized by ESXi 5.5 and Windows Server 2012 R2. No additional modifications or drivers are needed.

Power consumption

The 300 W Power Supply is more than enough. The processor has a TPD of 20 watt. This whitebox consumes around 35 W with a couple of VMware VMs on it.

IMG_3949

The ASRock C2750D4I system is part a VMware cluster with Distributed Power Management (DPM) enabled. When DPM kicks in only 4 watt is used.

Remote Management

Management and remote control is possible because of  the BMC (Baseboard Management Controllers) and IPMI (Intelligence Platform Management Interface).

remote remote2

VMware ESXi support

On the ASRock C2750D4I system, VMware ESXi 5.5 Update 2 with the latest updates is installed.

esxicpu esxi

The Intel i210 Gigabit NICs and the Avoton AHCI controllers are recognized out of the box. So VMware VSAN (unsupported) could be an option to use.

esxiintel esxicontroller

The SATA 6 Gb/s controller is not recognized by default. Follow the instruction explained in the following blog to enable the controller:

  • How to make your unsupported SATA AHCI Controller work with ESXi 5.5 and 6.0, link.

Update March 23, 2015: Today I installed VMware ESXi 6.0 on the C2750. Everything seems to work.

 Windows Hyper-V support

As test I installed vNext Server Technical Preview on the ASRock C2750D4I system (with SSD as local storage) with the Hyper-V role enabled. The two Intel i210 Gigabit NICs are recognized out of the box. It has great performance.

vnext hyper-v hvtaskmgr01

Conclusion

The ASRock C2750D4I motherboard is a great system for building or extending a home lab environment based on VMware or Hyper-V. This board gives enough performance for a home lab and meets all the requirements I had for an additional whitebox host. I use it mainly for nesting VMware ESXi and Hyper-V hypervisors.

Haswell low power whitebox for ESXi and Hyper-V

I was searching for a new whitebox for my home lab.  I had the following requirements for the new whitebox:

  • Low power consumption for 24×7 running
  • >16 GB memory
  • Expansion slots for PCI(-E) cards
  • Good performance
  • Low noise

A couple of weeks ago Intel released the new 4th generation Haswell CPUs that consumes less power. Seems to be interesting option for building a low power consumption whitebox. So I did some research and ordered the following hardware components:

CPU i5 4570S Boxed foto-7_thumb1
Motherboard Gigabyte GA-Z87-D3HP
Memory Corsair Vengeance 4 x 8 GB DDR3 PC3-12800 (DDR3-1600) 32GB
Power Supply Seasonic G-360 80 Plus Gold

The following components I reused:

  • The case
  • 2 Intel PCI-e NICS
  • A SSD drive

CPU

Haswell is the codename for the 4th generation Intel Core processors. One of the big improvements of the Haswell CPUs is the idle power consumption. The Intel S-version is also a low-power CPU.  It contains 4 cores and has a 65W TDP. The processor supports vPro, VT-x, VT-d, EPT etc.   A CPU cooler is in the box included. For the full specifications look here.

Motherboard

The Gigabyte GA-Z87-D3HP is a socket 1150 motherboard. The board has  4 memory sockets that support  DDR3 memory up to 32 GB.  It has onboard graphics , 6 x SATA 6 Gb/s connectors, a Intel WG-217v LAN adapter and the following expansion slots:

  • 1 x PCI Express x16 slot
  • 1 x PCI Express x16 slot, running at x4
  • 2 x PCI Express x1 slots
  • 2 x PCI slots

Memory

For the memory I choose the Corsair Vengeance CML32GX3M4A1600C10 4 x 8  PC3-12800 (DDR3-1600) CL10 kit. All the four memory sockets on the board are filled with a 8 GB module (total = 32 GB memory).

Power Supply

The Seasonic G-360 power supply has a 80 Plus Gold certification. This is a great power supply with high efficiency and low noise..

 

Hypervisor support

I tested VMware ESXi 5.1 Update 1 and vSphere 5.5 and Microsoft Windows Server 2012 with the Hyper-V role installed.

VMware ESXi

The onboard Intel I217-V NIC is not recognized  by ESXi 5.1 Update 1. To get the I217-V NIC working in ESXi5.x read the blog post found here. I reused 2  Intel PCI-e NICs, one for LAN and the other for iSCSI and NFS traffic. The onboard SATA controller  (Lynx Point AHCI) is recognized. I use an existing SSD for booting ESXi and running some important VMs. But is is possible to boot ESXi from USB stick. The other VMs are on NAS device. Passthrough is supported by the CPU and motherboard.

image image
image image

 

Microsoft Windows Server 2012 with the Hyper-V role

I tested Windows Server 2012 with the Hyper-V role enabled. The The onboard Intel I217-V NIC is not recognized by default. In the “Enable the Intel I217-V NIC in Windows Server 2012” blog post I explain how to enable the I217-V NIC Windows Server 2012.

 

image image
image image

Power consumption

The whitebox consumes a maximum between 40-50 W. When idle (and that is often) it consumes only 28 a 29 W!

foto (8)

The whitebox components cost about €615. and will be used for running VMs 24×7.  The whitebox meets all the requirements I had.

Enable the Intel I217-V NIC in Windows Server 2012

My new Gigabyte Z87-D3HP motherboard with support for 4th Generation Intel® Core™ processors (codename Haswell) has a onboard Intel I217-V NIC. The NIC is not recognized and supported in Windows Server 2012. As with the “Enable the Intel 82579V NIC in Windows Server 2012” blogpost it is possible to enable the NIC in Windows Server 2012. Here are the steps:

  • To be able to modify the drivers you need to run the the following commands: 
bcdedit -set loadoptions DISABLE_INTEGRITY_CHECKS 
bcdedit -set TESTSIGNING ON
  • Reboot the system
  • Download the Intel drivers from the motherboard vendor or from Intel. In my case I used the Intel Windows 8 64-bit  drivers Gigabyte has available for the Z87-D3HP motherboard on there website.

image

  • Save the drivers  and extract them
  • Open the folder PRO1000
  • Open the folder Winx64
  • Open the folder NDIS63
  • Open the e1d63x64.inf file, I used Notepad++  to edit the file
  • In the [ControlFlags] section delete the 3 lines (see 1)

 

image

  • Select and copy the three E153B lines ((see 2)
  • Paste the 3 lines in the [Intel.NTamd64.6.2] section below the %E155ANC lines (see 3)
  • After the modifications the e1d63x64.inf file must look like this:

image

  • Save the file
  • Install the Intel driver
  • During the installation the you got a “Windows can’t verify the publisher of this driver software” warning, select “Install this driver software anyway”  and continue

image

  • After the installation the Intel I217-V Gigabit NIC is recognized and enabled.

image

  • Enable the driver integrity checks and disable test signing again by using the following commands: 
bcdedit -set loadoptions ENABLE_INTEGRITY_CHECKS
bcdedit -set TESTSIGNING OFF
  • Reboot the system