Intel X79 whitebox for vSphere 5 and Hyper-V 3

Updates:

Update August 8 2012: Added Microsoft Windows 2012 Hyper-V screenshots and link to blog post how to enable the Intel 82579V NIC

Update August 9 2012: Updating to the latest BIOS enables support for DirectPath I/O in VMware vSphere. Screenshot added

In an earlier blog post (found here) I mentioned that it is time for a new homebrew whitebox based on the Intel X79 chipset. With the X79 chipset it is possible to install 64GB of memory (8 x 8 GB). Because the 8 GB DIMMs are expensive on the moment, I decided to use 8 x 4GB DIMMs (total 32GB).

I decided to create one physical host for testing VMware vSphere 5, vCloud Director, VMware SRM, VMware View 5 etc. The possibility to create a physical ESXi5 server, create virtual ESXi  hosts on it and start VM on the virtual hosts is great! This feature is called nesting. How to do this, can be found on William Lam blog found here.

Components used for the VMware ESXi 5 / Microsoft Windows Server 2012 whitebox:

  • Intel i7-3820 CPU 3.60 GHz, 4 cores, with Hyper threading 8 cores
  • Zalman CNPS10X performance cooler
  • Asus P9X79 s2011 motherboard. Some specs:
      • Socket 2011
      • 8 DIMM slots, supports 64GB memory
      • Expansions slots: 2 x PCIe 3.0 (dual x16), 1 x PCIe (x8 mode), 2 x PCIe 2.0 x1, 1 x PCI
      • 2 SATA 6 Gb/s port, 4 x SATA 3 Gb/s
      • LAN: Intel 82579V Gigabit LAN controller
  • 2 x Corsair Vengeance DDR3- 1600 16GB (4 x 4) kit, total 32GB memory (max 64GB)

The case, power supply, graphical card, RAID controller and extra NIC(S) are reused. Here are some photos of the configuration:

image image
image image
image

When the hardware  configuration was done and tried to power on the system, nothing happened (black screen). The appears that BIOS of the motherboard didn’t know the i7-3820 CPU yet. The cool thing is that the motherboard has a function called “USB BIOS Flashback”. It is possible to flash the BIOS without CPU or memory installed. Here are the steps:

  • Download the latest BIOS from the Asus site;
  • Extract the BIOS on a USB stick;
  • Rename the BIOS file, example: rename “P9X79-ASUS-0906.ROM” in “P9X79.ROM” (important);
  • Place the USB stick in the USB port with the WHITE interior on the back;
  • Press the BIOS flashback button for 3 seconds and the light will begin to flash;
  • Don’t turn of the computer during the BIOS flash;
  • When the flashing light stop, the BIOS has been complete;

After the BIOS update was finished, the system boots and I was able to install VMware ESXi and Windows Server 2012 and enable the Hyper-V role.

vSphere 5 / ESXi 5 screenshots:

image image
Hyper threading enables 8 cores 32GB memory
image image
The onboard SATA controller is listed as Patsburg 6 Port SATA AHCI controller.

Software-RAID does not work

The hardware RAID controller is added as extra PCI card

The onboard Intel 82579V NIC is not supported in ESXi5. Use the procedure found here to add the NIC. Use at your own risk!

The Intel 82574L NIC is added as extra PCIe card.

image

The latest firmware includes support for DirectPath I/O

 

Microsoft Windows Server 2012

It is possible to install Microsoft Windows 2012 and enable the Hyper-V role. Here are some screenshots:

image image
image image

The onboard Intel 82579V NIC is not recognized in Windows Server 2012 by default. How-to enable the Intel 82579V NIC is explained in this explained in this blog post.

This whitebox is a great extension to my home lab!

[ad#banner]

VMware ESXi 5 whitebox NIC support

I tested the NICs below in my VMware ESXi 5.x whitebox server at home. Here the status:

NIC Recognized by VMware ESXi 5 Listed in ESXi 5 as
Intel PRO/1000GT Desktop Adapter PCI Yes Intel 82541PI Gigabit Ethernet Controller
Realtek RTL 8111E Yes Realtek 8168 Gigabit Ethernet
Intel Gigabit CT Desktop Adapter PCI-e Yes Intel Corporation 82574L
Intel 82579 Gigabit LAN controller No

You need the make a customized ESXi 5 ISO or VIB file.
This is a not supported configuration!
Intel Corporation 82579V orIntel Corporation 82579LM

To add for example the Intel 82579 chipset, create a customized ESXi 5 ISO.  This is very simple because some people have already done the hard work.

Here are the steps:

1. Download ESXi-Customizer (create by Andreas Peetz) found here.

2. Download the driver (created by Chilly) found here.

3. Start the ESXi-Customizer and follow the 3 steps:

2011-12-04 17h03_20

And you’re ready to create the customized VMware ESXi 5 ISO. The ISO supports the Intel 82579V and 82579LM NIC(s) found on many whitebox motherboards today.  Possible future updated version(s) of the driver can be found in the following post.

 

Time for new whitebox for your VMware vSphere or MS Hyper-V home lab environment?

When using a whitebox lab environment at home and like to test for example vSphere 5, vCloud Director, VMware View and MS Hyper-V (nested in VMware vSphere Glimlach) you need a lot of processor power and memory. In almost all whitebox lab environment the processor power is not the problem but the amount of memory is.

Till now the Sandy Bridge desktop boards support up to 32GB memory with only four DIMM slots on the motherboard. For 32GB you need 4 * 8GB DIMMs, 8GB DIMMs are very expensive on the moment when writing this post.

Intel Introduced the Sandy Bridge-E  processors and motherboards with the X79 based chipset that support this processors. This gives new possibilities for building a new whitebox home lab.

Processor

Intel introduced the Sandy Bridge-E or the 2nd generation Core i7 Extreme Processors. On the moment there are two Sandy Bridge-E processors available:

– Intel Core-I7-3960X 3,3GHz,15M L3-cache, list price around 950,00

– Intel Core-I7-3930K 3,2GHz,12M L3-cache, list price around 550,00

 

As you can see the processors are pretty expensive. The Sandy Bridge-E has the following features:

– Socket LGA2011;

– 6 cores (12 cores with Hyper-Threading);

– Quad channel DDR3-1600 memory controller;

– Supports Hyper-Threading, Intel VT-x, VT-d;

– PCI-Express 3.0 support;

– 40 PCI-Express lanes;

– Max TDP 130 W;

– Multiplier unlocked.

The 2nd generation Core i7 Extreme Processors can be compared here.

Begin 2012 Intel will release the Core I7 3820 Sandy Bridge-E processor. This  processor will support 4 cores (8 with Hyper-Threading) and have 10MB L3-cache. The price is not announced yet but will be much lower as the Intel 3930K and 3960X processors. When you buy a Sandy Bridge-E processor there is no CPU cooler in the box.

Motherboard

The Intel X79 chipset support the Socket LGA2011. To choose a motherboard you can for example check the following things:

– How much DIMM slots it has (some X79 motherboard have 4 DIMM slots);

– How much expansion slots it has;

– Price;

– How many USB ports and what speed they have;

– Type and number of SATA controllers;

– Type of number of NIC(s).

 

Important for a whitebox lab configuration is that the SATA controller (if you want to use local storage) and NIC(s) are supported by VMware ESXi. When choosing a motherboard with enough expansion slots you can always add extra RAID and NIC cards that are supported.

The most X79 based motherboards have 8 DIMM slots and supports up to 64GB memory.  8GB memory modules are expensive. So you can make a configuration with 8 x 4GB = 32 GB DDR-3 memory which is much cheaper. When 8GB memory modules become cheaper you can upgrade to 64GB memory.

Asus has a nice overview of all the X79 bases series motherboards they have, found here.

Shopping list

I made a shopping without the case, storage and the power supply. The prices are taken from the Tweakers pricewatch (Dutch) an can change every day.

 

Shopping list:

Component List price ()
Intel Core-I7-3930K 520,00
Asus P9X79 230,00
4 x 4 x 2GB DDR3-1600 memory (total 32GB) 160,00
Simple processor cooler 30,00
Simple graphic card PCI-E 25,00
Total 965,00   

 

 

Conclusion

With the Sandy Bridge-E processor and X79 based motherboard you can build a monster whitebox lab environment with the best performance on the moment.

The advantage of using one huge whitebox you can use nesting to run your VMware vSphere and MS Hyper-V environment(s) on one box. An other advantage of using a whitebox is that you can make a low noise system. It’s a lot of money for a whitebox home lab environment but if you you wait for the Intel Core I7 3820 Sandy Bridge-E processor (announced in January 2012) can save you around 270,00 (this is an assumption because the list prices are not available yet).

 

[ad#banner]