ESXi Server Build Lessons Learned

While building the new ESXi box, I ran into a few issues that I've had to navigate through. Below is a list of those items.

• Dell Perc 5i RAID card - didn't work with DirectPath I/O enabled. After adding it to ESXi, it would 'disappear' under load and appear as a new drive after re-scanning. Discovered other users have had the same issue.
• The Broadcom NIC on the ASRock X79 Extreme 6 motherboard was supported by ESXi 5[0, 1]. I didn't need to purchase the 2 port PCI-e NIC. Will be returning this / selling  -- decided against this when the RAID controller card didn't work.
• Video card pass through - works great! However, did have to reserve all the memory for the VM up front while configuring the pciHole.start / pciHole.end values within the VM's configuration file. And, while using HDMI, it passes sound to my TV, so I didn't need a sound card. I did run into issues though when I tried using the HDMI extender.
• ESXi 5.1 had some problems with USB pass through. I had to downgrade to ESXi 5.0. (Discovered other users were having the same issue from blogs).
• ESXi pass through devices in 5.0 weren't listed properly (lots of 'unknowns'). However, 5.1 lists a few more of them correctly, so it was a good thing I started with 5.1 first before going to 5.0.
• USB 2.0 and pass through was a little different than expected. I was expecting the 10 on board ports to be divided in groups of 2 to pass through to VMs. However, the ASRock X79 Extreme 6 divided the ports onto 2 controllers. The first are for the 4 on the back and the 2 on the motherboard closest to the back of the motherboard. The second controller controls the other two mother board ports (I bought a 4 port USB bracket for $4 to get a connection to these)
• Other than only having 2 controllers, the USB pass through works great
• USB Boot of ESXi - this worked great. Highly recommended. However, I had to move to using an older 2.5" SATA drive since there were only 2 USB controllers, and I would rather have the USB ports going to the VMs (1 VM for keyboard / mouse and 1 VM for backup drive, USB TV tuner - which could be moved to another VM temporarily if necessary to mirror a 'dual boot' setup).
• USB 3.0 - currently not working (hope to get this working in ESXi 5.5)
• RAID - the on board RAID is not supported by ESXi 5. (currently looking at raid controllers to pull the 4 - 2.5 TB drives into a RAID 5 array -- see above note on RAID controller). The ports individually can be used within ESXi.
• ESXi free version of 5.1 and 5.2 only supports 32 GB of memory. I'll need to upgrade to 5.5 soon to get support for 64 GB (as soon as I have time to dedicate to a maintenance weekend!).
• PCI pass through - It appears that if you pass 1 of the PCI cards through, you have to pass both of them through. However, I haven't had any issues with assigning the PCI cards to separate machines.
• PCI sound card pass through - I was able to pass my PCI Sound Blaster Audigy 2 to a VM. But, it appeared to cause the VM to 'studder'. I'm not sure if this was related to the drivers (Windows 7); I will need to test this out again, as the first round of testing I was flushing out a lot of other items. However, this really won't be needed as the HDMI out on the video card pushes sound to the TV.
• UPS - UPS functionality is working, but currently pending resolving issues with the UPS and the Tripp-Lite software. The VM does see it when plugged in via USB as a battery, but the software isn't seeing it. I will need to test this on my laptop. -- Update Jan 2016, when I moved the UPS to be managed with my new hardware based FreeNAS, I couldn't get it working correctly. I migrated to an APC BackUP UPS PRO 1500 to work correctly with FreeNAS.
• VM Templates - I'm used to using VM Templates at work. However, this feature isn't available within ESXi free version. But, I was able to copy the vmdk files manually using the datastore browser and boot the OS ok (Windows 7). If you need to rename the virtual disk, there's a hidden file that gets copied that needs to be renamed and edited to reflect the new name of the vmdk file. I connected to ESXi via SSH and used VI to edit the file.
• SANS Digital 4 bay RAID tower - I'm not sure why, but I'm seeing very slow performance from this now. I don't remember having a problem with slowness, but I also wasn't moving very large files around. When copying (writing) files, it would start at around 70mb/sec, but would drop to around 6mb/sec (on RAID 5). When I went with no RAID (individual drives), it increased to around 25mb/sec. *Update - I think this might have just been the file copying protocol going slow due to the small size of files (i.e., copy file, check file, copy file, check file...), so I'll need to flush this out.
• Local Drives - I did have a local drive to the ESXi server go bad, and it really caused a lot of I/O issues. It was very difficult to get the ESXi host to do anything because it was running so slow, even after a reboot. I knew it was one of two drives (based on what was shown in the storage summary view), so I opted for powering off the host and pulling each of the drives one at a time until I found the culprit. It ended up being drive number two for me. With this happening, I don't recommend local drives if at all possible.

Below are some pictures of the DirectPath I/O Configuration. The first is when I tested with 5.1. A few items are labeled better than what it is within 5.0.0. The other two are from 5.0.0 with my current setup fully populated with cards.

ESX 5.1 DirectPath I/O Configuration

ESX 5.0.0 DirectPath I/O Configuration Part 1

ESX 5.0.0 DirectPath I/O Configuration Part 2

While using the ESX 5.0.0 pictures above:

• 00:10 USB #2 = The two USB 2 (4) connector ports on the motherboard closest to the south bridge
• 00:1b Audio = on board audio controller
• 00:1d USB #1 = All the USB 2 connector ports on the back I/O panel and the one USB 2 connector port on the MB closest to the back I/O panel (6 total)
• 00:1f = I'm not 100% on this, but I'm guessing the 4 SATA II ports (black connectors) and the 2 adjacent SATA III ports (gray connectors)
• 00:01 = My add on Hauppauge TV Tuner card (PCI-e)
• 00:02 = My add on Radeon video card (PCI-e)
• 00:03 = My add on Realtek based 2 port gigabit Ethernet card (PCI-e)
• 00:1c.1 = A set of USB 3 ports
• 00:1c.2 = A set of USB 3 ports
• 00:1c.3 = Motherboard's Broadcom gigabit Ethernet
• 00:1c:6 = A SATA controller; my best guess is the 2 SATA ports (grey connectors) adjacent to the 2 SATA III ports (far right on the picture below)
• 00:1c.7 = A SATA controller; my best guess is the single SATA port (grey connector facing upward) that is targeted for SATA BD/DVD/CD ROM drives
• 00:1e = Supports the two PCI slots; in the screen shot, I have the Sound Blaster Audigy 2 ZS (with the front panel) and a Digium TDM4 analog card installed.

Physical placement of cards (starting closest to CPU and moving outward):

1. Video card (PCI-e)
2. Video card (it's a wide card, it's not plugged into the PCI-e port)
3. Sound card (PCI)
4. Ethernet card (PCI-e)
5. Phone card (PCI)
6. TV card (PCI-e)

Here are some pictures of the motherboard for quick reference: