Welcome to vSphere-land!

vSphere 4.1 is finally out, here’s all the links you need to get you started with it, keep checking back as this post is a work in progress and will keep growing. Be sure and check out the valuable KB articles at the bottom of this post.

Features:

What’s New in VMware vSphere 4.1
What’s new in Networking with vSphere 4.1 (VMware white paper)
What’s new in vCenter Server 4.1 (VMware white paper)
What’s new in Availability & Resource Manager in vSphere 4.1 (VMware white paper)
What’s new in Performance for vSphere 4.1 update (VMware white paper)
What’s new in vSphere 4.1 storage (VMware white paper)

Release Notes:

VMware vSphere 4.1 Release Notes - ESX Edition
VMware vSphere 4.1 Release Notes - ESXi Edition
vSphere CLI 4.1 Release Notes

Other:

vSphere Hypervisor FAQ (Free ESXi)
ESXi 4.1 Migration Guide (VMware white paper)
VMware vSphere: The CPU Scheduler in VMware ESX 4.1 (VMware white paper)
Understanding Memory Resource Management in VMware ESX 4.1 (VMware white paper)
Managing performance variance of applications using Storage IO control (VMware white paper)
Microsoft Office SharePoint Server 2007 Performance on VMware vSphere 4.1 (VMware white paper)
What is new in vSphere SDKs 4.1 (YouTube video)
vSphere 4.1 and EMC - VAAI - Hardware Accelerated Full Copy (YouTube video)

Blog posts:

Tidbits on the new vSphere 4.1 release (vSphere-land)
Test ESXi 4.1 today, migrate smoothly from ESX tomorrow (SearchVMware.com)
VMware vSphere 4.1 HA and DRS clustering improvements (SearchVMware.com)
What’s new in ESXi 4.1: Active Directory integration, Tech Support Mode and more (SearchVMware.com)
VMware VAAI pros and cons and the hidden fourth primitive (SearchVMware.com)
VMware vSphere 4.1: Not the Typical .1 Release (Chris Wolf)
It’s here! The latest version of ESXi (ESXi Chronicles)
VMware vSphere 4.1: Advancing the Platform for Cloud Computing (The Console)
Proportional Allocation of Shared Storage Resources using Storage I/O Control (Vroom!)
Understanding Memory Resource Management in VMware ESX Server 4.1 (Vroom!)
Got Network I/O Control? (VMware Networking blog)
Steve Herrod, VMware’s CTO, Introduces VMware vSphere 4.1 (VMware TV)
Introducing VMware Sphere Hypervisor 4.1 - the free edition of VMware vSphere 4.1 (vSphere Blog)
Upgrading to SRM 4.1 - including upgrading to vSphere VirtualCenter 4.1 (VMware Uptime)
VDR and vSphere 4.1 compatibility (VMware Uptime)
A Few Gotchas With vSphere 4.1! (Daily Hypervisor)
PowerCLI 4.1 is out (PowerCLI blog)
vDS in PowerCLI 4.1 (PowerCLI blog)
PowerCLI 4.1: A fork in the road (Poshoholic)
VIDEO: New vSphere 4.1 ESXi Tech Support Mode (VMware Videos)
VIDEO: New vSphere 4.1 Windows Active Directory Authentication (VMware Videos)
VIDEO: vChat - Episode 1 - New vSphere 4.1 and Top 15 Killer Features (VMware Videos)
vSphere 4.1 and vStorage APIs for Array Integration (VAAI) (Virtual Storage Guy)
VMware vSphere VAAI Demo with NetApp (Virtual Storage Guy)
The Migration From ESX to ESXi is Happening. Moving Configurations (Kendrick Coleman)
vSphere 4.1 Memory Enhancements - Compression (NTPro.nl)
vSphere 4.1 - Virtual Serial Port Concentrator (NTPro.nl)
VMware boosts performance with vSphere 4.1 (SearchVirtualDataCentre)
Performance Improvements on vSphere 4.1 (another Case Study) (My Virtual Cloud)
vSphere 4.1 - What do the vStorage APIs for Array Integration mean to you? (Virtual Geek)
vSphere 4.1: VPXA Holds ESX Host at Ransom (Virtual VCP)
What’s New in vSphere 4.1 - Storage I/O Control (Technodrone)
What’s New in vSphere 4.1 - Network I/O Control (Technodrone)
VMware vSphere 4.1 released - What’s new? (Gabe’s Virtual World)
vSphere 4.1, VMware HA New maximums and DRS integration will make our life easier (Yellow Bricks)
New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 1 & Part 2 & Part 3 (Virtually Ghetto)
New way of enabling and disabling services using vSphere 4.1 (Virtually Ghetto)
ESXi 4.1 - Major Security Issue (Virtually Ghetto)
resxtop & vi-fastpass Downgraded Feature In vMA 4.1 (Virtually Ghetto)
ESXi 4.1 - Major Security Issue - The Sequel And The Workaround (DeinosCloud)
vSphere’s Total Lockdown (Everyday Virtualization)
VMware vSphere Hypervisor (ESXi) 4.1 kickstart - A.K.A. official “touchfree” ESXi installs (get-admin)
What’s New in vSphere 4.1 API? (Double Cloud)
vSphere 4.1 and ESXi (Virtualization Practice)
DPM scheduled tasks (Frank Denneman)
VM to Hosts affinity rule (Frank Denneman)
VMware vSphere 4.1 (Virtualised Reality)
vSphere 4.1 - Batteries (Client) Not Included in ESX/ESXi (Jase’s Place)
vSphere 4.1 Hidden Gem: Host Affinity Rules (Latoga Labs)
USB passthrough in vSphere 4.1 (vStorage)
VMware vSphere 4.1 Storage IO Control (SIOC) understanding (GeekSilver’s Blog)

News articles:

VMware Introduces Enhanced Virtualization Offerings for Small and Midsize Businesses (VMware news release)
VMware Advances Foundation for Cloud Computing With VMware vSphere 4.1 and Expanded Virtualization Management Portfolio (VMware news release)
VMware throws bone to enterprises, SMBs with vSphere 4.1 (SearchServerVirtualization)
VMware adopts per-VM licensing, pricing (SearchServerVirtualization)
First look: VMware vSphere 4.1 keeps the virtualization crown (Network World)
Will vSphere 4.1 help VMware compete with Microsoft, Citrix for SMB users? (Infoworld)
VMware scales up vSphere virt, rejigs prices (The Register)
VMware to change the vSphere architecture significantly (Virtualization.info)
Release: VMware vSphere 4.1 (Virtualization.info)
vSphere 4.1 Flexes Muscles, Trims Costs (Virtualization Review)
Why small businesses should care about VMware vSphere 4.1 (PC Pro)

Downloads:

Main Download Page
ESX 4.1
ESXi 4.1
vCenter Server 4.1
vSphere Hypervisor (Free ESXi)
vSphere Management Assistant (vMA) 4.1
vSphere CLI 4.1
vSphere PowerCLI 4.1
Cisco Nexus 1000 AV1 Virtual Ethernet Module

Documentation:

Main and Documentation sets:

Main documentation link
Download the complete ESXi 4.1 Embedded and vCenter Server 4.1 Documentation Set
Download the complete ESXi 4.1 Installable and vCenter Server 4.1 Documentation Set
Download the complete ESX 4.1 and vCenter Server 4.1 Documentation Set

Individual Docs:

Documentation Roadmap
Configuration Maximums for VMware vSphere 4.1
VMware vSphere Compatibility Matrixes
Introduction to VMware vSphere
Getting Started with ESX
ESX and vCenter Server Installation Guide
Upgrade Guide
Datacenter Administration Guide
Virtual Machine Administration Guide
vSphere Web Access Administrator’s Guide
ESX Configuration Guide
Resource Management Guide
Availability Guide
Fibre Channel SAN Configuration Guide
iSCSI SAN Configuration Guide

ESXi:

Getting Started with ESXi Installable
Getting Started with ESXi Embedded
ESXi Installable and vCenter Server Setup Guide
ESXi Embedded and vCenter Server Setup Guide
ESXi Configuration Guide

Upgrade:

Upgrading to ESX 4.1 and vCenter Server 4.1 best practices (1022104)
Upgrading ESX 4.0 to ESX 4.1 (1022140)
vSphere 4.1 upgrade pre-installation requirements and considerations (1022137)
Performing an offline upgrade from ESX 3.x to ESX 4.x (1009440)
How to: Upgrade to vSphere 4.1 (VMGuru.nl)
ESX host upgrade to 4.1 using esxupdate (vReality)
Upgrade ESXi4.0 to ESXi4.1 - The Unofficial Method (DeinosCloud)
Upgrading ESXi 3.x/4.0 to 4.1 - “Failed to read the upgrade package metadata.xml” (A. Mikkelsen)

Install:

Installing ESX 4.1 and vCenter Server 4.1 best practices (1022101)
Troubleshooting ESXi 4.1 Scripted Install errors (1022308)

KB Articles:

VMware ESX and ESXi 4.1 Comparison (1023990)
Changes to DRS in vSphere 4.1 (1022842)
Changes to VMware High Availability in vSphere 4.1 (1022843)
Changes to VMware Support Options in vSphere 4.1 (1023118)
Changes to vMotion in vSphere 4.1 (1022851)
Changes to Fault Tolerance in vSphere 4.1 (1022844)
vStorage APIs for Array Integration FAQ (1021976)
Overview of Active Directory integration in ESX 4.1 and ESXi 4.1 (1021970)
Improvements in handling thin provisioned devices (1021272)
ESX 4.1 and ESXi 4.1 root passwords are authenticated up to only 8 characters (1024500)
vCenter Server 4.1 fails to install or upgrade with the error: This installation package is not supported by this processor type (1019144)
Powering on virtual machines fails with the error: MAX VCPUs limit is reached (1020121)
Using Tech Support Mode in ESXi 4.1 (1017910)
USB support for ESX/ESXi 4.1 (1022290)
I/O Statistics in vSphere 4.1 (1021953)
Lockdown mode configuration after upgrading from ESXi 4.0 to 4.1 (1021935)
vSphere 4.1 upgrade pre-installation requirements and considerations (1022137)
Load Based Teaming in vSphere 4.1 (1022590)
Migrating to the vCenter Server 4.1 database (1021635)
Update Manager 4.1 patch repository features (1021695)
Upgrading to ESX 4.1 and vCenter Server 4.1 best practices (1022104)
Upgrading ESX 4.0 to ESX 4.1 (1022140)
Using vShieldZones 1.0 with ESX 4.1 (1022536)
Deploying ESXi 4.1 using the Scripted Install feature (1022263)
Network I/O Resource Management in vSphere 4.1 with vDS (1022585)
Changing the number of virtual CPUs per virtual socket in ESX/ESXi 4.1 (1022289)
Troubleshooting ESXi 4.1 Scripted Install errors (1022308)
Troubleshooting Storage I/O Control (1022091)
Installing ESX 4.1 and vCenter Server 4.1 best practices (1022101)
vCenter Server 4.1 network port requirements (1022256)
Configuring IPv6 with ESX and ESXi 4.1 (1021769)
Recreate vSphere 4.0 lockdown mode behavior in vSphere 4.1 (1017628)
Securing Credentials in vMA 4.1 (1017669)
Accessing USB storage and other USB devices from the service console (1023976)

I’ve been in the Beta for 4.1 for quite some time and also have participated in VMware’s blogger briefings on vSphere 4.1. The major features of 4.1 will no doubt be covered by many, this post is simply a collection of tidbits on vSphere 4.1 that I’ve picked up over the last few months. Keep checking back as this post is a work in progress and will grow as I add more to it.

• vCenter Server 4.1 now requires a 64-bit Windows operating system, it will not installl on a 32-bit Windows OS.
• This is the last major release to support ESX and the Service Console, ESXi will be the only choice in the next major release due out next year. Almost all vendors now support using APIs instead of Service Console agents.
• The Host Update utility that installed with the vSphere Client is now gone in 4.1. You’ll notice that the size of the vSphere Client download is much smaller in vSphere 4.1 as a result. To patch ESXi hosts you need to either use Update Manager or the CLI utilities.
• The vSphere Client is no longer bundled with ESX & ESXi installations, this was done to reduce the build size so the ISO file that is used to install ESX & ESXi with is smaller. In previous versions when you access the web interface of an ESX or ESXi host you had the option to download the vSphere Client directly from the host to install it on a workstation. Having the vSphere Client available as a download from the host was more for convenience. It can still be downloaded from VMware’s website or from the vCenter Server’s web interface. Doing this reduced the size of the ESX ISO from 814MB (4.0) to 631MB (4.1) and reduced the size of the ESXi ISO from 353MB (4.0) to 290MB (4.1).
• When installing vCenter Server you can now choose the JVM maximum memory size for the Tomcat application server that is installed on the vCenter Server. This can drastically reduce or increase the amount of RAM that the JVM uses which is the biggest memory consumer on the vCenter Server. For small (<100 hosts) the JVM is set to 1024MB, medium (100 - 400 hosts) the JVM is set to 2048MB and for large (>400 hosts) the JVM is set to 4096MB. This setting can be changed at anytime by loading the Configure Tomcat utility in the VMware start menu folder and selecting the Java tab.
• Application Monitoring is a new feature of HA which will monitor applications that have been modified to transmit a heartbeat that vSphere can detect and restart a VM if the application stops responding (crashes). This adds another layer of the stack that HA can monitor uptime for (Host, Operating system, Application), currently no applications support this but will there will probably be some in the future.
• Name/case changes:
  • VMotion -> vMotion
  • Storage VMotion -> Storage vMotion
  • ESXi free or standalone edition -> vSphere Hypervisor
  • ESX & ESXi paid editions -> Hypervisor Architectures
• The VMFS disk format has been upgraded from version 3.33 (vSphere 4.0) to version 3.46 (vSphere 4.1), like some prior updates this one is minor and not worth re-creating your VMFS volumes for. The new VMFS driver (not the disk format) in vSphere 4.1  includes new storage offloading algorithms which are introduced in vSphere 4.1 via VAAI (vStorage APIs for Array Integration).
• A new feature in the Performance views will show you VM/host power usage in Watts. ESX 4.1 can gather host power consumption on platforms which provide that data through IPMI sensors. Newer platforms from HP, Dell, IBM, and Fujitsu are supported, and there is a way to teach ESX on how to get host power consumption on other systems which have host power consumption IPMI sensors. If you go to vClient and click on “Performance” and then choose “Power” from the drop down list at the top, then you should see host power consumption chart if the host is supported. However this feature will not work by default and is considered experimental. To enable it click on the Configuation Tab on an ESX Host, in the Software box, click Advanced Settings. In the list of options click on Power and scroll down to near the end of the list on the right hand side and you will see a setting called Power.ChargeVMs , change this value to 1 and click OK.  This is one more step to make this work, you need to edit the /usr/share/sensors/vmware file and add information for your server, by default there is an example of a Fujitsu server showing the syntax that should be used when describing sensors. VMware added this functionality so that OEMs and customers can add support for their systems without the need for VMware to update sensord itself. For example for an HP385 G6 servers which has two IPMI sensors called “Power Supply 1″ and “Power Supply 2″, you can add a single line to the sensor file like this: ”default:power:HP:ProLiant DL385 G6:Power Supply 1,Power Supply 2:WATTS” You’ll need to restart sensord on the server afterwards. The names of the sensors put in these configuration files must match the correct vendor sensor name, the vendor and product names can be anything you want. VMware plans on having OEMs produce their own configuration files for sensord and either send them to VMware or ship them as part of their oem.tgz custom archives.
• Other new advanced power settings include:
  • Power.UsePStates
  • Power.UseCStates
  • Power.UseStallCtr
  • Power.CStateMaxLatency
  • Power.CStateResidencyCoef
  • Power.CStatePredictionCoef
  • Power.PerfBias
  • Power.PerfBiasEnable
  • Power.ChargeVMs
  • Power.ChargeMemoryPct
• These settings control what the Custom power policy does. By default Custom is the same as Balanced, but you can tweak those settings to change it. For example, in 4.1, the Balanced policy doesn’t use C-states (only P-states), while the Low policy uses both but also changes some other parameters to be more aggressive. You could make a Custom policy that is the same as Balanced except that C-states are used. Each option has a doc string that briefly explains what it does, which should be available in the UI. If the doc string starts “In Custom policy”, that option affects only the Custom policy. Otherwise it’s applied regardless of policy. There is not a lot of documentation on these settings as VMware anticipated practically no one would actually want to play with a custom policy or tweak the other options; they’d just choose from one the three predefined ones (High Performance, Balanced, or Low Power).
• The difference between a processor C-state and a P-state is this:
  • A P-state can alter the frequency and voltage of a CPU core from a low state (P-min) to the max state (P-max), this can help save power for workloads that do not require a CPU core full frequency.
  • A C-state shuts down a whole CPU core so it cannot be used, this is done during periods of low-activity and saves more power than simply lowering the CPU core frequency.
• A new feature in vSphere 4.1 call iSCSI Boot Firmware Table (iBFT) allows booting from an iSCSI target using software initiators. Previously only hardware initiators on ESX supported this feature. This feature has some restrictions though; it will only work ESXi (no ESX) and the only currently supported network card is the Broadcom 57711 10GBe NIC. When booting from software iSCSI the boot firmware on the network adapter logs into an iSCSI target. The firmware than saves the network and iSCSI boot parameters in the iBFT which is stored in the host’s memory. Before you can use iBFT you need to configure the boot order in your server’s BIOS so the iBFT NIC is first before all other devices. You than need to configure the iSCSI configuration and CHAP authentication in the BIOS of the NIC before you can use it to boot ESXi from. The ESXi installation media has special iSCSI initialization scripts that use iBFT to connect to the iSCSI target and present it to the BIOS. Once you select the iSCSI target as your boot device the installer copies the boot image to it. Once the media is removed and the host rebooted the iSCSI target is used to boot and the initialization script runs in first boot mode which configures the networking which afterwards is persistent.
• Memory Compression is a new feature to version 4.1 of vSphere than can offer VMs performance benefits. It provides a mechanism for swapping out memory which is between that of physical memory and disk, and works when a VM’s memory is under contention. The performance gains are had by the memory not being swapped out to slower disk based storage.
• Load-based teaming found in vSphere 4.1 provides the ability to dynamically adjust the teaming algorithm which will balance the network load across a team of physical adapters connected to a vNetwork Distributed Switch.
• A new feature with vSphere 4.1 is the extra storage performance and NFS statistics that can be accessed via the performance charts and esxtop. These metrics provide a useful insight into storage throughput and any host or virtual machine (VM) latency.
• You will receive a prompt when creating a Distributed vSwitch to choose a vDS version, either 4.0 or 4.1, if your hosts are all 4.1 you can choose the 4.1 version which enables additional features such as Network I/O control and dynamic load balancing.
• vSphere 4.1 added another new feature to HA that checks the Operational Status of the cluster. Available on the cluster summary tab, this detail window called Cluster Operational Status displays more information about the current HA operational status, including the specific status and errors for each host in the HA cluster.
• vStorage APIs for Data Protection (VADP)  now offer VSS quiescing support for Windows 2008 and Windows 2008 R2 servers. This enables application-consistent backup and restore operations for Windows 2008 and Windows 2008 R2 applications.
• VMware multi-core virtual CPU support lets you control the number of cores per virtual CPU in a virtual machine. This capability lets operating systems with socket restrictions use more of the host CPU’s cores, which increases overall performance. You can configure how the virtual CPUs are assigned in terms of sockets and cores. For example, you can
  configure a virtual machine with four virtual CPUs in the following ways:
  • Four sockets with one core per socket
  • Two sockets with two cores per socket
  • One socket with four cores per socket
• Using multi-core virtual CPUs can be useful when you run operating systems or applications that can take advantage of only a limited number of CPU sockets. Previously, each virtual CPU was, by default, assigned to a single-core socket, so that the virtual machine would have as many sockets as virtual CPUs. When you configure multicore virtual CPUs for a virtual machine, CPU hot Add/remove is disabled. TO set this in the vSphere Client inventory, right-click the virtual machine and select Edit Settings. Select the Hardware tab and select CPUs and select the number of virtual processors. 4 Select the Options tab and Click General in the Advanced options list, click Configuration Parameters, click Add Row and type cpuid.coresPerSocket in the Name column. Type a value [2, 4, or 8] in the Value column. The number of virtual CPUs must be divisible by the number of cores per socket. The coresPerSocket setting must be a power of two. Click OK and power on the virtual machine. You can verify the CPU settings for the virtual machine on the Resource Allocation tab.

After writing my first book I swore I’d never write another, it’s a relatively big investment in time with a fairly small financial return. Ask any author and they’d probably say the same thing right after finishing a book. Well fortunately that feeling does not last and the desire to write another eventually returns. I took my time preparing for this one, I didn’t feel comfortable writing about vSphere until I had used it for many months to gain as much knowledge and experience as I could with it first. When I did get started with it I wrote the whole book in about 2 months time. I did have a little help though, I approached Simon Seagrave to see if he would contribute by doing a few chapters. He was very happy to be onboard and ended up doing the chapters on Performance and vSphere Home Labs. It was great working with him and I think the book turned out pretty good. Due to publisher constraints we had to fit everything into 350 pages which was quite challenging and meant we had to trim some fat and eliminate many figures. The end result was a book packed with information on vSphere including information on the not yet released next version. I was lucky to have John Troyer return to do the foreword on this book which is due out right before VMworld. You can find out more about the book on Pearson’s website and it’s also available on Amazon for pre-order. Not only will this book give you great information, it will look cool on your book shelf!

I have 2 sessions in consideration for VMworld 2010 in the Virtualization 101 track. The first is a deep-dive session on vSphere features, I’m the curious type and always have to know how things work behind the scenes. As a result I do a lot of research trying to figure out the technical magic behind features like VMDirectPath, VMCI, Fault Tolerance, DVFS, etc. Once I have a firm understanding of how things work I can better explain it to everyone else so they can understand it as well without having to do all the work I did to find out. The second session is on home labs and small vSphere environments. I’ve had a lot of experience recently with different home lab configurations and done a lot of research on supported configurations, hardware and shared storage options. I’ve gone from building a powerful desktop to use with VMware Workstation and running ESX/ESXi as VMs to using low-cost brand name servers running ESX/ESXi bare metal. For a pretty affordable price I’ve built a pretty cool lab that any VMware admin would enjoy having. So if you’d like to hear more about either topic head on over to the VMworld website and vote for me.

Vote here: http://vmworld.com/community/conferences/2010/cfpvote/v101

Title: Building an affordable vSphere environment for a lab or small business

Session Id:V18328

Abstract:“This session will cover how to build a vSphere home lab or an environment for use in a small business. It will cover the following areas: 1) How to choose the hardware, including white box hardware, brand name hardware, network components and storage 2) What the hardware compatibility guide means and the consequences of using unsupported hardware 3) Making sure the hardware you choose will work with vSphere 4) Using VMware Workstation to run nested hypervisors and virtual machines 5) Using ESX or ESXi to run nested hypervisors or virtual machines 6) Installing ESXi on to a USB flash drive 7) How to choose and use affordable shared storage devices Using free ESXi and free administration tools to manage it 9) Using the affordable Essentials editions to gain big features at a reduced cost 10) Putting it all together to make a virtual environment”

Type: Breakout Session

Track:Virtualization 101

Speaker: Eric Siebert Company: Boston Market

According to the original vSphere feature list there is a new security feature called “VMkernel Protection” that uses a technology called Trusted Platform Module (TPM) to add a layer of protection to the VMkernel. The VMkernel (hypervisor) is the most critical component of a virtual host because if it is compromised the VM’s running on it can easily be compromised. Therefore VMware introduced a new protection mechanism in vSphere to ensure the integrity of the VMkernel both on disk and in memory. Here is how it is described by VMware:

VMkernel Protection - As part of ongoing efforts to protect the hypervisor from common attacks and exploits, mechanisms were introduced to assure the integrity of the VMkernel and loaded modules as they reside on disk and in memory. Disk-integrity techniques protect the boot-up of the hypervisor using the Trusted Platform Module (TPM), a hardware device embedded in servers. To ensure the authenticity and integrity of dynamically loaded code, VMkernel modules are digitally signed and validated during load-time. These disk integrity mechanisms protect against malware, which might attempt to overwrite or modify VMkernel as it persists on disk. VMkernel also uses memory integrity techniques at load-time coupled with microprocessor capabilities to protect itself from common buffer-overflow attacks that are used to exploit running code. These techniques create a stronger barrier of protection around the hypervisor. See the ESX Configuration Guide and the ESXi Configuration Guide.

Having a strong interest in security I was curious about this feature and wanted to try it out so I did some research on it. TPM is a security specification developed by Trusted Computing Group (TCG) that uses cryptographic keys to protect information. It relies on a TPM chip which has a unique RSA key burned into it and is capable of performing platform authentication and can be used to verify that software has not been changed. vSphere can use TPM to digitally sign VMkernel modules and validate them when the host is starting up to protect against malware that might overwrite them. This feature is similar to the Windows File Protection feature that Microsoft has built-in to Windows to prevent critical system files from being modified or overwritten.

TPM is integrated into processors and chipsets so just like every other technology Intel has their version of it and AMD their own. Intel’s is called Trusted Execution Technology (TXT) which has been available for some time and AMD’s is called Secure Execution Mode (AMD has very little information on this) and is not widely available. For TPM to work you must have both a CPU with the necessary processor extensions for TPM and a chipset that supports TPM. TPM uses Platform Configuration Registers (PCRs) that are like containers that can hold 160-bit values in them in the following manner:

• At boot PCRs are all initialized to a known value (either 0 or -1)
• An application can then measure things by computing its hash value
• The resulting measurement is inserted into a PCR, this process is called “extending the PCR”
• PCRs can be extended multiple times until a final value is calculated
• Each code segment is measured and validated and control passes from one code segment to the next
• PCRs represent an accumulated measurement of the history of the executed code beginning with power-up
• TPM signing keys can be used to sign the values of PCRs
• The system state can then be verified from the hashes that get stored into the PCRs

The technology behind TPM is a bit complex and if you wish to read more there are some great resources at the end of this post that you can check out. As I wanted to see this technology in action I ordered a TPM chip for one of our servers so I could try it out. The chips are fairly cheap, for HP servers they are about $39. They consist of a small little circuit board that plugs into a TPM slot located on the motherboard of the server.

There is also a pin that secures it so if it is ever removed you will know it has been tampered with.

Once the chip is inserted some new security options will appear in the server BIOS to configure the TPM chip as shown below.

Once I received the chip and put it in the server I turned to the vSphere documentation to set it up. The problem there was there was no documentation on how to do this despite it being advertised as a new vSphere security feature. The ESXi configuration guide had one little paragraph on TPM which didn’t tell how to set it up and use it:

This module is a hardware element that represents the core of trust for a platform and enables attestation of the boot process, as well as cryptographic key storage and protection. As part of the boot process, ESXi measures the VMkernel by the TPM, and changes to the VMkernel are logged from one boot to the next. Measurement values are propagated to vCenter Server, and can be retrieved by third-party agents using the vSphere API.

Frustrated I reached out to VMware to figure out how to use this feature, some of the information I was able to get is below:

• TPM is only supported with ESXi.
• You need a TCG compliant BIOS, TXT needs to be enabled from the BIOS. Once it is enabled, you need to enable use of tboot from the UI Advanced configuration option for the ESXi host (the host has to be added to VC to be able to do this).
• There are some logs in serial log which can be used to monitor TPM. A 3rd party VC API is provided to fetch the TPM PCRs. If TXT was successful, then VMkernel fingerprint is reported in PCR19 otherwise, if the host has TPM but TXT was not used, then it will show in PCR8, otherwise PCRs should be NULL.
• There might not be any production server platforms out there ‘today’ which can support TXT.

I never did find the “tboot” advanced parameter that was supposed to be enabled. I checked all through the VMkernel advanced settings and didn’t see anything that was even close. It seems like while TPM provides some additional great protection for the VMkernel it is not yet ready to be used. The building blocks are currently there in vSphere but none of the necessary support features to be able to use it effectively exist yet. For example there is no way to monitor the feature so even if you could enable it there would be not much value to it. I expect both 3rd party vendors and VMware will develop the missing pieces in a future release (note the ESX & ESXi 4.1/4.5 version #’s in the videos) and look forward to being able to fully utilize this new security feature.

• Introduction to the TPM 1.2
• Trusted Computing Group. TPM Specification version 1.2
• Thoughts about Trusted Computing - Joanna Rutkowska
• On ESX/ESXi 4.0, A TPM-related warning is issued even though TPM is unavailable on the system (KB article)
• Data security in HP ProLiant servers using the Trusted Platform Module and MS Windows BitLocker Drive Encryption
• Trusted Computing Platforms (YouTube video)
• IDF 2009: “Intel Trusted Execution Technology — A Building Block for Enterprise Security (YouTube video)
• Trusted Execution Technology (TXT) with VMware and HyTryst Demo (YouTube video)
• VMworld 2009: ” Intel Server Security Technologies” (YouTube video)

I see it frequently on Twitter, here’s an example:

• Vendor A - Our product does this and supports this, Vendor B’s does not
• Vendor B - Oh yeah well our product does this and this and your product doesn’t
• Vendor A - Your product is way behind ours, you have a lot of catching up to do
• Vendor B - Then why are so many people using our product, our product is the best

If you’re on Twitter you’ve probably experienced this first hand, you also see it on blogs. Vendors that are rivals frequently bait each other by making claims that there product is better in one way or another. Of course the other vendor can’t resist this challenge and takes the bait and you have a back and forth debate that usually gets nasty and that all of us must experience while each vendor tries to throw punches and claim victory. This kind of non-professional banter has no place in a professional business. Everyone is vulnerable to this, I’ve even seen C-level executives get caught up in this. You may think that you are only defending your company and products when you respond to this kind of stuff but to everyone else that is watching you go back and forth it seems very childish and benefits neither vendor. Because of this kind of behavior you almost have to question who to trust and who to believe, vendor A or vendor B. Some people may choose to trust neither vendor because of the mudslinging and go with vendor C instead.

If you really want to sell your products and impress people try taking the higher ground when challenged by another vendor. Yes we know you are proud of your products as you should be and believe them to be the best but arguing with and insulting other vendors in public forums is not the way to prove it. Instead use your websites or blogs to inform those who are interested why your product is good and why they should buy it. There is a damn big pool of customers out there and plenty of room for multiple vendors to thrive. I do not work for a vendor and I can tell you from my personal experience that seeing that kind of immature behavior makes me not want to deal with that vendor regardless of how good their products are. On the other hand I have a great deal of respect for vendors that take the higher ground and act in a classy manner.

So vendors, knock it off, please act professional and people will notice and more importantly will listen to you. You may not like your competitors but at least respect them, they’re trying to make a living just like you are. Vendors that have mutual respect for their rivals get real high marks in my book and that type of friendly, non-confrontational  interaction between vendors is productive and very beneficial to all. Now I’m not saying you shouldn’t debate your points on social media, I like a good debate and can learn from it as long you keep it civil and professional. Just try and remember that we’re all professionals and try not to get caught up in the heat of the moment and lash out in a insulting or derogative manner at someone who challenges you. The people reading and following you will take notice and you will score higher marks than getting caught up in a Twitter piss-match. Social media is a wonderful tool when used correctly, when used incorrectly though it can really hurt your business. I really enjoy the great information that I gather on social media, so lets keep it civil so everyone enjoys what you have to say.

Just a reminder about the upcoming Denver VMUG this week on 3/4. It’s at a new location for us at the Vitamin Cottage corporate offices in Lakewood. Scott Herold from Quest Software will be doing a presentation on using PowerShell with VESI. I’ll also be doing a presentation on some of the new things in vSphere that you probably don’t know much about.  Since we’re lucky to have one of the VMware worldwide support centers nearby in Broomfield, CO they will again be joining us and passing out great tips and information.

You can register at this link and here’s the agenda:

11:00 Start meeting/lunch served
11:30 Welcome
11:45 What’s new at VMware (if needed, Gavin/Jarod)
12:00 “Optimize Day-To-Day Administration using Windows PowerShell and Virtualization EcoShell” Scott Herold, Quest Software
12:45 Break
1:00 What’s new in vSphere (Eric Siebert)
1:45  VMware Support (recent KBs, recent common calls, Q&A)
2:30 Meeting Adjourned

Update: just checked and it’s full already, check back as we may open it up some more.

VMware publishes a list of all server hardware that is supported with vSphere which includes servers, I/O adapters and storage and SAN devices. This list is continually updated and is most commonly referred to as the Hardware Compatibility List (HCL),  VMware changed the name for it a while back to the VMware Compatibility Guide as it is now referred to. The guide used to be published as PDF files only that you could read through to see if your hardware was listed but is now available as an online interactive webpage that is searchable and filterable as well. The guide lists hardware that is supported by vSphere but if hardware is not listed it does not mean it will not work with vSphere. In many cases the hardware will still work but because it is not listed VMware may not provide you support if you have problems with it. Servers and storage devices are two areas that are very common where they work with vSphere despite not being listed in the guide. I/O devices like network adapters and storage controllers though are less likely to work if they are not listed because they rely on drivers loaded in the VMkernel to work properly. If the driver is not one of the limited ones load in the VMkernel than your device is not going to work.

I don’t go through the guide that often as I assume the new mainstream hardware from large vendors like HP will always work and be supported. However last week that assumption bit me. We had just received some new hardware from HP which included a DL385 G6 server and an MSA G2 iSCSI storage array. I wanted to use hardware iSCSI initiators with TCP/IP Offload Engines (TOE) due to high CPU and I/O demands of the applications running on that server so I initially ordered HP’s only TOE card that was available which was the NC380T, however after we placed the order we were informed that the card was no longer available and was replaced by the newer NC382T. After receiving everything and assembling it and loading ESX on it I found that the NC382T was not listed under Storage Adapters in the vSphere Client as TOE cards should be as they are not treated as network adapters in vSphere. Only my SATA adapter, P410 smart array controller for local storage, Fibre Channel adapters and iSCSI software iniator were showing up under Storage Adapters.

The NC382T was showing up under Network Adapters instead and could not be used as a hardware initiator.

So this new TOE card from HP that we bought to use as a hardware initiator could not be used for that purpose. After discovering this I checked VMware’s hardware guide to see what HP iSCSI adapters were listed. Much to my surprise there was only one HP model listed and after looking up that adapter on HP’s website I found that it was for blade systems only and was therefore no use to my server.

Now HP OEM’s many of their storage adapters as many of  them are made by QLogic and Emulex. Most of the other vendors listed in the guide for iSCSI adapters had many QLogic adapters re-branded under their name but not HP.

After confirming with HP that the blade adapter was the only one that they OEM’d I was forced to get the QLogic branded adapter instead. The card that seemed to be the most popular was the QLE4062C adapter, so now we have one of those on order and may end up just using the NC382T as a regular NIC instead.

So the moral of this story is always check the Compatibility Guide especially when ordering a I/O adapters so you don’t end up with hardware that you can’t use with vSphere. If you want to know more about the guide such as how hardware gets added/removed from it and VMware’s support policy check out this blog post that I did a while ago on it.

I received a copy of Train Signal’s new vSphere Pro Series Vol. 1 which includes 3 modules on VMware View/ThinApp, Cisco Nexus 1000V and PowerCLI. The new Pro series is aimed towards users that want more than the basics and want to learn more about some of the advanced features in vSphere. I particularly like this 1st volume because it covers 3 areas I’m not very strong in and would like to learn more about. The instructor on the first module on View & ThinApp is David Davis, Rick Scherer is the instructor on the Cisco Nexus 1000V module and Hal Rottenberg is the instructor for PowerCLI. All three are experienced and knowledgeable vSphere veterans and very qualified to teach each subject. So head on over to Train Signal’s website and check it out, it’s regularly $599 but is available for a limited time for half-price ($297), you can also check out a demo of it here.

Thought I would share a blog that I’ve been reading now for quite some time that has some great information in it. I found it by accident while searching for information once, it’s VMware’s Technical Account Manager (TAM) blog were they generate newsletters roughly every week with lots of information for VMware’s many TAM’s. If you’re a blogger it’s not unusual to find links to some of your posts in these newsletters as VMware uses them to help keep their TAM’s informed with all the latest tips and tricks. So head on over there and give it a read.

Just a reminder that VMware will remove older versions of VI3 from their download site and only allow you to download the very latest versions beginning in May. So if you don’t already have them downloaded, get them while you can. I have links to all the download locations for the older versions here.  Sometimes its nice to have older versions to play around with.

Also VI3 will switch to Extended support in May of this year, just shy of it’s 4 year anniversary, here’s VMware’s definition of it:

New hardware platforms are no longer supported, new guest OS updates may or may not be applied, and bug fixes are limited to critical issues. Critical bugs are deviations from specified product functionality that cause data corruption, data loss, system crash, or significant customer application down time and there is no work-around that can be implemented.

VI3 has been rock solid since its release and the switch to extended support is no big deal for those still using it. Extended support lasts for another 3 years, you can read the details here.

Here’s the official announcement:

Announcement: End of availability for VI

Effective May 2010, VMware will remove all but the most recent versions of our Virtual Infrastructure product binaries from our download web site. As of this date, these Virtual Infrastructure products will have reached end of general support according to the published support policy. The downloadable products removed will include both ESX and Virtual Center releases. For reference, the VMware VI3 support policy can be viewed at this location:  www.vmware.com/support/policies/lifecycle/vi.

By removing older releases, VMware is establishing a long-term sustainable product maintenance line for older ESX product releases which have transitioned into the Extended Support life cycle phase. This enables us to baseline all patches and critical fixes against these baselines. This translates to faster customer turn-around and greater product stability during the extended support phase.

Virtual Infrastructure products being removed by May 2010:

• ESX 3.5 versions 3.5 GA, Update 1, Update 2, Update 3 and Update 4
• ESX 3.0 versions 3.0 GA, 3.0.1, 3.0.2 and 3.0.3
• ESX 2.x versions 2.5.0 GA, 2.5.1, 2.5.2, 2.1.3, 2.5.3, 2.0.2, 2.1.2 and 2.5.4
• Virtual Center 2.5 GA, 2.5 Update 1, 2.5 Update 2, 2.5 Update 3, 2.5 Update 4 and 2.5 Update 5
• Virtual Center 2.0

Virtual Infrastructure products remaining for Extended Support:

These versions will be the baseline for ongoing support during the Extended Support phase. All subsequent patches issued will be based solely upon the releases below.

• ESX 3.5 Update 5 will remain throughout the duration of Extended Support
• ESX 3.0.3 Update 1 will remain throughout the duration of Extended Support
• Virtual Center 2.5 Update 6 expected in early 2010

Customers may stay at a prior version, however VMware’s patch release program during Extended Support will be continued with the condition that all subsequent patches will be based on the latest baseline. In some cases where there are release dependencies, prior update content may be included with patches.

http://www.vmware.com/support/policies/lifecycle/vi/faq.html

In the last week I’ve gotten some new storage devices, both at work and at home. Unfortunately I’ve experienced problems with both and its not been as fun of a week as I would of liked. The new work storage device is a HP MSA-2312i which is the iSCSI version of their Modular Storage Array line.

The new home storage device is an Iomega ix4-200d 4TB which is a relatively low cost network storage device that supports iSCSI & NFS and much more.

The MSA problems have all been firmware related, basically it kept getting stuck in a firmware upgrade loop, if you own one or plan on buying one I would say don’t upgrade the firmware unless you have a reason to and if you do make sure you schedule downtime. I’ll be sharing some tips for upgrading the firmware on that unit later on.

The Iomega problems are from a flaky hard drive presumably, not long after I plugged the unit in and started configuring it I received the message that drive 3 was missing. After talking to support and rebuilding the RAID group the problem briefly went away and then came right back. They graciously waived the $25 replacement fee (it’s brand new, they better!) and refused to expedite the shipping unless I paid $40 (again, its brand new, you would think they would want to make a new customer happy). Having a flaky drive in a brand new unit doesn’t exactly inspire confidence in storing critical data on the device so I’ll have to see how it goes once the drive is replaced.

So look forward to some upcoming posts on using and configuring both devices. The MSA will be used as part of a Domino virtualization project and I’ll be doing performance testing on it in various configurations. The Iomega I’ll be using with VMware Workstation 7 on my home computer as both iSCSI & NFS datastores.

I started doing Lotus Domino server administration back in 1994 on version 3.0, about 5 years ago the company I work for decided to outsource it to a hosting provider. While it was outsourced I was still pretty involved with the management and administration of the environment. We recently decided to bring it back in-house due to cost-savings and because we have a DR site that we can leverage for business continuity. I was the project lead for transitioning the environment to the provider and I am again the project lead for bringing it back. That means I have to architect our environment, build it, plan and perform the migration and make sure everything works smoothly. Since we virtualized most of our environment several years ago it’s only natural to also virtualize the new Domino environment. This series of blog posts will cover the many steps of the project and will focus on the virtualization aspect of it. The project will likely take about 3-4 months to complete and right now I’m at the very beginning stages. So I’ll begin with a recap of where the project is at right now.

Planning & architecting

This is the most importance phase and one you can’t afford to screw up, I chose to virtualize Domino so it was necessary to make sure I had a virtual environment that could handle the taxing resource demands that Domino servers require. Our environment is about 1,500 total users and somewhere between small and medium, we have many access methods including LAN, WAN, internet and both Notes Client, web-based, Blackberry and other smartphone users. I planned for 2 clustered servers at our corporate HQ, 1 server in the DMZ for Notes Client passthru, Lotus Traveler and SMTP, 1 Blackberry Enterprise Server and 1 server at our DR warm site. Web access to email from the public internet was a requirement so the challenge was trying to find a secure solution for that. I focused on 2 solutions for that, using a hardware reverse proxy device in the DMZ and using a server running reverse proxy software. I looked at hardware solutions from F5 Networks who make very nice hardware that is very customizable and integrates well with Domino, unfortunately we didn’t have the budget for it though. So the solution I chose was using IBM Websphere Secure Proxy Server running on a Windows server in the DMZ.

Next it was time to find hardware to use, I needed one new virtual host at corporate HQ, one at the DR site and more storage. We have a Hitachi AMS-500 SAN (2GB fiber) that we could add storage to as an option, but this was expensive and didn’t get us much storage for our budget. Instead I looked at lower cost iSCSI solutions, in particular the MSA line from HP, they are very affordable, easy to use and perform very well. My biggest concern though was making sure it had enough performance for the very intensive disk I/O demands that Domino had. So I did some research looking at IOPs numbers for the MSA and comparing them to Iometer tests that I ran inside a VM running on the AMS-500. I found that the MSA had equal or better performance compared to the AMS-500 so I chose that instead. We were able to obtain an iSCSI MSA with dual controllers, an extra shelf loaded with 450GB drives for a total of about 10TB raw storage and were able to stay in our budget. For servers we went with HP DL-385 G6 servers with AMD 6-core processors.

Now with Domino there are a couple of important hardware considerations that you should be aware of. First is Domino loves memory, it will use everything you give it as it does a lot of caching. So having an adequate amount of memory is a must. The next is with CPU’s, Domino  takes full advantage of the new AMD RVI and Intel EPT  technology in newer processors so having this is a must. In fact this can make a huge difference in performance compared to processors without it. Finally when it comes to virtualization using vSphere was a must, there have been numerous performance improvements in vSphere that are very beneficial to Domino. IBM has published some tech notes in the past about high CPU usage on virtualized Domino servers compared to physical ones with identical workloads. They point out that you should make sure you have suitable hardware and the proper architecture but also seem to place the blame on the overhead of the virtualization layer. They go as far as to not recommend virtualizing larger Domino workloads. Well the combination of newer CPU technologies and vSphere now make that possible and allow almost any Domino workload to be successfully virtualized. So to summarize:

• Only use CPUs with AMD RVI & Intel EPT
• Have plenty of memory, don’t plan on using overcommitment
• Make sure your storage can keep up with Domino
• Make sure you are using vSphere and not VI3

Next up is building the hardware including the MSA and vSphere servers and configuring the iSCSI storage and networking. Some additional bonuses with using vSphere is the many improvements to iSCSI including support for Jumbo Frames, faster initiators and support for multi-pathing. So stay tuned for part 2…

Having experienced several complete data center power outages I’ve learned some important lessons over the years when it comes to virtualizing your infrastructure. Today I experienced another power outage and my previous experiences ensured that I was ready for this one and I thought I would share some tips:

• DNS is the most critical component in your environment, for almost anything to work properly you need a DNS server up first. If you have all your DNS servers virtualized and they are on shared storage its going to be very difficult to bring them up because everything else that needs to come up first usually relies on DNS. Therefore you should make sure you have at least one DNS server on the local storage of one of your hosts, that way you can get it up early on and not have to wait for your shared storage to come up. If you want to go a step further keep one as a physical server also, so you can quickly and easily get DNS up right away.
• Active Directory, DHCP & any other authentication servers are also very important, having a workstation up and running is very handy so you can centrally connect to hosts and power things back on. If you’re using DHCP for your workstations and servers you again will want one up as soon as possible so you can get them on the network and get going. Not having an Active Directory server available can make Windows servers take a very long time to boot. So again, keep an AD & DHCP server on local storage or on a physical server so you can quickly get them up as soon as possible. AD, DNS & DHCP are critical to a Windows environment and without them available you’ll find that the rest of your environment is mostly useless.
• Know your ESX command line, if your vCenter Server and other workstations are not available you’ll need to start VM’s using the command line. Even if your DNS server is on a local VM you won’t be able to start it without the vSphere Client. Therefore you’ll have to log into the ESX console and manually start it, if you don’t know the command to do this that could be a problem. Keep a cheat sheet by your hosts with the basic commands that you’ll need like vmware-cmd to get things up and running using the console.
• Know your host IP addresses, if DNS is not up yet you won’t be able to connect to your hosts using putty or the vSphere client using their host names, you probably won’t know their IP address and without a DNS server can’t look them up. Therefore keep a list of your host IP addresses so you can use that to connect to them.
• Know how to re-scan storage, your hosts may come up before your shared storage, once your shared storage is up you’ll need to rescan from your hosts so they will see it and you can restart the VM’s on it. You can do this using the vSphere client by clicking on Configuration, Storage Adapters, selecting your HBA and clicking the Rescan button and then select search for new devices. You can also do this using the command line esxcfg-rescan utility.
• Make sure you know where your datacenter keys are if you use a electronic card scanner to open your doors. Most systems are placed in the datacenter and if the power goes out your doors are not going to work. There is nothing worse than running around trying to find keys in a crisis to get into the datacenter. And make sure you don’t keep the keys in datacenter or you’ll have to break down the door to get in. (thanks Tony DiMaggio for reminding me about this one)

Being prepared is critical in crisis situation to ensure you can react quickly to get things back up and running. Sometimes it takes a crisis to point out any shortcomings that you may have in your environment but thinking ahead can save you from big headaches later on.

I’ve been maintaining the vLaunchpad using Dreamweaver which has been difficult at times so I thought I would explore other options. I’ve resisted using Wordpress as I had for my main site because the page layout is kind of unique and not easily re-produced in Wordpress. Well after much theme hacking I think I have it ported over to Wordpress pretty well. I’ve done extensive CSS modifications to make sure I would fit as much information as possible in one area and not have to resort to lots of scrolling or multiple pages. It’s not done yet but it’s getting close, I still need to add all the blog links and do some other work to it. I’ve also moved it to the clouds, which is GoDaddy’s new hosted grid computing offering. So let me know what you think, I’m open to all your suggestions, tell me what you like, don’t like and want to see added to it. One thing you’ll notice is I added all of Hany Michael’s great vDiagrams to the sidebar for easy access to them.

So go check out the new version of the vLaunchpad and let me know what you think, just remember it’s not done yet and I have lots of changes to make to it. Thanks to the Netherlands gang (Duncan, Eric & Gabe) for their early feedback that helped me get the design right.