-=-=-=-=-=-=-=-=-=-=-=-=
Also check out these other great VMware websites:
-=-=-=-=-=-=-=-=-=-=-=-=
VM Blog
Xtravirt
RTFM Education
Virtual Tricks
VM Professional
Virtualization.info
Vmweekly

Jump To: Networking ESX Server Troubleshooting VirtualCenter Virtual Machine

Tip List:
Networking:
How does network traffic route on vSwitches?
How can I display what VLAN tags are configured on a physical switch that my ESX server connects to?
How can I set a static MAC address for my VM?
How can I recover from vmnic re-numbering on ESX servers?
What TCP/IP ports are used by ESX, VirtualCenter and the License Server?

ESX Server:
How can I specify an alternate location for VM snapshot files?
How do I create an account so I do not have to use root?
What are the ESX specific commands that can be used on the Service Console?
Why do my VMFS volumes have both a name and a long string of numbers and letters?
How do I create a NFS volume on a Windows Server to use for storing ISO files for my VM’s?
How do I manually install the VC management agent?
How can I find vmdk files that are orphaned from VM’s and old snapshot files?
How do I create a ISO mount point that points to a Windows Server for storing ISO files for my VM’s?
How can I login to my ESX server using Active Directory instead of a local Linux account
How can I display disk/directory information on the Service Console?
Why did my all my VM's shutdown and restart when I restarted the mgmt-vmware service on the ESX console?
How do I change the time on the ESX Service Console?
How can I configure my ESX Server to automatically sync the time from a NTP source?
How can I reset a forgotten root password on my ESX Server?

Troubleshooting:
How do I troubleshoot ESX server issues?
How do I check my ESX & VirtualCenter versions?

VirtualCenter:
How can I modify the ESX/VC UI web page?
How do I register an existing Virtual Machine in VirtualCenter?
How does the HA (High Availability) feature work?
How can I purge old statistic data from the VirtualCenter database?
How can I purge old Event and Task data from VirtualCenter?
What data is stored in the VirtualCenter database?
How can I migrate my VirtualCenter database from MSDE to SQL server?
How can I move my VirtualCenter SQL database to another SQL server?
How can I find out information about my VirtualCenter SQL database?

Virtual Machine:
How can I migrate my Vmware Server VM’s to ESX?
How can I kill a stuck virtual machine?
How do I upgrade a existing virtual disk from BusLogic to LSI Logic?
How can I hide the Vmware tools icon in the system tray?
What are all the files that are located in my virtual machines directory on the ESX server for?
How can I limit the number of vmware.log files that are created for my virtual machine?
How can I disable Copy & Paste operations between the guest operating system and remote console?
What happens to virtual machines in case of a active path failure to my SAN?
What is the vswp file that is in my VM’s directory on the VMFS volume?
How can I disable my VM’s from page sharing?
Why does VirtualCenter show my VM’s Host Memory Usage higher then the amount of memory that is assigned to the VM?
Why does my VM show a high Guest Mem % when it first boots even though it is not using that much RAM?
How can I clear an active vmware.log file that is large without restarting the VM?
How can I hot clone a VM without using VMware Converter?
How can I disable serial and parallel ports so my VM does not see them?
How can I clone a VM without using VirtualCenter?
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Networking:

How does network traffic route on vSwitches?
Same vSwitch network traffic stays within the server and travels at the speed of the subsystem. Virtual Switches allow VM guest communication over the switch at maximum bus/processor capability which on current hardware technology should easily exceed 100Mbps.

Same vSwitch is 'Routed Locally' doesn't hit any pNICs

Same vSwitch different PortGroup/VLAN same thing is 'Routed Locally' doesn't hit any pNICs

Between different vSwitches is 'Routed Externally' ie needs to leave one pNIC and come in the other pNIC of the other vSwitch.

How can I display what VLAN tags are configured on a physical switch that my ESX server connects to?
You can see this in the VI client by clicking on the Configuration tab, and then Network Adapters. If you hover the mouse over the "Networks" field of any row in the table and your upstream switch is configured with VLAN tags you will see a whole list of VLAN tags that have been sniffed off of the network.

A better more accurate way to do this via the Service Console is with the esxcfg-info command. The Vlans and networks are listed under the network hint for each adapter. You can type esxcfg-info -n | grep -E -i "_name|Hint" to see a list of all of the network hints for each adapter.
 
If you want to display what is configured on the ESX vSwitch side you can type esxcfg-vswitch –l to display this.

How can I set a static MAC address for my VM?
This is useful for software that is licensed by the MAC address of the server it is running on or if you migrate a physical server to virtual and want to keep the same MAC address. A VM’s MAC address will change if the location of the VM changes, ie. different path on the same host. During a hot migration (Vmotion) the location of the VM does not change, this is also true of a VM that changes hosts due to HA/DRS migration. During a cold migration the location of the VM does change so the MAC address of the VM will change.

MAC addresses are hex values and consist of six groups of hex numbers. The first 3 octets of the MAC address is a unique code assigned to each NIC vendor, this is also called the Organizationally Unique Identifier (OUI), VMware’s OUI is “00:50:56”. The last 3 octets are assigned to all of the NIC’s for that vendor. Each MAC address must be unique to avoid conflicts with other network devices. You do not have to use the VMware assigned range (00:50:56) for your NIC’s. If you are coming from a physical server you can use the previous NIC’s range instead, (ie. HP/Compaq NIC’s use 00:08:02) as the first 3 octets. Just make sure the physical NIC and virtual NIC with the same MAC address are not active on the same physical network at the same time. There are several methods for setting a static MAC address on a VM which are listed below. If you use the first method you must stay inside VMware’s allowed MAC addresses (00:50:56:00:00:00 – 00:50:56:3F:FF:FF) or the VM will not start

Editing the VMX file of the VM:
• Edit the VMX file of the VM
• Change the following line from ethernetN.addressType="vpx" to ethernetN.addressType="static" (N is the number of your ethernet adapter, usually 0)
• Next change the line “ethernetN.GeneratedAddress” to “ethernetN.address” and then change the current MAC address to “00:50:56:XX:YY:ZZ” (again N is the number of your ethernet adapter and XX is a valid hex number between 00 and 3F, and YY and ZZ are valid hex numbers between 00 and FF. The value for XX must not be greater than 3F in order to avoid conflict with MAC addresses that are generated by the VMware Workstation and VMware GSX Server products.)
• Power your VM back on. Login to the OS, go to the CMD prompt and type “ipconfig /all”, your manually assigned MAC address should be listed for the NIC that you changed

Setting the NIC properties in Windows:
• Edit the Local Area Connection propeties for the NIC you want to change
• Click the Configure button next to the NIC name
• On the Advanced tab select NetworkAddress
• In the Value field enter a new value for the MAC address, enter only numbers or letters, no spaces, dashes or colons. This MAC address can be any valid hex numbers between 00 and FF for any of the octets
• Click OK and that’s it, the new MAC address takes effect immediately and will override any MAC address set by Vmware

How can I recover from vmnic re-numbering on ESX servers?
If you add or replace NIC’s in your ESX server you may run into a situation where ESX will re-number your NIC’s causing your already configured networking to no longer work. This can also happen if you make any other hardware changes on your ESX server like replacing a fiber channel card. Your servers system BIOS will sometimes change the physical topology of hardware devices, when changes are made it can subsequently change the order in which devices are discovered. When this happens ESX will see the NIC’s as new devices and assign them new NIC numbers. For example if you replace a 2-port NIC with a 4-port NIC, your original NIC’s were probably numbered vmnic0 and vmnic1. Once ESX sees the new NIC it will be numbered vmnic2, vmnic3, vmnic4 and vmnic5 because vmnic0 and vmnic1 already exist in the ESX configuration (even though they are no longer physically present). Because your existing vswitches and service console are linked to vmnic0 and vmnic1 they will no longer work. There are two ways to recover from this, my preference is method 1 which is generally easier then re-configuring your vswitches. Also method 1 preserves your existing NIC numbering if you don’t want your NIC’s starting at vmnic2.

Method 1 – Editing the esx.conf file Also see these kb articles for more info on this:
• Login to the Service Console
• Check your existing NIC numbering by typing ‘esxcfg-nics –l’
• Type ‘cd /etc/vmware’ to change to the correct directory
• Type ‘cp esx.conf esx.con.bak’ to make a backup of this file as it is a critical configuration file for ESX
• Type ‘nano esx.conf’ to open the file for editing
• Type CTRL-W and then enter ‘vmnic2’ to search for the new first NIC
• Change ‘vmnic2’ to ‘vmnic0’
• Change the subsequent NIC’s from ‘vmnic3’ to ‘vmnic1’, ‘vmnic4’ to ‘vmnic2’ and ‘vmnic5’ to ‘vmnic3’
• Type CTRL-O to save the file
• Type CTRL-X to exit the Nano editor
• Shutdown and restart the ESX server, when the server comes back up the NIC’s will be numbered vmnic0 – vmnic3, verify this by typing ‘esxcfg-nics –l’

Method 2 – Modify your vswitch configuration
• Login to the Service Console
• Check your existing NIC numbering by typing ‘esxcfg-nics –l’
• Check your current vswitch configuration by typing ‘esxcfg-vswitch –l’ , note which NIC’s are assigned to which vswitches (uplink column)
• Remove the old NIC’s that have been renamed by typing ‘esxcfg-vswitch –U <vmnic#> <vswitch name>’, ie. esxcfg-vswitch –U vmnic0 vSwitch1
• Add the new NIC’s with the correct names by typing ‘esxcfg-vswitch –L <vmnic#> <vswitch name>’, ie. esxcfg-vswitch –L vmnic2 vSwitch1
• Repeat this process for any additional NIC’s. Once you have the vswitch that contains the Service Console corrected you can also log in via the VI Client and correct t he other vswitches that way
• Your newly renamed NIC’s should now be assigned to the original vswitches and your networking should now work again

How To Configure Networking from the Service Console Command Line - http://kb.vmware.com/kb/1000258
VI Client loses connectivity to the ESX Server Host after you add a new network adapter - http://kb.vmware.com/kb/2243

What TCP/IP ports are used by ESX, VirtualCenter and the License Server?
The ports that are used are very well documented in the Server Configuration guide on pages 183 – 188. Here’s a summary of the core TCP/IP ports:
• Port 80 – HTTP access - The default non-secure TCP Web port typically used in conjunction with port 443 as a front end for access to ESX Server networks from the Web [Incoming TCP]
• Port 443 – HTTPS access - The default SSL Web port used for Connection to VI Web Access from the Web, VI Web Access and third-party network management client connections to the VirtualCenter Server and direct VI Web Access and third-party network management clients access to ESX Server hosts [Incoming TCP]
• Port 902 - Authentication traffic for the ESX Server host and virtual machine configuration - Used for VI Client access to the VirtualCenter Server, VirtualCenter Server access to ESX Server hosts, direct VI Client access to ESX Server hosts and ESX Server host access to other ESX Server hosts for migration and provisioning [Incoming TCP] [Outgoing UDP]
• Port 903 - Remote console traffic generated by user access to virtual machines on a specific ESX Server host - Used for VI Client access to virtual machine consoles and VI Web Access Client access to virtual machine consoles [Incoming TCP]
• Port 2049 - Transactions from your NFS storage devices - Used on the VMkernel interface rather than the service console interface [Incoming TCP] [Outgoing TCP]
• Ports 2050-5000 - Traffic between ESX Server hosts for VMware High Availability (HA) and EMC Autostart Manager [Outgoing TCP] [Incoming UDP] [Outgoing UDP]
• Port 3260 - Transactions from your iSCSI storage devices - Used on the VMkernel interface rather than the service console interface [Outgoing TCP]
• Port 8000 - Incoming requests from Vmotion - Used on the VMkernel interface rather than the service console interface [Incoming TCP] [Outgoing TCP]
• Port 8042-8045 - Traffic between ESX Server hosts for HA and EMC Autostart Manager [Outgoing TCP] [Incoming UDP] [Outgoing UDP]
• Port 27000 - License transactions from ESX Server to the license server - [Outgoing TCP]
• Port 27010 - License transactions from the license server - [Incoming TCP]
Additional optional ports:
• Port 22 – Used by ESX for SSH server access - [Incoming TCP]
• Port 123 – Used by ESX if you setup NTP to sync from a time source - [Outgoing UDP]

ESX Server:

How can I specify an alternate location for VM snapshot files?
To specify a new directory or datastore for snapshot files follow the below procedure. When you specify a new working directory for a VM the vswp file that is created when the VM is powered on is also stored in that location. If your VM is on shared storage and you specify local storage as a location you will not be able to use Vmotion/HA/DRS.
• Power off your VM
• Login to service console
• Edit the VMX file of your VM with Nano or Vi
• Add a new line using the following syntax: workingDir = "/vmfs/volumes/SnapVolume/Snapshots/"
• Optionally if you want your vswp file to stay in the VM’s directory add the following line to the VMX file: sched.swap.dir = "/vmfs/volumes/VM-Volume1/MyVM/" (You do not need to worry about updating the existing “sched.swap.derivedName” parameter, it is generated by the VM and written to the config file each time the VM powers on.)
• Power on your VM, your vswp, vmsn and snapshot (delta-vmdk) files will now be located in this directory

How do I create an account so I do not have to use root?
Xtravirt has published a great white paper on how to create an alternate user and use sudo to elevate it to root privileges. Check it out--> http://www.xtravirt.com/index.php?option=com_remository&Itemid=75&func=fileinfo&id=10

What are the ESX specific commands that can be used on the Service Console?
Below is a list of the commands, more detail for using these commands can be found in the Server Configuration Guide (http://www.vmware.com/pdf/vi3_server_config.pdf). In general (with a few exceptions) anything that can be done with these commands can also be done in the VI client, sometimes the situation requires doing these directly on the service console. Also Mike Laverick has put together a great guide on using a lot of these commands (http://www.rtfm-ed.eu/docs/vmwdocs/esx3.x-vc2.x-serviceconsole-guide.pdf)

Networking
Esxcfg-firewall – Configures the service console firewall ports
Esxcfg-nics – Prints a list of physical network adapters along with information on the driver, PCI device, and link state of each NIC. You can also use this command to control a physical network adapter’s speed and duplexing
Esxcfg-route – Sets or retrieves the default VMkernel gateway route
Esxcfg-vmknic - Creates and updates VMkernel TCP/IP settings for VMotion, NAS, and iSCSI
Esxcfg-vswif – Creates and updates service console network settings. This command is used if you cannot manage the ESX Server host through the VI Client because of network configuration issues
Esxcfg-vswitch – Creates and updates virtual machine (vswitch) network settings

Storage
Esxcfg-mpath - Configures multipath settings for your Fibre Channel or iSCSI disks
Esxcfg-nas – Manages NAS mounts. You use this command to add, delete, list, and change the attributes of NAS devices
Esxcfg-swiscsi – Configures your software iSCSI software adapter
Esxcfg-vmhbadevs – Prints a map of VMkernel storage devices to service console devices (no VI client equivalent)

General
Esxcfg-advcfg - Configures advanced options for ESX Server.
Esxcfg-auth – Configures authentication (no VI client equivalent)
Esxcfg-info - Prints information about the state of the service console, VMkernel, various subsystems in the virtual network, and storage resource hardware
Esxcfg-resgrp - Restores resource group settings and lets you perform basic resource group management
Esxcfg-upgrade - Upgrades ESX Server from ESX Server 2.x to ESX Server 3.x. This command is not for general use

Boot/diagnostic
Esxcfg-boot – Configures bootstrap settings (no VI client equivalent)
Esxcfg-dumppart - Configures a diagnostic partition or searches for existing diagnostic partitions
Esxcfg-init - Performs internal initialization routines (no VI client equivalent)
Esxcfg-linuxnet - Converts vswif to eth when booting ESX Server into service-console-only mode rather than into ESX mode (no VI client equivalent)
Esxcfg-module - Sets driver parameters and modifies which drivers are loaded during startup (no VI client equivalent)

How can I display disk/directory information on the Service Console?
Type “df –h” to show drives and space
Type “fdisk –l” to show physical drive information
Type “cat /proc/scsi/scsi” to show SCSI device information
Type “vdf –h” to show vmfs volumes and space
Type “ls –ltr” to show files and space

Why do my VMFS volumes have both a name and a long string of numbers and letters?
When you create a VMFS volume you are prompted for a name for it. This name is not what the Service Console uses to reference the volume, it is purely to make it easier for the user to identify the volume. The Service Console actually uses a unique identifier called a UUID to reference the volume. The name you specify when you create it is a user-defined device name which is a symbolic link to the UUID. This is done to solve the problem of changing the device name, when you change the name you are only changing the user-defined device name and not the UUID of the volume. So when you look in your /vmfs/volumes directory you will see both a UUID, ie. 4404e8b4-bcfd52fc-1e4b-0017a4a91076 and the symbolic link, ie. ServerA-Local. Clicking on the symbolic link simply takes you to the UUID directory, it is a lot easier to remember this then the long UUID of a volume.

For more on this see this link: http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-devlabel.html

How do I create a NFS volume on a Windows Server to use for storing ISO files for my VM’s?
You can do this if you have Windows 2003 Server R2 which has built-in NFS Services
• On the Windows 2003 Server make sure “Microsoft Services for NFS” in installed. If not you need to add it under Add/Remove Programs, Windows Components, Other Network File and Print Services
• Next go to folder you want to share and right-click on it and select Properties
• Click on the NFS Sharing tab and select “Share this Folder”
• Enter a Share Name, check “Anonymous Access” and make sure the UID and GID are both -2
• In VirtualCenter, select your ESX server and click the “Configuration” tab and then select “Storage”
• Click on “Add Storage” and select “Network File System” as the storage type
• Enter the Windows Server name, the folder (share) name and a descriptive Datastore Name
• Once it finishes the configuration you can now map your VM’s CD-ROM devices to this new VMFS volume

How do I manually install the VC management agent?
If after upgrading VirtualCenter you find some of your ESX hosts disconnected you can manually upgrade the management agent on the ESX server by following the below steps:
• First log into the ESX server console and check the version on the servers that are disconnected by typing “vpxa- v” The version needs to match the version of VirtualCenter being used. 
   - VC 2.0.1 build number is 32042
   - VC 2.0.1 build number is 32042
   - VC 2.0.1 Patch 1 build number is 33643
   - VC 2.0.1 Patch 2 build number is 40644
• Open the folder for the VC 2.0 installation. By default this will be "C:\Program Files\VMware\VMware VirtualCenter 2.0\upgrade"
• You need to use the correct file for different version of ESX server. You can find your answer in bundleversion.xml
   - 2.0.1+ = vpx-upgrade-esx-0-linux-*
   - 2.1.0+ = vpx-upgrade-esx-1-linux-*
   - 2.5.0 = vpx-upgrade-esx-2-linux-*
   - 2.5.1 = vpx-upgrade-esx-3-linux-*
   - 2.5.2 = vpx-upgrade-esx-4-linux-*
   - 2.5.3+ = vpx-upgrade-esx-5-linux-*
   - 3.0.0+ = vpx-upgrade-esx-6-linux-*
   - e.x.p = vpx-upgrade-esx-6-linux-*
• Copy file "vpx-upgrade-esx-y-linux-xxxxx" to your ESX host, where y and xxxxx are based on bundleversion.xml. xxxxx is the build number, ie. vpx-upgrade-esx-6-linux-40644. Use a secure copy utility such as WinSCP or PuTTY PSFTP to copy this file to the ESX server.
• Login to the ESX server as root.
• In the directory where you copied the upgrade bundle run the command: sh ./ vpx-upgrade-esx-y-linux-xxxxx (xxxxx is the build number)
• Once the install is complete run the command “service vmware-vpxa restart” followed by “service mgmt-vmware restart”
• Check the version again by typing “vpxa –v”, the version should now be the new version. Now open your VI Client, try to connect to the ESX host.

How can I find vmdk files that are orphaned from VM’s and old snapshot files?
• Login to service console
• Change to your /vmfs/volumes/ directory
• Type find -iname "*-flat.vmdk" -mtime +7 -ls to find –flat vmdk files that have not been modified in 7 days
• Type find -iname "*-delta.vmdk" -mtime +7 -ls to find snapshot files that have not been modified in 7 days

How do I create a ISO mount point that points to a Windows Server for storing ISO files for my VM’s?
You can create a mount point inside /vmimages so your VM’s can access ISO files when needed. You could just create a directory on your VMFS volume and copy them there but this takes valuable disk space away from the ESX server. You can also use this technique to create other mount points on the ESX server if you need to copy files from a Windows server to or from it for patches and other things.
• First you will have to open the firewall port in ESX by typing in the service console “esxcfg-firewall -e smbClient”. This opens outbound TCP ports 137 – 139 and 445.
• Next make sure the Local Security Policy on the Windows Server you are going to map to has the following settings under Security options. If you do not do this you may get a “Stale NFS Handle” or “Permission denied” error when trying to mount.
   - Microsoft network client - digitally sign communications (always) – DISABLED
   - Microsoft network client - digitally sign communications (if server agrees) – DISABLED
   - Microsoft network server - digitally sign communications (always) – DISABLED
   - Microsoft network server - digitally sign communications (if server agrees) – DISABLED
• Next type “cd /vmimages” and then “mkdir ISO”
• Next type “mount -t smbfs -o username= //windows server/share /vmimages/ISO” you can also use a windows domain username by typing “username=,workgroup=”
• You will be prompted for a password for the user account you are using.
• You can type “cd /vmimages/ISO” and do “ls” and you should see all your files. You can now map your VM’s CD-ROM using the Datastore ISO file device type and select the /vmimages/ISO folder.
• If you no longer need this mount point you can un-mount it by typing “umount /vmimages/ISO”
• To automatically have your ESX server mount this when it restarts do the following:
   - Edit /etc/fstab using Nano or Vi
   - Add the following line to the bottom of the file: “//windows server/share /vmimages/ISO smbfs noauto,username=,password= 0 0”
   - Edit /etc/rc.local using Nano or Vi
   - Add the following line to the bottom of the file “mount –a”

How can I login to my ESX server using Active Directory instead of a local Linux account?
PAM (Pluggable Authentication Modules) authentication gives you the ability to use your Active Directory passwords to login to the ESX Service Console. You still need to create a identical account on the ESX server but once you do that you can login using your AD password after PAM is configured. The advantage of this is password management, you do not have to manage ESX passwords separately and can leverage AD’s powerful account management features. Prior to setting this up you want to ensure that your ESX servers sync time from the same sources as your AD servers, if you use NTP on both systems then this should not be an issue. Kerberos is very sensitive to time variations and by default in AD environments  requires computer clocks to be synchronized within five minutes. If your ESX servers are not time synced with AD you will probably run into login issues (you will see clock skew too great errors in /var/log/messages). Below are the steps to set this up on the ESX server.

• Login to service console
• If you’re adding a user that does not exist in the Service Console you must first add it. Linux user accounts are stored in the etc/passwd file, you can open this file with Nano to see all the accounts that are created. Alternately you can load the VI client and connect directly to the ESX server and click the Users & Groups tab to see user accounts.
• To add a account type “useradd <username>” The username must match the samaccountname of the AD user. Alternately you can add the user using the VI client by clicking “Add” in the Users view.
• Next you need to enable PAM authentication in ESX, you use the esxcfg-auth command for this like below:
> esxcfg-auth -–enablead -–addomain=addomain.com -–addc=dcname.addomain.com -–krb5realm=addomain.com –-krb5kdc= dcname.addomain.com –-krb5adminserver= dcname.addomain.com
> “–-enablead” enables Active Directory authentication
> “—addomain” sets the Active Directory domain
> “--addc” sets the Active Directory domain controller, use this multiple times to add domain controllers for redundancy
> “—krb5realm” sets the Kerberos realm (AD domain)
> “—krb5kdc” sets the Kerberos Key Distribution Center (Domain Controller)
> “—krb5adminserver” sets the Kerberos Admin Server (Domain Controller)
• This takes effect immediately, log back in to the console and you should be able to use your AD password for that account. If you run into difficulty you can check the /var/log/messages file for errors. Also ensure that your ESX server can resolve the FQDN of the AD server you entered above.
• If you plan on logging in using the VI client directly to the ESX server you will first need to click on the Permissions tab in the VI Client and add a permission for the user.
• If you wish to disable this type “esxcfg-auth –-disablead”. Esxcfg-auth writes the configuration information to various files including /etc/krb5.conf

How can I display disk/directory information on the Service Console?
• Type “df –h” to show drives and space (no vmfs)
• Type “fdisk –l” to show physical drive information
• Type “cat /proc/scsi/scsi” to show SCSI device information
• Type “vdf –h” to show vmfs volumes and space
• Type “ls –ltr” to show files and sizes

Why did my all my VM's shutdown and restart when I restarted the mgmt-vmware service on the ESX console?
This behavior is caused by a nasty bug that was introduced in ESX 3.0.1. If you have Virtual Machine Auto Startups enabled on your ESX host and restart the mgmt-vmware service on the ESX service console it will cause all of your VM's on that host to shutdown and restart. This bug was fixed in the 5/15/07 ESX server patches. If you need to restart the mgmt-vmware service and do not have this patch you must disable Auto Startups first. Installing the patch will prevent this from happening.
Patch link: http://www.vmware.com/support/vi3/doc/esx-7557441-patch.html
KB Article: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=7557441
My post on Vmtn (I got bit by this bug): http://www.vmware.com/community/thread.jspa?threadID=75462&tstart=0

How do I change the time on the ESX Service Console?
• Login to service console
• Type “date –s yyyymmdd” to set the date
• Type “date –s hh:mm” to set the time

How can I configure my ESX Server to automatically sync the time from a NTP source?
• Login to service console
• Edit /etc/ntp.conf with Nano or Vi
• Add the below lines to the OUR TIMESERVERS section
     • restrict default kod nomodify notrap
     • server 0.pool.ntp.org
     • server 1.pool.ntp.org
     • server 2.pool.ntp.org
• Save & exit
• Edit /etc/ntp/step-tickers with Nano or Vi
• Add the below lines
     • 0.pool.ntp.org
     • 1.pool.ntp.org
     • 2.pool.ntp.org
     • pool.ntp.org
• Save & exit
• Type “esxcfg-firewall –enableService ntpClient” or use VI Client to enable under ConfigurationàSecurity ProfileàProperties
• Type “service ntpd restart” – may say failed for stop if not currently running
• Type “chkconfig --level 345 ntpd on” to enable NTP daemon to autostart
• Type “hwclock -–systohc” to set hardware clock to system clock
• Type “ntpdate –q 0.pool.ntp.org” to see the offset between local and ntp clock

There is also a script available to automatically set this up for you: http://www.vmcolonel.net/?p=13
Read more on NTP pool servers at the following links: http://en.wikipedia.org/wiki/NTP_pool and http://www.pool.ntp.org/
Read more on timekeeping in ESX at the following link: http://www.vmware.com/pdf/vmware_timekeeping.pdf

How can I reset a forgotten root password on my ESX Server?
• If you have forgotten the root password and have no way to change it rather then re-install ESX you can use the below procedure.
> Shutdown ESX and restart it at the first menu type ‘a’
> At the next prompt type ‘single’
> ESX will now go through the boot process and you will end up at a “#” prompt
> Type ‘passwd’, enter a new password for root and re-type it when prompted
> Type ‘reboot’ and you’re done

Troubleshooting:

How do I troubleshoot ESX server issues?
You can check several log files on the ESX server based on the problem you are experiencing, these include:
• Vmkernel - /var/log/vmkernel – records activities related to the virtual machines and ESX server
• Vmkernel Warnings - /var/log/vmkwarning – records activities with the virtual machines
• Vmkernel Summary - /var/log/vmksummary - Used to determine uptime and availability statistics for ESX Server; human-readable summary found in /var/log/vmksummary.txt
• ESX Server host agent log - /var/log/vmware/hostd.log - Contains information on the agent that manages and configures the ESX Server host and its virtual machines (Search the file date/time stamps to find the log file it is currently outputting to.)
• Service Console - /var/log/messages - Contain all general log messages used to troubleshoot virtual machines or ESX Server
• Web Access - /var/log/vmware/webAccess - Records information on Web-based access to ESX Server
• Authentication log - /var/log/secure - Contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.
• VirtualCenter agent - /var/log/vmware/vpx - Contains information on the agent that communicates with VirtualCenter
• Virtual Machines - The same directory as the affected virtual machine’s configuration files; named vmware.log - Contain information when a virtual machine crashes or ends abnormally

How do I check my ESX & VirtualCenter versions?
• ESX Server
>Login to service console
>Type “vmware –v” – ie. “VMware ESX Server 3.0.1 build-32039”
>Type “esxupdate –l query” to see what updates are installed.
• ESX Server management agent
>Login to service console
>Type “vpxa –v” – ie. “VMware VirtualCenter Agent Daemon 2.0.1 build-40644”
• VirtualCenter Server
>Check VC Server log file with the latest date in C:\WINDOWS\Temp\vpx – ie. “Log for VMware VirtualCenter, pid=4928, version=2.0.1, build=build-40644”
>Optionally local the VI Client on the VirtualCenter server and select Help, then About.
• License Server
>On the License Server, load the Vmware License Server Tools (LMTools) client and select Help, About – ie. 10.13
>Optionally, right-click on either lmgrd.exe or lmtools.exe which is usually located in C:\Program Files\Vmware\Vmware License Server and select properties, then click on the Version tab and select “File Version”
• Virtual Infrastructure Client
>Load client
>Select Help from the top menu and then About Virtual Infrastructure – ie. “Version 2.0.1 Build 40644” this should match your VC Server
• Vmware Tools (on a VM)
>Right-click on icon inside VM
>Click on About tab – ie. “build-32039” this should match your ESX server version
• Vmware Tools (on a ESX Server)
>Login to service console
>Type “rpm –qa | grep VMware-esx-tools” – ie. “VMware-esx-tools-3.0.1-32039”

VirtualCenter:

How can I modify the ESX/VC UI web page?
It is not recommended to do this extensively because future product upgrades will overwrite the web files. Some people wish to remove the download links on the page or force a redirect directly to the login page.
• To modify the login page on the ESX server edit the 'index.html' file from within /var/lib/vmware/hostd/docroot
• To modify the login page on the VirtualCenter Windows server edit the ‘index.html’ in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\docRoot
• To comment out the download links simply comment out the code. You must place your comment start and end tags in the right locations so the page works properly. Add and add --> after the second tag to comment out the two download links under getting started. Remember to always make a backup copy of the index.html before editing it.
• To set the main UI page to automatically re-direct to the login page then add the following line at the beginning of the index.html file o o If you do not want the main page to show before the redirect either delete all the other code in the file or add a at the very bottom of the file, this comments ouot all the code in the file except the first line.

How do I register an existing Virtual Machine in VirtualCenter?
If you have an existing VM that was removed from VirtualCenter and just need to add it back in then follow the below steps
• Select the ESX server in the VirtualCenter that the VM files are located on
• On the summary tab right-click on the datastore that the VM files are located on and click “Browse Datastore”
• Select the folder that the VM files are located in
• Right-click on the VMX file for the VM and select “Add to Inventory”
• Select a name and location for the VM and click Next
• Select the Host or Cluster for the VM and click Next
• Select a Resource Pool and click Next
• Click Finish and the VM will now appear in VirtualCenter

How does the HA (High Availability) feature work?
VMware HA continuously monitors all ESX Server hosts in a cluster and detects failures. An agent placed on each host maintains a "heartbeat" with the other hosts in the cluster and loss of a heartbeat initiates the process of restarting all affected virtual machines on other hosts. You create and manage clusters using VirtualCenter. The VirtualCenter Management Server places an agent on each host in the cluster so each host can communicate with other hosts to maintain state information and know what to do in case of another host's failure. (The VirtualCenter Management Server does not provide a single point of failure.) If the VirtualCenter Management Server host goes down, HA functionality changes as follows. HA clusters can still restart virtual machines on other hosts in case of failure; however, the information about what extra resources are available will be based on the state of the cluster before the VirtualCenter Management Server went down. HA monitors whether sufficient resources are available in the cluster at all times in order to be able to restart virtual machines on different physical host machines in the event of host failure. Safe restart of virtual machines is made possible by the locking technology in the ESX Server storage stack, which allows multiple ESX Servers to have access to the same virtual machines file simultaneously.

Host failure detection occurs 15 seconds after the HA service on a host has stopped sending heartbeats to the other hosts in the cluster. A host stops sending heartbeats if it is isolated from the network. At that time, other hosts in the cluster treat this host as failed, while this host declares itself as isolated from the network. By default, the isolated host powers off its virtual machines. These virtual machines can then successfully fail over to other hosts in the cluster. If the isolated host has SAN access, it retains the disk lock on the virtual machine files, and attempts to fail over the virtual machine to another host fails. The virtual machine continues to run on the isolated host. VMFS disk locking prevents simultaneous write operations to the virtual machine disk files and potential corruption.

If the network connection is restored before 12 seconds have elapsed, other hosts in the cluster will not treat this as a host failure. In addition, the host with the transient network connection problem does not declare itself isolated from the network and continues running. In the window between 12 and 14 seconds, the clustering service on the isolated host declares itself as isolated and starts powering off virtual machines with default isolation response settings. If the network connection is restored during that time, the virtual machine that had been powered off is not restarted on other hosts because the HA services on the other hosts do not consider this host as failed yet. As a result, if the network connection is restored in this window between 12 and 14 seconds after the host has lost connectivity, the virtual machines are powered off but not failed over.

For more information on HA see http://download3.vmware.com/vmworld/2006/tac9413.pdf and http://kb.vmware.com/KanisaPlatform/Publishing/894/2956923_f.SAL_Public.html and http://www.vmware.com/pdf/vmware_ha_wp.pdf

How can I purge old statistic data from the VirtualCenter database?
The size of the Statistics database (VPX_HIST_STAT) will vary based on the number of hosts & VM’s managed, frequency of performance data collection, collection level and type of database. Each stat sample collected is about 60 bytes for SQL, 100 bytes for Oracle, and each event stored is 1600 bytes for SQL, 600 bytes for Oracle. Using default settings, the statistical data for 25 hosts running 8-16 VMs per host will plateau around 40-60 MB in a year (80-140 MB if set to “full”). Each month, the average number of events generated will also consume about 190 MB in SQL, and 70 MB in Oracle. Total DB size after a year is expected to be around 2.20 Gb in SQL, and 1.0 Gb in Oracle. Using default settings, the statistical data for 75 hosts running 8-16 VMs per host will plateau around 90-150 MB in a year (200-330 MB if set to “full”). Each month, the average number of events generated will also consume about 190 MB in SQL, and 70 MB in Oracle. Total DB size after a year is expected to be around 2.40 Gb in SQL, and 1.2 Gb in Oracle.

The statistic sample frequency and retention is specified in the VI client. Select Administration, VirtualCenter Management Server Configuration from the top menu then select Statistics. The default settings are:

• Past Day – 5 minutes per sample, 288 total samples
• Past Week – 15 minutes per sample, 672 total samples
• Past Month - 60 minutes per sample, 720 total samples
• Past Year – 1440 minutes per sample, 365 total samples

For example if you wanted to collect less statistical data you could change the settings to something like below:

• Past Day – 15 minutes per sample, 96 total samples
• Past Week – 60 minutes per sample, 168 total samples
• Past Month - 120 minutes per sample, 360 total samples
• Past Year – 1440 minutes per sample, 365 total samples

Or if you wanted to collect more statistical data you could change the settings to something like below (caution your database will be very large):

• Past Day – 1 minutes per sample, 1440 total samples
• Past Week – 5 minutes per sample, 2016 total samples
• Past Month - 30 minutes per sample, 1440 total samples
• Past Year – 60 minutes per sample, 8760 total samples

You can also delete any ones you do not want, for example if you only want a weeks worth of data then delete the Past Month and Past Year. Once the total samples number has been reached the older samples are automatically purged from the database. If you modify these values you must follow these rules:

• The new sampling period must be a multiple of a previous one. In the defaults you’ll notice that week (15) is a multiple of day (5), month (60) is a multiple of week(15) and year (1440) is a multiple of month (60).
• The new sampling length must be longer than a previous one. Week (15) is longer then day (5), etc.
• The user-specified name of the historical interval must be unique. (In case you add new ones)

Note: Existing data is reset (lost) when you change the interval configuration. However, only the data for that interval is reset. For example, if you change only the weekly time interval, the daily and monthly data are retained.

You can also specify the amount of statistical detail that is captured by setting the Statistic Collection Level. The default for this setting is Level 1 which includes basic metrics, it can be changed anywhere up to Level 4 which provides the most statistical detail but can significantly increase the size of the database and can cause additional performance overhead. You can also change the number of threads that are used for collecting performance statistics from managed hosts. Use multiple threads to increase the number of managed hosts and take advantage of the additional processor.

How can I purge old Event and Task data from VirtualCenter?
VirtualCenter does not support doing this directly, the only way you can do this by deleting directly from the database tables. It is strongly recommend to have a functional backup of the VC database before clearing these tables in case of an unforeseen problem. Also the the VC service must be stopped in order to avoid data corruption. This is not a supported procedure by Vmware but should be in my opinion, these tables can get pretty large in big environments as every single task and event is preserved in the database.

The tables where the data is stored are VPX_TASK (Tasks) and VPX_EVENT, VPX_EVENT_ARG (Events). The VPX_EVENT_ARG will usually have more rows then the VPX_EVENT table. This is because the VPX_EVENT table only has one row per event but the VPX_EVENT_ARG table can have multiple rows per event (The link between the 2 tables is the EVENT_ID column).

• Shutdown the VirtualCenter service
• Connect to the database server that is hosting the VC database with a SQL browser/client. You can use a free tool like WinSQL Lite (http://www.snapfiles.com/get/winsql.html) for this.
• To delete all data in the tables type: ‘delete from VPX_TASK’ then ‘delete from VPX_EVENT’ and then ‘delete from VPX_EVENT_ARG’ and finally ‘Commit’
• Optionally you can do ‘truncate table VPX_TASK’ then ‘truncate table VPX_EVENT’ and then ‘truncate table VPX_EVENT_ARG’ (Truncate is faster and does not use as much undo space as delete)
• To selectively delete data older then 30 days: ‘delete from VPX_TASK where complete_time < sysdate - 30)
• To selectively delete data older then 30 days from VPX_EVENT and VPX_EVENT_ARG is trickier because VPX_EVENT_ARG does not have a date field and it tied to the events in VPX_EVENT by the Event_id field.
• First delete from the child table (VPX_EVENT_ARG): ‘delete from vpx_event_arg where event_id in (select a.event_id from vpx_event_arg a, vpx_event b where a.event_id = b.event_id and create_time < sysdate - 30)
• Next commit the delete by typing ‘Commit’
• Then delete from the parent table (VPX_EVENT): ‘delete from vpx_event where create_time < sysdate – 30’
• Finally commit again by typing ‘Commit’
• Start the VirtualCenter service

What data is stored in the VirtualCenter database?
As of version 2.0.1 of VirtualCenter the database consists of the following tables. The database mainly consists of alarm/event data, HA/DRS data, ESX host information, task/scheduled tasks and VM information. All ESX server and VM configuration data is stored on each ESX server and is simply read and displayed by VC. VC is just a central management front end to set configuration information and displays information that is read from all ESX hosts instead of having to manage each server individually. You can also use the VIC to connect directly to the ESX servers without VirtualCenter and modify the same configuration data. Once you add a ESX host back into VC it reads all the configuration info from that host.

The database is not critical to the operation of ESX servers or their virtual machines, they would continue to function normally if VC or it’s database were unavailable (Except for DRS and vMotion which would not work, HA would still work). If the database were to crash and a new one created you could add your ESX servers back in and it would repopulate the configuration information. The only data unique to the database is performance statistics, alarms, events, tasks, resource pools and custom attributes. This is not official documentation and is based on my browsing the database schema and data with a SQL client. Using a SQL client like Toad (Quest Software) or WinSQL Lite (Free) you can browse the data contained in these tables, you can also write SQL code to query information contained in the tables. Below is some sample SQL code to query information on VM’s and hosts. For information on the views defined in VirtualCenter see this white paper: http://www.vmware.com/pdf/vc_dbviews_2x.pdf

How can I migrate my VirtualCenter database from MSDE to SQL server?
Note: This has only been tested on SQL 2000 but should work OK on SQL 2005
• Shutdown the VirtualCenter Server so it disconnects from the SQL database.
• At a CMD prompt type “osql –E –D <ODBC datasource name>” ie. “osql –E –D ‘VMware VirtualCenter’”  This will load the osql command line SQL client for MSDE, login using the local trusted connection and connect using the VC ODBC data source.
• At the 1> prompt type the following, “use master” closes the connection to the vcdb so it can be detached, go executes the previous SQL statements.
   • Use master
   • Exec sp_detach_db ‘vcdb’
   • Go
• Copy the two database files (vcdb.mdf and vcdb.ldf) to the destination SQL server data directory.
• Load SQL Enterprise Manager on the destination database server.
• Select Security and then right click on Logins and select “New Login”. In this example we are using vpxadmin as the SQL user account.
• On the General tab specify the username, if using SQL authentication make sure the password also matches and click OK.
• Right click on “Databases” and select “All Tasks” and then “Attach Database”
• Browse to the directory that you copied the database and log files to and select the MDF file for VC, select vpxadmin as the owner and click OK.
• Select Security and Logins, right-click on the vpxadmin user and select “Properties”.
• Change the Default Database from “Master” to “VCDB”
• On the VC Server, edit the existing ODBC connection for VC under the System DSN tab. Change the Server from MSDE to the appropriate SQL Server and click Next.
• Select the appropriate authentication method, if using SQL authentication enter the username/password that you created in the Security step above and click Next.
• Click Next again, the default database should already be VCDB, click Finish and that’s it.
• Click the “Test Datasource” button to verify that it works and then OK.
• On the VC server, run the VC installation program, click Next at the Welcome screen and then select Repair and click Next.
• Select “Use an existing database server” and click Next, enter the SQL db username and password and click Next.
• At the Overwrite Data message select NO.
• At the License Server screen select the appropriate option (usually use an existing license server) and click Next.
• Enter the appropriate license server address (usually 27000@localhost) and click Next.
• Enter the appropriate port information (usually the default is fine) and click Next.
• Enter the appropriate Apache web server port (usually 8086) and click Next then click the Install button.
• Start the VC server and that should be it. If the VC Server fails to start check the log files localed in C:\Windows\Temp\VPX for the reason why.

How can I move my VirtualCenter SQL database to another SQL server?
Note: This has only been tested on SQL 2000 but should work OK on SQL 2005.
• Shutdown the VirtualCenter Server so it disconnects from the SQL database.
• Load SQL Enterprise Manager on the source database server.
• Right click on the VirtualCenter database and select “Detach Database”
• Click “OK” to detach the database.
• Copy the VC database (MDF) and log file (LDF) from the SQL Data directory on the source server to the SQL Data directory on the new SQL server.
• Load SQL Enterprise Manager on the destination database server.
• Select Security and then right click on Logins and select “New Login”. In this example we are using vpxadmin as the SQL user account.
• Specify the same SQL username as the previous SQL server, if using SQL authentication make sure the password also matches. Accept the defaults and click OK.
• Right click on “Databases” and select “All Tasks” and then “Attach Database”
• Browse to the directory that you copied the database and log files to and select the MDF file for VC, leave SA as the owner and click OK.
• If you are using a local SQL or Windows account for VC you will have to fix the orphaned usersid on the destination server, you can use the below procedure or download the Sync SQL Logins utility to do this.
   • Load Query Analyzer, login as the SA account.
   • Verify the orphaned user (vpxadmin) by executing the below SQL, it should return one orphaned username and usersid.
      • use virtualcenter
      • exec sp_change_users_login 'report'
   • Fix the orphaned user by executing the below SQL, this will link the old vpxadmin account to the new and correct the SID.
      • use virtualcenter
      • exec sp_change_users_login 'update_one', 'vpxadmin', 'vpxadmin'
   • Verify the orphaned user (vpxadmin) has been fixed by executing the below SQL, it should nothing.
      • use virtualcenter
      • exec sp_change_users_login 'report'
• On the VC Server open the ODBC connection for VirtualCenter and click Configure
• Select or type in your new SQL Server name and click Next
• Select your authentication method and enter the username/password and click Next.
• Select Change the Default Database To and choose the VC database and click Next.
• Click Finish and then click the Test DataSource button to verify connectivity.
• Start the VirtualCenter server and you should be all set. If the service will not start check the VC log files in the c:\windows\temp\vpx directory for any errors.
• Reference these Microsoft kb articles for more information:
   • Moving DB's between Servers - http://www.support.microsoft.com/?id=314546
   • How to Resolve Permission Issues When a Database Is Moved Between SQL Servers - http://www.support.microsoft.com/?id=240872
   • How To Transfer Logins and Passwords Between SQL Servers - http://www.support.microsoft.com/?id=246133
   • Troubleshooting Orphaned Users - http://support.microsoft.com/kb/274188/
   • User Logons and Permissions Incorrect after a Database Restore - http://support.microsoft.com/?id=168001
   • Sync SQL Logins utility - http://www.dbmaint.com/SyncSqlLogins.asp
   • MapSids Utility - http://support.microsoft.com/?id=298897

How can I find out information about my VirtualCenter SQL database?
You can use the sp_helpfile stored procedure to get general information on your VC SQL database.
• Load the SQL Query Analyzer on the SQL Server and login with your virtualcenter database credentials.
• Type the following in the query window
   • Use <database name>  ie. virtualcenter
   • Sp_helpfile
• Press F5 or the Execute Query icon.
• The results window will return the database file names (ie. virtualcenter_Data.mdf, virtualcenter_Log.mdf), the current size, maximum size, automatic growth percentage and usage.

To find more detail on individual tables in the VC database you can use the sp_spaceused stored procedure.
• Load the SQL Query Analyzer on the SQL Server and login with your virtualcenter database credentials
• Type the following in the query window
   • Use <database name>  ie. virtualcenter
   •EXEC Sp_spaceused ‘table name’  ie. vpx_hist_stat
• Press F5 or the Execute Query icon.
• The results window will return the table name, number of rows (records), total amount of space reserved for the table, total amount of space used by the data, total amount of space used by indexes and the total amount of space reserved but not yet used.


Virtual Machine:

How can I migrate my Vmware Server VM’s to ESX?
There are several methods for doing this that are listed below:

Method 1 – Vmware Converter running directly on Vmware Server (Cold clone)
• Install Vmware Converter on to the Vmware Server and reboot if prompted
• Shutdown the VM to be converted
• Run Converter and select Import Machine
• Click Next at the Welcome screen and Next again at the Source screen
• Select “Standalone virtual machine, backup or disk image” as your source and click Next
• Browse to your VM’s vmx file and click Next
• Select either “Import all disks and maintain size” or “Select volumes and resize to save or add space”, select your volumes and enter a new disk size if necessary and click Next
• Click Next at the Destination screen
• Select “VMware ESX Server or VirtualCenter virtual machine” as your destination and click Next
• At the destination login screen enter your VC/ESX server name and login credentials and click Next
• Enter a Virtual Machine name and folder and click Next
• Select a Host/Cluster and click Next then select a Datastore and click Next
• Enter your NIC information and click Next then click Next again at the customization screen
• Click Finish when it completes
• Edit your new VM’s settings and remove any extra hardware if not needed, ie. USB devices, serial and parallel devices, Floppy drive and change the SCSI adapter to LSI Logic if needed
• Power on your new VM and uninstall VMware tools (VMware server version)
• Restart server
• Remove old virtual machine hardware
    o Open CMD prompt and type “SET DEVMGR_SHOW_NONPRESENT_DEVICES=1” and then “DEVMGMT.MSC”
    o Select “Show Hidden Devices” from top menu
    o Remove old greyed-out non-present hardware, ie. old processor, disk drives, IDE/SCSI controllers, storage volumes, etc.
• Reboot and install VMware tools (ESX version) and you are finished

Method 2- Vmware Converter running directly on Virtual Machine (Hot clone)
• Install Vmware Converter on to the Virtual Machine and reboot if prompted
• Run Converter and select Import Machine
• Click Next at the Welcome screen and Next again at the Source screen
• Select “Physical Computer” as your source and click Next
• Select “This local machine” and click Next
• Select either “Import all disks and maintain size” or “Select volumes and resize to save or add space”, select your volumes and enter a new disk size if necessary and click Next
• Click Next at the Destination screen
• Select “VMware ESX Server or VirtualCenter virtual machine” as your destination and click Next
• At the destination login screen enter your VC/ESX server name and login credentials and click Next • Enter a Virtual Machine name and folder and click Next
• Select a Host/Cluster and click Next then select a Datastore and click Next
• Enter your NIC information and click Next then click Next again at the customization screen
• Click Finish when it completes
• Edit your new VM’s settings and remove any extra hardware if not needed, ie. USB devices, serial and parallel devices, Floppy drive and change the SCSI adapter to LSI Logic if needed
• Power on your new VM and uninstall VMware tools (VMware server version)
• Restart server
• Remove old virtual machine hardware
    o Open CMD prompt and type “SET DEVMGR_SHOW_NONPRESENT_DEVICES=1” and then “DEVMGMT.MSC”
    o Select “Show Hidden Devices” from top menu
    o Remove old greyed-out non-present hardware, ie. old processor, disk drives, IDE/SCSI controllers, storage volumes, etc.
• Reboot and install VMware tools (ESX version) and you are finished

Method 3 – Use FastSCP and vmkfstools to copy the disk to ESX and convert it to VMFS3 format
• If your Server VM uses IDE disks you will have to convert them to SCSI disks prior to using this method since ESX does not support IDE hard drives. To convert your disks to IDE follow the steps in this VMware knowledge base article: http://kb.vmware.com/kb/1881 Once you complete the conversion to SCSI you can proceed with the below steps.
• Download FastSCP (http://www.veeam.com/veeam_fast_scp.asp) and install on the VMware Server, FastSCP requires the dot net framework 2.0 so download and install this first if you do not have it (http://www.microsoft.com/downloads/details.aspx?FamilyID=0856eacb-4362-4b0d-8edd-aab15c5e04f5&displaylang=en), alternately you can use WinSCP (http://winscp.net/eng/index.php) which is a bit slower then FastSCP
• Once FastSCP is installed run it, click “Add Server” enter your ESX server name/IP, the default port of 22, username and password and click Finish
• Connect to your ESX server, browse to your /vmfs/volumes/ and select New Folder and call it “temp” or whatever you want
• Shutdown the VM from Server that you will be copying the vmdk file from
• Using Windows Explorer, browse to your vmdk files and select both the descriptor vmdk file (small file) and the data vmdk file (-flat large file)
• Paste these into your temp directory on the ESX server and the transfer will begin
• Once the transfer completes login to your ESX service console and change to your temp directory, ie. cd /vmfs/volumes//temp
• To import the file to VMFS3 format type “vmkfstools –i ” the file name is the name of the small descriptor vmdk file, ie. “vmkfstools –i Win2003vm1.vmdk Win2003vm1-new.vmdk”. This create a new copy of your source vmdk file in VMFS3 format, it will automatically create both the descriptor and the data vmdk files
• Create a new VM on your ESX server, select Custom and then select the VM’s configuration. When you get to Select a Disk choose “Use an Existing Disk” and browse to you destination file name you selected above
• Power on your new VM and you should be all set, you can delete your original source file from the temp directory
• Optionally you can move your vmdk files from the temp directory to the VM’s directory that it created
    o Shutdown your VM
    o Login to the Service Console
    o Change to your VM’s directory, ie. cd /vmfs/volumes//myVM1
    o Copy file from the temp directory to the VM’s directory using vmkfstools, ie. vmkfstools –i /vmfs/volumes//temp/Win2003vm1-new.vmdk myVM1.vmdk This will make a copy of your vmdk file in your VM’s directory, you can also change the destination file name to match your VM’s name
    o In the VI Client, edit your VM’s settings, remove the current hard disk (don’t delete it yet) and add new new hard disk, select “Use an Existing Disk” and browse to the new vmdk file in your VM’s directory
    o Power on the VM and if it boots OK you can delete the original virtual disk, you can use FastSCP for this or right-click on your Datastore in the VI Client and select “Browse your Datastore” with the VI Client to delete the two original virtual disk files

How can I kill a stuck virtual machine?
ESX 3.0 method
• Login to the service console
• You can check the VM state by typing “vmware-cmd //server.vmx getstate”
• Type “ps -ef | grep ”
• The second column is your pid of the vmkload_app of the Virtual Machine, you can also type “ps –eaf” to see all running processes
• Type “kill -9 ”
• Check VM state again, it should now be off
• Type “vmware-cmd //server.vmx start” to power on VM

ESX 3.0 Alternate method
• Login to the service console
• Type “vmware-cmd –l” to get a list of all VM’s and there paths
• You can check the VM state by typing “vmware-cmd /<path to VM directory>/server.vmx getstate”
• To forcibly stop type vmware-cmd /<path to VM directory>/server.vmx stop hard”
• Check VM state again, it should now be off
• Type “vmware-cmd /<path to VM directory>/server.vmx start” to power on VM

ESX 3.0 Alternate method
• Login to the service console
• Get the vmid of the VM you want to kill by typing “vm-support –x”
• Kill the VM and generate core dumps and logs by typing “vm-support –X ”
• You will be prompted if you want to include a screenshot of the VM, send an NMI to the VM and send a ABORT to the VM, you must answer Yes to the ABORT question to kill the VM. The entire process will take about 5-10 minutes to run. It will create a tar archive in the directory you run it in.

ESX 2.5 method
• Log into the MUI, and get the PID of the VM from there
• Login to the service console
• Type “kill -9 ” Note - In rare conditions, doing a kill -9 on a VM can take down the entire host

How do I upgrade a existing virtual disk from BusLogic to LSI Logic?
If you upgraded a server from Win2000 to Win2003 it will usually blue screen if you just change the SCSI controller. It will still try and load the BusLogic driver and will not be able to boot. Use this procedure to force it to load the LSI driver prior to changing the controller type to LSI Logic
• Power off the VM you want to change controllers on
• Connect to the Service Console and edit the vmx file for the VM
• Add the following lines to the vmx file o scsi1.present = "true" o scsi1.virtualDev = "lsilogic"
• Power on the VM and it will discover the new SCSI card
• Power off the VM and edit the SCSI Controller settings, change the type to LSI Logic
• Power VM back on, answer Yes for the adapter change message
• Once it boots successfully shut the VM down again (it will have to LSI controllers at this point)
• Edit the vmx file and remove the lines you added above
• Power on the VM again and you will be all set

How can I hide the Vmware tools icon in the system tray?
Edit the registry key “HKEY_CURRENT_USER\Software\VMware, Inc.\VMware Tools” and set ShowTray equal to 0

What are all the files that are located in my virtual machines directory on the ESX server for?
*.nvram file – This file contains the CMOS/BIOS for the VM. The BIOS is based off the PhoenixBIOS 4.0 Release 6 and is one of the most successful and widely used BIOS and is compliant with all the major standards, including USB, PCI, ACPI, 1394, WfM and PC2001. If the NVRAM file is deleted or missing it will automatically be re-created when the VM is powered on. Any changes made to the BIOS via the Setup program (F2 at boot) will be saved in this file. This file is usually less then 10K in size and is not in a text format (binary).

vmdk files – These are the disk files that are created for each virtual hard drive in your VM. There are 3 different types of files that use the vmdk extension, they are:
• *–flat.vmdk file - This is the actual raw disk file that is created for each virtual hard drive. Almost all of a .vmdk file's content is the virtual machine's data, with a small portion allotted to virtual machine overhead. This file will be roughly the same size as your virtual hard drive.
• *.vmdk file – This isn't the file containing the raw data anymore. Instead it is the disk descriptor file which describes the size and geometry of the virtual disk file. This file is in text format and contains the name of the –flat.vmdk file for which it is associated with and also the hard drive adapter type, drive sectors, heads and cylinders, etc. One of these files will exist for each virtual hard drive that is assigned to your virtual machine. You can tell which –flat.vmdk file it is associated with by opening the file and looking at the Extent Description field.
• *–delta.vmdk file - This is the differential file created when you take a snapshot of a VM (also known as REDO log). When you snapshot a VM it stops writing to the base vmdk and starts writing changes to the snapshot delta file. The snapshot delta will initially be small and then start growing as changes are made to the base vmdk file, The delta file is a bitmap of the changes to the base vmdk thus is can never grow larger than the base vmdk. A delta file will be created for each snapshot that you create for a VM. These files are automatically deleted when the snapshot is deleted or reverted in snapshot manager.

*.vmx file – This file is the primary configuration file for a virtual machine. When you create a new virtual machine and configure the hardware settings for it that information is stored in this file. This file is in text format and contains entries for the hard disk, network adapters, memory, CPU, ports, power options, etc. You can either edit these files directly if you know what to add or use the Vmware GUI (Edit Settings on the VM) which will automatically update the file.

*.vswp file – This is the VM swap file (earlier ESX versions had a per host swap file) and is created to allow for memory overcommitment on a ESX server. The file is created when a VM is powered on and deleted when it is powered off. By default when you create a VM the memory reservation is set to zero, meaning no memory is reserved for the VM and it can potentially be 100% overcommitted. As a result of this a vswp file is created equal to the amount of memory that the VM is assigned minus the memory reservation that is configured for the VM. So a VM that is configured with 2GB of memory will create a 2GB vswp file when it is powered on, if you set a memory reservation for 1GB, then it will only create a 1GB vswp file. If you specify a 2GB reservation then it creates a 0 byte file that it does not use. When you do specify a memory reservation then physical RAM from the host will be reserved for the VM and not usable by any other VM’s on that host. A VM will not use it vswp file as long as physical RAM is available on the host. Once all physical RAM is used on the host by all its VM’s and it becomes overcommitted then VM’s start to use their vswp files instead of physical memory. Since the vswp file is a disk file it will effect the performance of the VM when this happens. If you specify a reservation and the host does not have enough physical RAM when the VM is powered on then the VM will not start.

*.vmss file – This file is created when a VM is put into Suspend (pause) mode and is used to save the suspend state. It is basically a copy of the VM’s RAM and will be a few megabytes larger than the maximum RAM memory allocated to the VM. If you delete this file while the VM is in a suspend state It will start the VM from a normal boot up instead of starting the vm from the state it was when it was suspended. This file is not automatically deleted when the VM is brought out of Suspend mode. Like the Vswp file this file will only be deleted when the VM is powered off (not rebooted). If a Vmss file exists from a previous suspend and the VM is suspended again then the previous file is re-used for the subsequent suspensions. Also note that if a vswp file is present it is deleted when a VM is suspended and then re-created when the VM is powered on again. The reason for this is that the VM is essentially powered off in the suspend state, it’s RAM contents are just preserved in the vmss file so it can be quickly powered back on.

*.log file – This is the file that keeps a log of the virtual machine activity and is useful in troubleshooting virtual machine problems. Every time a VM is powered off and then back on a new log file is created. The current log file for the VM is always vmware.log. The older log files are incremented with a -# in the filename and up to 6 of them will be retained. (ie. vmware-4.log) The older .log files are always deleteable at will, the latest .log file can be deleted when the VM is powered off. As the log files do not take much disk space, most administrators let them be.

*.vmxf file – This is a supplemental configuration file in text format for virtual machines that are in a team. Note that the .vmxf file remains if a virtual machine is removed from the team. Teaming virtual machines is a Vmware Workstation feature and includes the ability to designate multiple virtual machines as a team, which administrators can then power on and off, suspend and resume as a single object — making it particularly useful for testing client-server environments. This file still exists with ESX server virtual machines but only for compatibility purposes with Workstation.

*.vmsd file – This file is used to store metadata and information about snapshots. This file is in text format and will contain information such as the snapshot display name, uid, disk file name, etc. It is initially a 0 byte file until you create your first snapshot of a VM and from that point it will populate the file and continue to update it whenever new snapshots are taken. This file does not cleanup completely after snapshots are taken. Once you delete a snapshot it will still leave the fields in the file for each snapshot and just increment the uid and set the name to “Consolidate Helper” presumably to be used with Consolidated Backups.

*.vmsn file - This is the snapshot state file, which stores the exact running state of a virtual machine at the time you take that snapshot. This file will either be small or large depending on if you select to preserve the VM’s memory as part of the snapshot. If you do choose to preserve the VM’s memory then this file will be a few megabytes larger then the maximum RAM memory allocated to the VM. This file is similar to the vmss (Suspend) file. A vmsn file will be created for each snapshot taken on the VM, these files are automatically deleted when the snapshot is removed.

How can I limit the number of vmware.log files that are created for my virtual machine?
You can do this by powering off the VM and then Edit the Settings on the VM.
• Select the Options tab and then Advanced and then click the Configuration Parameters button
• Click the Add Row button and enter log.keepOld for the Name and as many historical log files as you would like to keep for the value (in addition the the original vmware.log file), ie. 3
• Click the Add Row button again and enter log.rotateSize for the Name and a maximum size in bytes for the log files to grow to, ie. 500000 for 500kb
• Click OK and power the VM back on, it will now create vmware.log files up to the size you specified and also only keep as many as you specified.
• Alternately you can disable VM logging by de-selecting “Enable logging” on the Options tab, Advanced page.
• If you do this all logging except for Vmware tools logging will be disabled. If you also wish to disable that you can add a additional row in Configuration Parameters with a name isolation.tools.setinfo.disable with a value of true

How can I disable Copy & Paste operations between the guest operating system and remote console?
You can do this by powering off the VM and then Edit the Settings on the VM.
• Select the Options tab and then Advanced and then click the Configuration Parameters button.
• Click the Add Row button and add the following names with values of false: isolation.tools.copy.enable, isolation.tools.paste.enable and isolation.tools.setGUIOptions.enable
• Click OK and power the VM back on, copy & paste operations will no longer work in the remote console.

What happens to virtual machines in case of a active path failure to my SAN?
When a cable is pulled, I/O freezes for approximately 30-60 seconds, until the SAN driver determines that the link is down, and failover occurs. During that time, the virtual machines (with their virtual disks installed on a SAN) may appear unresponsive, and any operations on the /vmfs directory may appear to hang. After the failover occurs, I/O should resume normally. Even though ESX Server's failover feature ensures high availability and prevents connection loss to SAN devices, all connections to SAN devices may be lost due to disastrous events, that include multiple breakages. If all connections to the storage device are not working, then the virtual machines will begin to encounter I/O errors on their virtual SCSI disks. Also, operations in the /vmfs directory may eventually fail after reporting an "I/O error".

For QLogic cards, you may want to adjust the PortDownRetryCount value in the QLogic BIOS. This value determines how quickly a failover occurs when a link goes down. If the PortDownRetryCount value is , then a failover typically takes a little longer than multiplied by 2 seconds. A typical recommended value for is 15, so in this case, failover takes a little longer than 30 seconds. For more information on changing the PortDownRetryCount value, refer to your QLogic documentation.

For the Windows 2000 and Windows Server 2003 guest operating systems, you may want to increase the standard disk TimeOutValue so that Windows will not be extensively disrupted during failover. For a VMware environment, the Disk TimeOutValue must be set to 60 seconds. o Select Start > Run, type regedit.exe, and click OK. o In the left panel hierarchy view, double-click HKEY_LOCAL_MACHINE, System, CurrentControlSet, Services, then Disk. o Select the TimeOutValue if it exists and set the Data value to x03c (hexadecimal) or 60 (decimal). By making this change, Windows waits at least 60 seconds, for delayed disk operations to complete, before generating errors. o If the TimeOutValue does not exist, select New from the Edit Menu and then DWORD value. In the Name field type TimeOutValue and then set the Data value to x03c (hexadecimal) or 60 (decimal). o Click OK and exit the Registry Editor program.

What is the vswp file that is in my VM’s directory on the VMFS volume?
The vswp file is created to allow for memory overcommitment on a ESX server. By default when you create a VM the memory reservation is set to zero, meaning no memory is reserved for the VM and it can potentially be 100% overcommitted. As a result of this a vswp file is created equal to the amount of memory that the VM is assigned minus the memory reservation that is configured for the VM. So a VM that is configured with 2GB of memory will create a 2GB vswp file when it is powered on, if you set a memory reservation for 1GB, then it will only create a 1GB vswp file. If you specify a 2GB reservation then it creates a 0 byte file that it does not use. When you do specify a memory reservation then physical RAM from the host will be reserved for the VM and not usable by any other VM’s on that host.

A VM will not use it vswp file as long as physical RAM is available on the host. Once all physical RAM is used on the host by all it’s VM’s and it becomes overcommitted then VM’s start to use their vswp files instead of physical memory. Since the vswp file is a disk file it will effect the performance of the VM when this happens. If you specify a reservation and the host does not have enough physical RAM when the VM is powered on then the VM will not start. If you change a memory reservation for a VM it will not take effect until the VM is powered off and then back on. Simply restarting a VM is not enough. The vswp file is only create and deleted at power on/off, once you power off a VM it’s vswp file is deleted until it is powered back on at which time it is re-created.

By default the vswp file is kept in the same directory as the VM’s virtual disk. It’s a good idea to either create partial memory reservation for your VM’s or specify an alternate location for this file so it does not use up valuable VMFS SAN space. By creating a partial memory reservation you can decrease the size of this file and still allow for memory overcommitment, you also allow for ESX to use it’s advanced memory techniques such as page sharing. Alternately you can specify an alternate location for the vswp file so it is not stored in the same directory as your vmdk file. You can do this be editing the vmx file for the machine and adding a parameter “sched.swap.dir = ”. Alternately you can do this through VirutalCenter by powering off the VM and editing the settings for the VM. Then click on “Options” then “Advanced” then “Configuration Parameters”. Click on “Add Row” to add the parameter and it’s value. The value is the path to the directory you want to store the vswp file, ie. “/vmfs/volumes/ServerA-Local/Swap/” Note it is OK to use the symbolic link name of a VMFS volume instead of the UUID. You do not need to worry about updating the existing “sched.swap.derivedName” parameter, it is generated by the VM and written to the config file each time the VM powers on. Thus any changes you make to that file before you power-on the VM will be overwritten by the VM when it powers on. Make sure the directory you set it to is on a shared VMFS volume if you want to still be able to VMotion the VM or have it work under DRS and HA.

How can I disable my VM’s from page sharing?
Page sharing, which is the process ESX uses to conserve memory by eliminating duplicate memory pages, can be disabled by changing a VM’s memory reservation to be the same as the amount of memory allocated to the VM. With a 100% memory reservation the VM will be excluding from any page sharing. Page sharing can be beneficial to the VM’s performance and thought should be given to this before disabling this feature.

Why does VirtualCenter show my VM’s Host Memory Usage higher then the amount of memory that is assigned to the VM?
Host Memory Usage is the amount of memory used by the host that is the VM's memory plus the host memory (overhead) or the amount of memory used to manage the environment. That is why the Host Memory Usage can be higher then what the VM is configured for.

Why does my VM show a high Guest Mem % when it first boots even though it is not using that much RAM?
This is common when a virtual machine first boots. Windows zeroes the contents of all pages in physical memory while booting. This causes the system to become overcommitted almost immediately, as each VM accesses all of its memory. Since the Windows balloon drivers are not started until late in the boot sequence, ESX Server is forced to start paging to disk. Soon after booting the amount of shared memory drops rapidly, and ESX Server compensates by using ballooning to reclaim memory. Page sharing continues to exploit sharing opportunities over time saving additional memory. See this white paper for more on this.  http://www.vmware.com/pdf/usenix_resource_mgmt.pdf

How can I clear an active vmware.log file that is large without restarting the VM?
Normally a new vmware.log file is created everytime a VM is started. Old log files have -# (ie. vmware-9.log) appended to them and by default VMware will save 6 old log files. You can change this behavior as documented in another tip. If you need to clear a large vmware.log file without restarting the server you can use the below technique. “Cat /dev/null >file” deletes the contents of a file, but preserves the file itself, with all attendant permissions.
• Login to the ESX Service Console
• Switch to the VM’s sub-directory, ie. cd /vmfs/volumes/MyVolume/MyVM
• Type “cat /dev/null > vmware.log” this will make the file 0 bytes without effecting the VM.

How can I hot clone a VM without using VMware Converter?
You cannot hot clone a VM with VirtualCenter, a VM must be shutdown to be able to clone it. One option to hot clone a running VM is to install VMware Converter on it. A reboot will not be required if it is running Windows 2003 or Windows XP. Windows 2000 and earlier require reboots because a special driver is installed on the system. Another option to hot clone is using snapshots and vmkfstools, creating a snapshot will prevent any writes to the VM’s original vmdk file as all subsequent writes are written to the –delta.vmdk files that the snapshot creates. You can also use this procedure to cold clone a VM if you do not have VirtualCenter. To cold clone power off your source VM first and there is not need to create a snapshot, you can omit steps 3 and 9 below.
1) Login to the Service Console
2) Switch to your VM’s directory, ie. cd /vmfs/volumes/MyVolume/MyVM1
3) To create a snapshot type through the command line you can use vmware-cmd, the syntax is: “vmware-cmd <path & name.vmx file> createsnapshot <name> <description> <quiesce> <memory>” ie. vmware-cmd MyVM1.vmx createsnapshot “MyVM1 Snapshot” “Clone snapshot” 1 0. Setting memory to 0 prevents the snapshotting of the VM’s memory which we do not want for the clone. It will return “createsnapshot(MyVM1 Clone snapshot 1 0) = 1” when it successfully creates the snapshot. Optionally you can create the snapshot using the Snapshot Manager in the VI Client.
4) Next create a new VM (for this example we will call it MyVM2) on the ESX host using the VI Client. It’s best to assign the NIC for this VM to an Internal Only vswitch (no physical NIC’s assigned to the vswitch) so it does not conflict with the existing VM. When it comes to the hard drive you can accept the 4GB default or make it smaller since you will be deleting it anyway. Do not power this new VM on.
5) Switch to your new VM’s directory and delete the vmdk files it created. Ie. “cd /vmfs/volumes/MyVolume/MyVM2” and then “rm *.vmdk”, you will be prompted for deletion confirmation of the two vmdk files for the VM.
6) Switch back to your original VM’s directory, ie. “cd /vmfs/volumes/MyVolume/MyVM1”
7) Use vmkfstools to copy your original disk to the new VM’s directory, the format is “vmkfstools –i <source vmdk file> <destination vmdk file>” ie. “vmkfstools –i MyVM1.vmdk /vmfs/volumes/MyVM2/MyVM2.vmdk”
8) Once the copy completes power on your new VM, you might run chkdsk on it once it boots since you effectively powered off the VM while running and powered it back on.
9) Yo