0845 094 3366
 

VI-4 Beta

Taupo are currently working with VMware Virtual Infrastructure 4

ESX Command Line Guides

Taupo consultants contribute to popular community site b2v.co.uk

Capacity Planner

Taupo now offer a capacity planner service based on VMware Capacity Planner

Veeam Partners

We partner with Veeam. Check out  Reporter and Configurator!.

 




 

 

 

Home | About Us | Energy Saving | VMware Consulting | Education | Command Line Guide | Contact | Links |

 

In the time we have been working with VMware products we have learned many commands, hints and tips along the way. This page started as an in-house command line reference document for ESX server. However it has grown into a knowledge base for all aspects of deploying on the VMware. Please note, this guide is not affiliated or content-verified by VMware Inc. We recommend our visitors only rely upon the published support documentation found at www.vmware.com

This is very much a living document and updates, suggestions and corrections are all very welcome. Please email us at vmware@taupoconsulting.co.uk  If you find this document useful, feel free to link to it.

NEW!! We now maintain the most up-to-date version of this ESX server guide on a dedicated community website for virtualisation technologies called B2V. You can find the latest version of the guide at www.b2v.co.uk/b2vguide2vmware3.htm

 

 

/etc/lilo.conf LILO is the boot loader (LILO=Linux Loader) used for ESX server (remember the service console is based on a modified version of Red Hat Linux 7.2). Lilo.conf is the text file to describe the OS to boot. This text file is compiled into a binary file that is actually used by LILO. If you are troubleshooting the APPEND line, then use vmkpcidivy tool. You should not have to revert to manually editing this file. If you ever do edit this file, then you need to compile it with lilo –c to write the binary file.

Append=

The pci device mask specified in the append line of lilo.conf is actually an include, not a mask out. The important thing to remember is the append line defines the hardware visible to the service console. Don’t replace LILO with another boot loader, e.g. GRUB. ESX manages allocation of PCI devices between service console and VM kernel with the expectation of LILO.

You can also view PCI device allocation using the MUI, found under Startup Options

Alternatively, you can use the legacy web interface using the URL

http://esxserver/pcidivy

 

Boot Order LILO loads the service console kernel, e.g. /boot/initrd-2.4.9-vmnix2.img This kernel obeys what is stored in the file /etc/inittab
 
/etc/inittab This file is read by the init process and specifies the run level to be used by the service console. The line that states the run level will look something like this.

id:3:initdefault:

The init process then works through the start up scripts in the appropriate diectory. For run level 3, this directory would be 

/etc/rc.d/rc3.d

The file also starts up the virtual terminals on the ESX server mingetty tty2 through mingetty tty5. The mingetty process is minimum getty.

 

/etc/rc.d/rc.local A text file used most commonly when we need to share the service console NIC with the VM kernel. This may be required in say a blade server which has only 2 NICs, and we need to dedicate 1 NIC to Vmotion.

insmod vmxnet_console devName=vmnic0 ifup eth0
 

/etc/rc.d/rc2.d Contains start-up scripts for run level 2

 

/etc/rc.d/rc3.d Contains start-up scripts (logical links) for run level 3. Run level 3 is used most of the time as it is command line full multi-user mode. The ones we are interested in are shown below:

S00vmkstart
S12syslog
S55sshd
S56xinetd
S90vmware
S91httpd.vmware

 

chkconfig --list This service console tool displays a table showing which daemons are enabled for the run levels for the vmnix operating system.

ntpd   0:off 1:off 2:on  3:on  4:on  5:off 6:off
syslog 0:off 1:off 2:on  3:on  4:on  5:off 6:off
snmpd  0:off 1:off 2:off 3:off 4:off 5:off 6:off

If we wanted to change a service so that it is enabled for a particular run level, then we can use chkconfig –level.

chkconfig --level 1 ntpd on

The above command would turn on ntpd for run level 1, this would not affect the run levels that ntpd was already set for. So in this example, the ntpd run levels would be

ntpd 0:off 1:on 2:on 3:on 4:on 5:off 6:off
 

service --status-all This lists all the service daemons and their status. We can find running services by looking for the running status

service –-status-all |grep running

would produce an output similar to the following:

crond (pid 1423) is running
httpd (pid 1486 1482 1479) is running
syslogd (pid 1136) is running
sshd (pid 1208) is running

To avoid unnecessarily rebooting an ESX server after making certain configuration changes, we can frequently just restart the appropriate daemon. For example we could restart the Apache web server for the MUI with the command:

service httpd.vmware restart

and we can also check a named service running status with

service httpd.vmware status
 

S12syslogd The centralised logging system. When ESX is running, both the service console and VMkernel log messages through it.
 
/etc/ssh Secure shell settings

Quick way to restart the ssh daemon. Use full path to do this.

/etc/init.d/sshd restart

/etc/ssh/ssh_config

Text file with the configuration of SSH client.

/etc/ssh/sshd_config

Text file with the configuration file for SSH daemon. An important setting in this file is PermitRootLogin=Yes/No. You can quickly check this with a grep on the file.

grep permit /etc/ssh/sshd_config

~/.ssh/known_hosts

A text file that stores the RSA keys for known hosts. If we rebuild a host and then reconnect to it over SSH we may be prevented if the known_hosts file has cached the old key. I think SCP (secure copy) uses RSA keys as well as SSH sessions. Note this file is maintained on a per-user basis.
 

/etc/xinetd.conf The config file for Xinetd, the network services daemon? The daemon itself is stored in /usr/sbin/xinetd This launches the daemons that are bound to it on demand
 
vmware-authd This is the authentication daemon. This daemon authenticates users of the management interface (MUI) and remote consoles using the username/password database defined in /etc/passwd. This service binds via the xinetd daemon and so the configuration file that specifies the listening port is in

/etc/xinetd.d/vmware-authd

This text file contains the settings for the VMware remote access authentication daemon. This file specifies the TCP:902 port used by remote console. If this port was changed here, it must also be changed in the file /etc/vmware/config.

If we wanted to add Kerberos off-box authentication for MUI access, then its in the pluggable authentication module configuration file that corresponds to this daemon that we would make a change. This file is found at

/etc/pam.d/vmware-authd

We would need to change the current "auth required" to "auth sufficient" and add a last line of "auth required" using the Kerberos authentication module. 
 

S90vmware This script performs the following actions:

vmnix driver   Loaded by modprobe vmnixmod.o
VMkernel       Loaded by vmkloader
Logger        
VMkdump        Any dump will be copied to /root
Starts VMs     Performed by vmware-serverd

 

vmware-serverd This daemon runs on demand in the service console to provide information to any VMware service that needs it. This performs actions in the service console on behalf of the VMware Remote Console and the web based MUI. It is started at boot time to do any VM autostarts. This process is replaced with vmware-ccagent if Virtual Center is installed.
 
S91httpd.vmware This script starts the Apache web server which provides the ESX Server MUI. Configuration is stored in

/usr/lib/vmware-mui/apache/conf/httpd.conf

This process communicates with vmware-serverd for backend data. Remember a refresh in the browser is only a refresh to Apache, to get new data, click on the refresh button to get new kernel data. Remember if the httpd.vmware service starts and then stops immediately, check your service console disk space.
 

vmware-snmpd
vmware-snmptrap
/usr/sbin/vmkstatus This is a perl script which runs every x seconds on tty1.
 
lsmod Lists drivers loaded for the service console linux. Remember this command differs from the vmkload_mod utility which shows the modules loaded for the VMkernel.

Module          Size   Used by Tainted: PF
vmnixmod        177056 121
e1000           68456  0 (unused)
usb-uhci        21220  0 (unused)
usbcore         50112  1 [usb-uhci]
megaraid2       32928  6

The same information can be found by cat /proc/modules
 

 

Starting & Stopping ESX Server

 

init 0 Instructing a halt
 
init 1 Instructing run level 1
 
esx 1 If this is entered at the LILO boot prompt we can get a root shell. We are instructing the vmnix kernel to execute at run level 1 (single user mode).
 
esx 3 Again at the LILO boot prompt, this time, we are instructing the vmnix kernel to execute at run level 3 (the default). If we suspected the run level was wrong we can use this to get back up and running without having to revert to booting Linux on its own.
 
linux rescue Used when you boot the ESX server with a Red Hat Linux CD
 
chroot Change root directory to new directory specified as a parameter
 
shutdown Brings down the system in a secure way.

shutdown -h now       Halt after shutdown
shutdown -r now       Restart after shutdown

 

linux -s At the LILO boot loader, the default options are

esx            Normal ESX boot
linux          Linux SMP kernel, no VMkernel load
linux-up       Linux Uni-processor kernel, no VMkernel load

If we use the cursor key at the LILO screen to select one of the three default choices, the boot prompt (displayed below the menu) changes to reflect this. This allows us to augment the boot command with an option switch.

boot: linux –s

In this case, the –s instructs Linux to boot in single user mode. A critical security point here is that in single user mode, Linux automatically logs on as root! Once in single user mode if we wish to continue into multi-user mode then we type either exit or CTRL-D
 

 

RPM Packages

 

rpm As ESX service console is based on modified Red Hat Linux, we can use the RPM package installation method.

rpm -qa
 

 

Networking

 

/etc/modules.conf Identifies interface eth0 and aliases

alias eth0 e1000
alias eth1 e1000
options e1000 Speed=100

 

Ifup eth0 Brings interface “eth0” up
 
Ifdown eth0   Takes interface “eth0” down.
 
Ifconfig Prints a list of the network interfaces. A quick way of getting IP and MAC of interfaces Can also be used to bring interfaces up or down

ifconfig eth0 up
ifconfig lo down

 

mii-tool

Interface tool. Doesn’t work correctly with some network cards, including Intel 1000 Pro copper NICs.
 

service network restart Great way to do orderly restart of vmnix network configuration. Another way would be to do

ifdown eth0; ifup eth0
 

/etc/hosts Host lookup file, just like windows. The ESX server MUST have an entry for itself in its own HOSTS file.
 
/etc/resolv.conf The DNS name server settings, This file does not need to be present if you are not using DNS.
 
/etc/sysconfig/network

Text file containing hostname and default gateway.
 

/etc/sysconfig/network-scripts/ifcfg-eth0 IP, mask and device name. The text file contains IP configuration for interface eth0, which is reserved for the service console.
 
route Modify or print routing table

route                               Prints routing table
route del –net default              Deletes the default gateway
route add –net default gw w.x.y.z   Adds a new default gateway

 

netconfig Network configuration setup program
 
/etc/issue File which shows ESX and vmnix version

VMware ESX Server 2.1.2
Kernel 2.4.9-vmnix2 on an i686

 

uname -a Outputs details of the server as shown

Linux esx1.taupoconsulting.net 2.4.9-vmnix2 #1 Fri Aug 6 04:38:44 PDT 2004 i686
 

/etc/fstab Local and remote file systems which are mounted at boot. This file is explained in more detail in the mount section.
 
/proc/net/NICfamily/eth0.info A text file that can be checked to see what the service console NIC is doing. The speed of the console NIC specified in the file modules.conf can be confirmed by this file.
 
 

User Administration

 

id Displays the UID for the currently logged on user or if supplied with a parameter can be used to display the UID of a named user.
 
passwd Used to change the logged on users password or for a named account

passwd <user>

Adds a user to /etc/passwd

By default a user added with default options has a group created of the same name. We can add the user to another group, e.g. have a group membership of say ESXUsers

useradd alistair –G vmsqladmins

The above command adds a user called alistair whose primary group (-g) is alistair and other group (-G) membership is vmsqladmins

We can add additional parameters to more fully specify the account.

useradd financeflagshipuser –g Finance –s /bin/false

In the above example the users’ primary group is Finance and the shell is specified. In this case the shell is /bin/false which is a bogus shell which would prevent logon as this user. By default in the service console vmnix, this would be /bin/bash for the Bourne-again shell. It appears the only other shell shipped with vmnix is csh (the C shell).
 

groupadd Adds a group to /etc/group

groupadd esxadmins

In the above example, a new group called esxadmins is created and therefore a new line appears in /etc/group.
 

gpasswd The best tool for modifying the /etc/group file. The following command adds greg to the ESXUsers secondary group.

gpasswd –a greg ESXUsers

Group removal is simple with the –d switch:

gpasswd –d tony ESXUsers
 

useradd This command is used to modify a user. Be very careful with this command as when used with –G to set the users group membership, it is not adding the user to a group but is actually setting the list of secondary groups a user belongs to. Therefore in the following example if bill had secondary group list of ESXUsers and SQLadmin, then after entering:

usermod –G ali bill

then bill would only have a secondary group of ali and nothing else!
 

groupmod Modify a group
 
su Switch user, super user and used without parameters expects root. If we restrict root from logging in over SSH, then we force remote users to authenticate as themselves and then SU to run privileged commands if need be, thus leaving a decent audit trail.
 
sudo Allows delegation of administration in terms of certain commands that normally only a particular user can execute (usually root).
 
visudo This is just the "vi" text editor, but it automatically opens the /etc/sudoers file. The point of visudo is to ensure we always edit the right file as the location of the sudoers file differs between nix distributions, but this command is constant and will utilise the right sudoers file for the distro being used.
 
/etc/sudoers

The text file that contains the sudo users and the rules that apply to them.
 

/etc/group List of security groups.

kirsten:x:505:kirsten
ESXUsers:x:507:kirsten,flagship
flagship:x:508:flagship
vpxuser:x:511:
adminaccount:x:512:
JohnSmith:x:513:

This may look like a list of users, but it is groups. As the service console (vmnix) is a modified version of Red Hat Linux, the Linux security configuration is the same as Red Hat. One feature of Red Hat no found in all Linux distributions is that of the user private group (UPG). Whenever you create a user, a group of the same name is created also and the user is made a member. The format of the file is:

groupname:x:user1,user2

so when we see groups like JohnSmith:x:513 we can assume the 513 is the UID for the user JohnSmith and this is his UPG.
 

/etc/passwd List of security users. When we add a user with either the MUI or a command line tool such as useradd, we are adding to this text file.

ali:x:500:500:Alistair Sutherland:/home/ali:/bin/bash
sara:x:501:501:Sara Daniels:/home/sara:/bin/bash bill:x:502:502::/home/bill:/bin/bash
andy:x:504:504::/home/andy:/bin/bash

the format of the file is

username:x:userID:groupID:fullname:homedirectory:shell

Normally the group ID will match the user ID.

 

/etc/shadow Password file
 
/etc/skel Skeleton directory – new home directories take copies of the files stored here.
 
/var/log This directory stores key log files for both the service console and the VMkernel. Of note are the vmkernel, vmkwarning & messages file logs. These logs can be viewed with the more, cat, head and tail command line tools.

We can also access these logs via the MUI via the following link in the Options tab.


If you use the sudo tool to run a command under a different security context then the log file
/var/log/secure will contain the audit trail for such activity.

Check the file /etc/syslog.conf for logging settings.
 

netstat

Shows currently active network connections
 

lsof List open files Pipe the results into grep to check for open ports

lsof |grep IPv4.\*LISTEN
 

pam Pluggable Authentication Module
 
free -m Shows free memory in the service console. The "-m" switch specifies to display the results in megabytes.
 
fdisk Disk partitioning tool

 

du Disk usage. Great for finding out which folders are using disk space

du –h /home/ali/vmare
du –h ~
du –s summary

 

df -h Print disk partitions (with human readable switch)
 
vdf -h Print disk partitions with knowledge of VMFS partitions (type FB) (with human readable switch)
 
dd Disk dump utility common to Linux. Copy a file while converting and formatting. This can be a quick and dirty way of making an ISO. This could be done in the service console with

dd if=/dev/cdrom of=/vmimages/new.iso bs=32k

This tool could also be used to go from ASCII to EBCDIC etc.

This tool can be used to create an additional swap file. For example, if we did not allocate a big enough swap partition for the service console during ESX installation, we can create one now in a file of 64MB.

dd if=/dev/zero of=/swapfile bs=1M count=64

If we did add a swap file, we would need to make sure it is started when ESX starts. Therefore, an entry in /etc/fstab would be needed as this file describes the local and remote file systems to mount at boot. The total amount of service console swap space is the sum of the swap partition and any swap files that are active.

 

mkswap A command that must be run against a newly created swap file in order to activate it. Think of creating a swap file with dd like creating a partition, then mkswap is like formatting that partition. The swapon command then turns it on when needed.
 
swapon Enables swap file for service console
 
swapoff Disables swap file for service console
 
/proc/swaps A text file that can be checked to see what swap the service console is using. The output contains a priority which shows which swap device will be used first before the other(s). Useful to determine if swap space is getting used and if there is more than 1 swap. Remember this is vmnix (service console) swap, not VMkernel. The VMkernel swap is in one or more files on a VMFS volume (hence the strong recommendation that even when using a SAN, a vmfs volume is created on direct attached storage to allow local swap).

Filename                Type       Size   Used Priority
/dev/sda3               partition  554232 0    -1
/swapfile               file       65528  0    -2

 

 

Basic File Commands

 

touch <filename> When used with a non-existent filename, this tool creates an empty file of name filename. However, this can be used to touch an existing file and update its last modified or last accessed attributes. This could be scripted if required. Watch for running touch against any file stored on a VMFS as there appears to be a problem. Remember that not all Linux tools are modified for VMFS awareness. The VMFS is not an ext3 partition but is a mount point, it’s the VMkernel that accesses any VMFS volumes.
 
cat > file

Enter text at command line and then CTRL-D to write to file.
 

echo blah > file Writes the text following echo command to file. This could be good for quickly creating files

echo modprobe usb-uhci > S92usb
echo modprobe usb-ohci >>S92usb

Another great use of this technique is to make changes to the ESX server configuration via the /proc hierarchy, e.g. changing the number of shares for a VM

echo 2500 > /proc/vmware/vm/nnn/cpu/shares

would change the VM CPU shares to 2500. However such a change would only exist for the duration of the world created for that VM. After the VM is powered off this in memory structure is lost. To make such a change persistent, we would need to add the line

sched.cpu.shares = "2476"

to the VMX file of the virtual machine.
 

head filename By default, the head command prints the first 10 lines of the specified file. We can choose how many lines we want instead of 10 by specifying the –n switch. This is good for looking at the file /proc/vmware/vmhba:x:x:x/0:0 with the –n 22 switch. Also good for using with the file command to determine whether a virtual disk is in ESX format or COW format.

head server.dsk | file -

The “–“ is crucial to making the above command work. For an ESX virtual disk we would expect to see something like

standard input: x86 boot sector

 

tail filename Prints the last 10 lines of the specified file. Just like the head command, there is a –n switch that can be specified to list the last n lines of the named file.
 
sort Can alphabetically or numerically sort redirected command output or files.

sort fruit

or to sort a basic score sheet

sort –g –k 2 scores.txt
 

grep Group regular expression, used to string search the files or command outputs. You can use grep –i to indicate search with case insensitive.
 
find The find utility is used much in the same way as many Windows people used the DIR command. If you know roughly what files you are looking for, then this is the tool. The ls tool simply lists, whereas the find tool will find according to one or more criteria, a common one being find files modified in the last day using the –mtime switch as shown in the table.

–mount    used to ensure it doesn't traverse to remote file systems
-size        obvious
-mtime     modified in the last n*24
-mmin      modified in the last n minutes
-ls          use output format as if ls were used
-name     name the file you are looking for (you just don’t know where it is!)

find –mmin -30                files modified in last 30 minutes
find –mtime 1                 files modified in last 24 hours
find –size +10000             files in excess of 10,000 bytes
find –mount –size +10000 -ls  files on non-remote file system
find –name “hosts” -ls        file called hosts

 

 

vi Standard UNIX text editor

I
ESC
:wq
:q!
 
nano Another text editor, more friendly but you should use –w to avoid word wrap.
 
setup Allows changing of NIC, region, firewall, mouse, keyboard
 
ls -a List files in a directory including hidden (also known as dot files due to their prefix) files.
 
ls -dl */ List directories in long format (does not display files). Could add as a shell alias, say lsd.
 
ll Same as ls –al
 
less Scrollable command line, great for piping large output into
 
more Exactly the same as DOS and Windows, also great for piping large output into.
 
chown Changes file ownership. If only 1 user name is specified then the user ownership is set only and the group ownership if left unchanged as shown in the solaris.vmx example below. However rather than having to use chown and then chgrp straight after it, you can set user and group ownership in one operation by specifying the username(s) separated by a colon as in the netware5.vmx example shown.

#chown ali solaris.vmx
#chown ali:ali netware5.vmx

 

chgrp Changes the group owner for a file, leaving the user owner unchanged.

#chgrp ali file.txt

 

chmod Changes file permission. We can use either letters or numeric equivalency when setting permissions. We set permission for 3 principals, the user, the group and others (ugo). If you are from a Windows background then don’t confuse “o” with owner.

#chmod u+rx,g+r,o+r file.txt

Note that using + or – indicates we are adding to or removing from existing permission. If we wish to reset the permission we use “=” to explicitly set the object permissions.

#chmod u=rx,g=r,o=r file.txt yields r-xr—-r--

Sometimes you will see a chmod using “a” to specify all (user, group & other), so we could quickly set read permissions by

#chmod a-wx,a+r file.txt yields r—-r-—r--

A more common way to set permission is using chmod is using numeric equivalent values (4,2,1 for r,w,x) and permutations thereof.

chmod 777 windows2k.vmx set perms to rwxrwxrwx
chmod 754 windows2k.vmx set perms to rwxr-xr-- (default)

Watch for chmod commands with 4 digits, e.g. chmod 0754. This refers to additional attributes

Sticky bit SUID (Set User ID) SGID (Set Group ID)

The sticky bit can be set on executables which tells Linux to keep the application in memory. The reason for this is to improve load times for other users who wish to run the same executable. This relates to the multi-user nature of UNIX/Linux. Given the speed of memory and disk access nowadays the need to keep applications in memory is much less important and so the sticky bit needed so much.

chmod

Set User ID is used on an executable so that when it is run, it is run under the security context of the file owner not the current user. That means in case I have an application whose owner is ' root ' and it has its SUID bit set, then when I run this application as a normal user, that application would still run as root. Since the SUID bit tells Linux that the the User ID root is set for this application and whenever this application executes it must execute as if root was executing it (since root owns this file).

Set Group ID. Just like SUID, setting the SGID bit for a file sets your group ID to the file's group while the file is executing

To set any of these 3 attributes, we use a 4th digit preceding the usual 3 used with chmod. That digit is set using the following:

4 Set user ID (s) 2 Set group ID 1 Set sticky bit (t)

So if we want to set a file with permission rwxr-xr-x and set the user ID bit we could use

chmod 4755

 

umask Another permissions feature you may encounter is that of umask.
 
 

VMware Tools

 

vmkpcidivy A VMware tool. Used to divide up the PCI resources between the service console and VMs or if they are shared. E.g. 1 NIC to SC, 1 SCSI adapter to share between SC and VMs, 1 SCSI adapter to VMs. This tool is stored in /usr/sbin/vmkpcidivy

This is a command line tool which asks a series of questions. Use with the –i switch for interactive.

This is also used to refreshnames and –q vmhba_devs

For example, if I had a SAN LUN of vmhba1:0:25 and lets say I removed the VMFS from this LUN and wished to use it from the service console, I’d refresh names and then query the vmhba_devs to find out what device name the service console was going to use for this LUN, e.g. /dev/sda.

A very useful feature of this tool is the ability to create a new profile. This adds a new boot option to the LILO boot menu that will have its own allocation of memory and PCI devices. If you are unsure about the changes you are making, then create a new profile e.g. esx (modified)
 

vmkfstools The vmkfstools utility is the tool for managing virtual disks. Remember that to copy into a VMFS could have an adverse affect on other VMs with virtual disks on the same LUN. We always want to avoid using copy to populate a VMFS. Copy operations will update the volume in 16k blocks causing unnecessary SCSI reservations to update vmfs metadata.

The switches that can be used with the command are listed below:

-i                   to import a virtual disk to VMFS
-e
                   to export a virtual disk from VMFS
-s
                   to re-scan for new LUNs
-S                   to set vmfs metadata volume label
-X 6000M ./file.dsk
  to extend an existing DSK to 6GB
-c 4000M ./file.dsk
  to create a new empty virtual disk
-C                   to create a new vmfs volume
-l vmfsname
          to list virtual disks on specified vmfs
-F
                   to set the access mode e.g. public/shared
-k
                   to create a VMkernel swap file
-w
                   to activate a VMkernel swap file
-y
                   to deactivate a VMkernel swap file
-T
                   to convert a vmfs1 volume to vmfs2

Remember that the vmfs parameter always goes last on this command parameter set for vmkfstools. This can be confusing for the beginner as the source and target order is different for imports and exports.

If we want to simply list the files on a vmfs volumes we use the -l switch.

vmkfstools –l /vmfs/vmhba0:0:0:8

or if we wish to use the volume label;

vmkfstools –l <vmfs-metadatalabel>

would produce an output similar to the following

Name: VMFS2-VOL1 (public)
Capacity 129465874944 (123461 file blocks * 1048576) avail
Permission Uid Gid Attr Bytes (Blocks)    Last Modified Filename
rw-------   0   0 swap 2146435072 ( 2047) Nov 18 18:25  Swap.vswp
rw------- 500 500 disk 4194304000 ( 4000) Nov 16 14:12  VM1.dsk
rw------- 500 500 disk 6291456000 ( 6000) Nov 23 22:19  VM2.dsk
rw------- 500 500 disk 2621440000 ( 2500) Nov 17 23:09  VM3.dsk
rw------- 500 500 disk 4194304000 ( 4000) Nov 24 18:11  VM4.dsk

To create a new virtual disk we use the -c switch

vmkfstools –c 2048M /vmfs/vmhba0:0:0:8:newdisk.dsk

This command would create a new virtual disk (monolithic) on the specified vmfs volume. Remember it is always better to use the VMFS name as this will not change even if your hardware does.

To import a virtual disk into the VMFS we use vmkfstools with the -i switch. This will take a virtual disk in COW format into monolithic format without causing excessive SCSI reservations on the LUN holding the target VMFS

vmkfstools –i /vmimages/file.vmdk /vmfs/vmhba1:0:25:1/virtdisk.dsk

As always with this command, the parameter specifying the VMFS location is always the last parameter.
 

vmware-cmd A command line tool to perform VM operations. This tool is found in /usr/bin

vmware-cmd –s register /home/ali/vmware/newVM/newVM.vmx

vmware-cmd –s unregister /<path to VM>

vmware-cmd –l

vmware-cmd /home/user/vmware/server/server.vmx stop soft

vmware-cmd /home/user/vmware/server/server.vmx getheartbeat

vmware-cmd /home/user/vmware/server/server.vmx getstate

vmware-cmd /home/user/vmware/server/server.vmx getid

vmware-cmd /home/user/vmware/server/server.vmx getpid

The stop type can be soft, try soft or hard.

 

vmkdump This is used to manage the VM kernel core dump partition. We can change the partition used if required. This tool is also needed if the core dump partition had been removed because ESX expects it to be there when starting up, so we need to tell ESX that it has gone.

vmkdump -q              Query the VM kernel for which partition it will use
vmkdump -p vmhba0:0:0:3 Set VM kernel dump partition to partition 3
vmkdump –p none:0:0:0     Set VM kernel dump partition to none

Remember the vmkcore partition does not have a mount point in the service console and is not specified as ext3. We can use the fdisk -l command to view where the core dump partition is in relation to the disk layout.
 

vmkload_mod Allows viewing with the –l switch, loaded and unloaded VMkernel modules. This command differs from lsmod which lists the modules loaded for the service console. This is a very good way of differentiating what modules the kernel is using versus the ones used by vmnix.

#vmkload_mod -l

Name      R/O Addr Length R/W Addr  Length ID Loaded
vmklinux  0x4de000 0xf000 0x12516b0 0x53000 1 Yes
nfshaper  0x4ed000 0x1000 0x12a81b0 0x1000  2 Yes
e1000     0x4ee000 0xf000 0x12a91b8 0x6000  3 Yes
megaraid2 0x4fd000 0x6000 0x12f6008 0x3000  4 Yes
bond      0x503000 0x2000 0x138a158 0x2000  5 Yes

#

Note, the -l parameter can also be specified as --list
 

 

Devices

 

/etc/modules.conf Not only lists eth0 (as discussed in network section) but in fact this text file sometimes describes the devices that are assigned to the service console by vmkpcidivy.
 
modinfo This tool takes a service console driver and displays the options it supports. For example

modinfo e1000

Would produce a list of flow control settings for the Intel gigabit NIC.

modinfo cciss

Would produce the file details and version of the HP Smart Array controller.

 

lspci Great tool for listing pci devices. Could be used to demo what the VM is presenting to the guest OS.

lspci –v   verbose output
lspci –t   print tree of PCI devices (use –tv for verbose tree)

You may wish to examine /proc/pci also in order to correctly identify PCI devices and their slot configurations. One point to note is that when you are faced with PCI slot numbers is that not all hardware vendors number their slots in a straight forward left to right configuration. Make sure you know your slot numbers and their layout!
 

lsusb A tool to list USB devices. Gives out way more info that is actually required. Remember that USB devices cannot be presented to virtual machines in ESX Server. If you wish to use a USB device in ESX, then you will have to use a USB over IP device and install the appropriate driver software into your guest OS for this. The most common USB over IP device is companyX
 
kudzu A Red Hat tool to detect and configure hardware.
 
/etc/vmware/netmap.conf This text file maps ESX virtual switch names to device names. It is a network map config file as opposed to the more generic devices map config file (devnames.conf).

network0.name = "SecuredGigabit"
network0.device = "vmnic2"
network1.name = "VirtualSwitch1"
network1.device = "vmnet_1"
network2.name = "InternetSwitch"
network2.device = "vmnic1"

Remember that if a virtual switch has no physical adapters, then it is vmnet_x. If a virtual switch has only 1 physical adapter is is vmnic_x. If there are two or more physical adapters assigned to a virtual switch, then the device is bondx.

A bond can be in one of three modes, out-mac (default), out-ip and standby

out-mac A VM virtual NIC is assigned to a pNIC in the bond and it uses only that
out-ip A VM TCP conversation is placed on an available pNIC
standby A VM will only use one NIC until a failure, then the other is used

 

/etc/vmware/devnames.conf This text file maps device names (example above) to modules and their PCI addresses. Note that the devnames.conf file contains SCSI devices and NIC devices.

002:14.0 megaraid vmhba0
005:04.1 nic vmnic0
011:07.0 nic vmnic1
012:08.0 nic vmnic2

 

/etc/vmware/vmware-devices.map Appears to be like a hardware compatibility list. Watch out for creating your own device map, devices.local The /etc/vmware/vmware-devices.map file contains a list of devices supported by ESX Server. This release includes support for a local version of this file, /etc/vmware/vmware-devices.map.local. Modify the vmware-devices.map.local to select different device drivers. This file is not modified during an ESX Server upgrade, preserving your customizations. The vmware-devices.map.local is read when the VMkernel is loaded:

Any changes to the vmware-devices.map.local file require a reboot, or at least an unload/reload of the VMkernel to take effect.

Entries in the vmware-devices.map.local files are used in addition to the entries in the vmware-devices.map file. The vmware-devices.map.local file does not need to mirror the vmware-devices.map file.

Any vmware-devices.map.local file entries that correspond to the vmware-devices.map file entries supercede the vmware-devices.map file entries.

 

/etc/vmware/config
/etc/vmware/hwconfig Loads of information in this text file. Useful for finding which nic is in which team. If using alongside devnames.conf and netmap.conf use the following command

grep –i vmnic /etc/vmware/hwconfig

 

/dev/fd0 How to address the floppy disk drive
 
/dev/sda SCSI device
 
/proc/vmware/pci Text file with discovered PCI devices

Bus:Sl.F Vend:Dvid Subv:Subd Type Vendor ISA/irq/Vec P M Module Name Spawned bus
000:00.0 8086:3590 1028:016e Host/PCI Intel C
000:02.0 8086:3595 0000:0000 PCI/PCI Intel 001 C
000:03.0 8086:3596 0000:0000 PCI/PCI Intel 004 C
000:04.0 8086:3597 0000:0000 PCI/PCI Intel 007 C
000:05.0 8086:3598 0000:0000 PCI/PCI Intel 010 C
000:06.0 8086:3599 0000:0000 PCI/PCI Intel 013 C
000:29.0 8086:24d2 1028:016e USB Intel 11/ 16/0x69 A C
000:29.1 8086:24d4 1028:016e USB Intel 10/ 19/0x71 B C
000:29.2 8086:24d7 1028:016e USB Intel 7/ 18/0x79 C C
000:29.7 8086:24dd 1028:016e USB Intel 3/ 23/0x81 D C
000:30.0 8086:244e 0000:0000 PCI/PCI Intel 016 C
000:31.0 8086:24d0 0000:0000 PCI/ISA Intel C
000:31.1 8086:24db 1028:016e IDE Intel 0/ / A C
001:00.0 8086:0330 0000:0000 PCI/PCI Intel 002 C
001:00.2 8086:0332 0000:0000 PCI/PCI Intel 003 C
002:14.0 1028:0013 1028:016e RAID Dell 7/ 38/0x91 A S vmhba0 004:00.0 8086:0329 0000:0000 PCI/PCI Intel 005 C
004:00.2 8086:032a 0000:0000 PCI/PCI Intel 006 C
005:04.0 8086:1010 8086:1012 Ethernet Intel 7/ 58/0x99 A C 005:04.1 8086:1010 8086:1012 Ethernet Intel 10/ 59/0xa1 B V e1000 vmnic0
010:00.0 8086:0329 0000:0000 PCI/PCI Intel 011 C
010:00.2 8086:032a 0000:0000 PCI/PCI Intel 012 C
011:07.0 8086:1076 1028:016d Ethernet Intel 11/ 72/0xa9 A V e1000 vmnic1
012:08.0 8086:1076 1028:016d Ethernet Intel 11/ 73/0xb1 A V e1000 vmnic2
016:13.0 1002:5159 1028:016e Display ATI 7/ 18/0x79 A C

 

findnic This tool is also known as “The VMkernel Network Card Locator”. It locates a physical NIC in an ESX server by using pings. It is left to the operator to check which NIC is sending the echo requests by either unplugging network cables or inspecting flickering lights on the NIC or the switch. The useful feature of this tool is we can ping based on the device name that the VMkernel uses to access the NIC. We should remember that each physical NIC in an ESX server that is allocated to the VMkernel does not itself have an IP address, therefore in to perform an ICMP echo request, we need to temporarily give that NIC an IP; this is specified as the first IP parameter in the command, the second IP parameter being the ICMP destination.

findnic –i 5 vmnic2 10.0.0.1 192.168.1.3

The above command will send ICMP echo requests to 192.168.1.3 every 5 seconds. We could also use the –f switch which would flood ping.

 

 

File Systems

 

procfs The volatile /proc directory hierarchy can be treated as a file system but is held in RAM.
 
/proc/vmware/sched/cpu A text file snapshot of cpu scheduling. You could increase CPU shares using this method

echo 10000 > /proc/vmware/vm/<number>/cpu/shares

 

watch This is a fantastic utility that polls whatever command you supply it with an displays a running changing status. For example, we could use the command

watch cat /proc/vmware/mem

to obtain a dynamic view of memory usage by VMware. If you are viewing lots of output but can't see what is actually changing we use the -d parameter to specify differences, thus highlighting changes between refreshes.

watch –d cat /proc/vmware/mem

To specify a different refresh rate, we use the --interval=n parameter

 

smbclient This is a redirector type tool to view and connect to SMB (Microsoft networking) hosts. Before updating the /etc/fstab file with remote file system information, check first using smbclient that the share is visible. The following was produced with

smbclient –I 192.168.1150 –U username –L computername

Sharename Type Comment
--------- ---- -------
E$        Disk Default share
Microsoft Disk
IPC$      IPC Remote IPC
D$        Disk Default share
NETLOGON  Disk Logon server share
REMINST   Disk Remote Installation Share
ADMIN$    Disk Remote Admin
SYSVOL    Disk Logon server share
VPLOGON   Disk Symantec AntiVirus
C$        Disk Default share
VPHOME    Disk Symantec AntiVirus

To create a mount point to a Microsoft share is very straightforward. Remember, we are allowing the service console to access a remote file system. This is not related to what virtual machines are doing. Further, we need to be careful if we are attempting to do any file operations due to potential limits with 2GB file sizes.

1. Add entry to local hosts (/etc/hosts) for MS host
2. Create directory where you wish the mount point with mkdir
3. Run smbclient –I <ipaddress> -U user –L NetBIOSComputername to check you can see the shares
4. Add an entry to /etc/fstab

//server/share /mountdir smbfs ip=ipaddress,username=user,password=pass,noauto 0 0

5. Mount the remote file system with mount /mountdir
6. Change directory into mount point and the directory will no longer be empty but will appear just like a mapped drive.

Alternatively, if you just want to map temporarily to a Microsoft host and not have to modify fstab, then use smbclient interactively as shown:

smbclient //ipaddress/share –U NThost\NTuser Password: ****** Domain=[TAUPO] OS=[Windows 5.0] Server=[Windows 2000 LAN Manager] smb: \>

 

NFS Network File System (NFS) is provided by rpc.nfsd and would normally be launched by an nfs script in /etc/rc.d. To start using NFS to mount directories on other ESX or Linux servers, we can use the following steps:

1. Change the ESX Server which is to be the NFS server to use medium security (using the MUI is easiest for this)

2. Add an entry to the file /etc/exports on the NFS server by either using vi text editor (/vmimages *) or use the exportfs command

3. Check the NFS client is not excluded by the server file /etc/hosts.deny

4. Up to you if you explicitly allow the NFS client by adding the NFS client to /etc/hosts.allow on the NFS server

5. Start the NFS server daemons
  a.
/etc/rc.d/init.d/service portmap start
  b.
/etc/rc.d/init.d/service nfslock start
  c.
/etc/rc.d/init.d/service nfs start

6. Start the NFS client daemons
  a.
/etc/rc.d/init.d/service netfs start
  b.
/etc/rc.d/init.d/service portmap start

7. Mount remote export directory from the client with the command
  a.
mount –t nfs server:/export localdir/localmount

 

showmount This command is used by a NFS client to see what directories are being exported by a NFS server.

showmount –e nfsserver

This command can be specified with the hostname name or IP address of the NFS server holding the exported directories.

 

exportfs The exportfs command allows you to selectively export or unexport directories without restarting the various NFS services.
 
NIS NIS stands for Network Information Service. Ensures numeric owner IDs are unique across the organisation. This is because numeric owner IDs are used in NFS, so we are often going to have mistaken identity as used 515 on the nfs client will not be the same as user 515 on the nfs server.
 
vmware-mount.pl Just like the Windows utility to mount virtual disks when they are powered off so you can check what’s in them. This will mount ext3 and vfat as read/write but NTFS as read only. To find out what file systems are in the virtual disk, use the –p switch.

vmware-mount.pl –p /vmfs/VMFS-VOL1/win2k3.dsk

--------------------------------------------
VMware for Linux - Virtual Hard Disk Mounter Version: 1.0 build-9638\nCopyright 1998 VMware, Inc. All rights reserved. -- VMware Confidential
--------------------------------------------

Nr      Start       Size Type Id Sytem
-- ---------- ---------- ---- -- ------------------------
1          63    8160957 BIOS  7 HPFS/NTFS

If we actually want to mount a partition then we need to be specific and create a directory (or use an existing) to be our mount point.

mkdir /myntfs
vmware-mount.pl /vmfs/VMFS-VOL1/win2k3.dsk 1 –t ntfs –o ro /myntfs

This command will tie up the console window hence you’ll need to spawn a new window first to navigate to /myntfs to view the contents.

 

mount Mount file system This command can be used with the –t switch to specify file system type, e.g. nfs, smbfs or iso9660

mount -t nfs [-o options] esx4:/vmimages /root/other_server_vmimages
 

umount Un-mount file system. Note it is u-mount and not unmount!
 
smbmount A mount –t smbfs passes control across to this utility. We can use this utility directly if we prefer for mounting SMB host file systems.
 
/etc/fstab If we have smb mount points permanently, then this file could end up with user credentials in it. This file is readable by everyone so this is not good. We can place the credentials for the smbmount in a hidden secured file in our home folder eg. /root/.smbcreds

echo username=user > .smbcreds echo password=pass >> .smbcreds chmod 600 .smbcreds

Then in the /etc/fstab file we substitute the username and password for credentials=/root/.smbcreds. Therefore the whole line in the fstab would be

//server/share /mountdir smbfs ip=ipaddress,credentials=/root/.smcreds,noauto 0 0

 

 

Shell

 

.bashrc A hidden file that extends shell script for the BASH shell. This is found in the users home folder.
 
/etc/bashrc

System-wide bash shell settings
 

clear Equivalent of CLS command in MS-DOS and in Windows command prompt.
 
sum Prints the 16-bit checksum and size of the specified file.
 
md5sum Prints the MD5 checksum for the file supplied as a command parameter. VMware publish md5 checksums for all downloads to allow the end user to confirm that the file had not be corrupted in download.
 
cal Display calendar for current month -3 last, current and next month 200x display year cal 2005 | less to get scrollable calendar
 
/etc/vmware/vm-list Auto generated text file listing the VMs on the server. This file should not be manually edited, should match up with output of vmware-cmd –l.
 
 

Process Management

 

PID PID stands for Process ID. Every running process has a process ID that is valid for the length of process execution.
 
PPID PPID stands for Parent Process ID, which is the PID of the process that launched that process.
 
ps Show running processes in the service console.

ps –A ps –eaf
ps –eaf |grep vmware-serverd
ps –efw

is useful as the “w” indicates wide format, so we can see the full directory path to the vmx file.

 

pstree -ph Show process tree with PIDs and ancestors
 
esxtop VMware version of process viewer top
 
renice Change process priority. If there were many VMs running concurrently, it may be necessary to temporarily raise the priority of the MUI in order to complete administrative tasks

#ps –eaf |grep vmware
#renice –p <pid of vmware-serverd>
#renice –p <pid of httpd.vmware>

To reset the PID of these processes back to their defaults, use renice again to set the priority to zero.

 

pidof Finds the PID of a named process.

#pidof vmware-authd

 

& We can configure processes to run in the background by adding a “&” suffix when launching the program.

#sleep 10 &

 

jobs List the process jobs running in the background.
 
 

SAN & Disk Management

 

Disk.MaxLUN=8 This is the default setting, which scans 0 to 7 on start up. Remember if we what to scan up to LUN N we must set Disk.MaxLUN to N+1. This setting is found in the /etc/vmware/vmkconfig file but should be modified via the MUI (Options tab, Advanced Settings).
 
Disk.MaskLUNs This parameter controls LUN visibility, again this is in the /etc/vmware/vmkconfig file. LUN masking is only supported on fibre channel HBAs. This overrides the Disk.MaxLUN setting.

vmhba0:0:4,6-255 would scan 0,1,2,3,5 i.e. skip 4 and skip 6 through 255 vmhba0:0:3,4,9-255 would scan 0,1,2,5,6,7,8 i.e. skip 3 & 4 and skip 9 through 255

 

/etc/vmware/vmkconfig This is a text file which stores vmkernel configuration, like an INI file or registry key. The important fact to note is that this file does not exist until you make an edit away from the default setting.
 
vmkfstools –s vmhba0

Used to re-scan for new LUNs on specified host bus adapter.

 

proc/vmware/scsi
 
WWN SAN devices are identified by a world wide name, a unique 64-bit address. Remember we can use the perl script wwpn.pl to determine quickly what the WWN is for the installed FC hba.
 
LUN A LUN identifies individual units of storage behind a SCSI ID. A LUN could be a single disk or a RAID5 volume.
 
Zoning Zoning is either hard (switch port) or soft (WWN controlled)
 
LUN Masking LUN Masking is a disk array feature that controls which LUNs are presented to which WWNs
 
/proc/scsi/driver/number The WWN for an adapter would be found in this file.
 
Disk.SupportSparseLUN The setting Disk.SupportSparseLUN should =1 as LUNs may be discontiguous. For example if there are disk volumes at LUNs 0,1,2 and 6 then we want to be sure that after LUN 2, the VMkernel storage driver does not stop scanning. We want the LUN scanning to reach the last LUN specified in DiskMaxLUN parameter, regardless of whether the visible LUN numbers are contiguous or not.
 
Disk.RetryUnitAttention=1 This setting tells ESX server to retry SCSI commands as vendor specific status codes may have been received and ESX may think the volume is present but not accessible when in fact its just a message that cache has been upgraded.
 
 

Performance

 

Free A simple utility to display free memory in the service console.
 
esxtop This is the VMware version of top and provides CPU, memory and disk configuration realtime information just like top does, but this time we only see the data relating to the VMkernel so we see worlds instead of processes as resource consumers.
 
top Shows the running processes in the service console and lists the top consumers of CPU time.
 
iostat
vmstat Don’t get confused by this command, it is a Linux command, not a VMware command. This is meant to view Linux processes, memory and paging. We generally use vmstat with 2 numeric parameters, the first parameter is how frequently the tool should run (specified in seconds). The second parameter specifies how many times the tool should run e.g.

vmstat 3 10

would run the tool every 3 seconds for 10 times and then exit.
 

man cpu Great info on using the /proc/vmware file system to change share allocation and processor affinity.
 
/proc/vmware In a subdirectory called /vm there are subdirectories for each vm labelled by number How do you find out what number corresponds to what VM?

grep –i worldid= ~ali/vmware/ISAserver/vmware.log

Some other examples of great info that can be extracted are outlined in the following table:

/proc/vmware/vm/139/cpu affinity 0,1,2,3
/proc/vmware/vm/139/cpu shares 1000
/proc/vmware/vm/139/cpu status
/proc/vmware/vm/138/cpu hyperthreading

/proc/vmware/sched/cpu

 

vmkusage Generates web page usage reports. Was not automatically installed in previous versions of ESX

vmkusagectl install to install the utilisation web pages & setup a cron job vmkusage –graph to generate graph images vmkusage –regroove to wipe the db and start stats again

 

hstatus This is a web page like the MUI accessible via

http://esxserver/hstatus

Not sure if we need to be logged in for this to work, but vmkusage does appear to be required. We get loads of output on this page, similar to running command line tools. This is part of what looks like the legacy interface to ESX server, i.e. it doesn’t look as cool as the MUI of ESX 2.x.
 

crontab Lists the cron jobs scheduled for the user when used with the –l (list) switch.
 
 

SNMP

 

snmpwalk Used to walkthrough SNMP mibs. -M – use MIBSDIR -m all use mibs list instead of default mibs list
 
Trap Receiver A utility to display SNMP traps. MIBs can be loaded into this.
 
 

Perl Scripts

 

vmsnap.pl Backup script to backup a virtual disk whilst the virtual machine is still running. This is achieved by the script creating a REDO file and if necessary a REDO file of a REDO file; thus reducing the changing data and allowing a straight copy of the non-changing file.
 
vmres.pl This script is the restore tool for vmsnap.pl
 
wwpn.pl Used to determine the world wide name (WWN) for hba.
 
vmkstatus.pl This Perl script runs on tty1 of the ESX server.
 
pbind.pl Create or delete HBA persistent bindings
 
vmware-mount.pl This is used to mount a virtual disk in the service console while the virtual machine is not running. For example, if a configuration file in the guest OS virtual disk is preventing a virtual machine from booting correctly, you could