There are several virtualization technologies within the full realm of virtualization. They are:
|Type 1 Hypervisors (bare Metal)||Exmaples: VMware ESX, VMware ESXi, Microsoft HyperV, Citrix XenServer|
|Type 2 Hypervisors (hosted)||Examples: VMware Server, VMware Workstation, VMware Fusion, Microsoft Virtual Server, Parallels|
|Shared operating system bits (containers)||Examples: Parallels Virtuozzo Containers, Solaris Containers, OpenVZ, Linux chroot|
This Refcard, the first in a multi-part series on virtualization technologies, will cover Virtual Networking for Type 1 Hypervisors, specifically VMware ESX and ESXi.
A Virtualization Host is an appliance that contains storage, networking, and computer resources
Getting your ducks in a row for virtualization is very important. Those steps are:
In this stage you are looking at whether you should virtualize or not. Some considerations that need to be made include:
|New Hardware||Do you need new hardware in order to virtualize your environment.|
|Peripherals||If you are virtualizing systems that contain peripherals you may need to purchase additional hardware such as USB over IP devices.|
|Special Hardware||Verify that there is something within the virtual environment that could replace any specialized hardware. The available virtual devices include:
|New Software||Do you need to invest in different management agents and tools on top of the virtualization software.|
|New Licensing||Do you need to change your licensing for operating systems, and applications.|
|New Management||Do you need to invest in new management tools to properly manage your virtual environment.|
|Training||What training do administrators, users, and managers need to go through to use the virtual environment.|
Everyday Users should NOT know they are within a virtual environment.
Capacity Planning helps you determine how much virtualization is needed in your environment. You will need to minimally know the following attributes:
These items and many more are picked up by automated tools such as VMware Capacity Planner, Xcedex X-Factor, and Novell Power Recon.
Those items with high CPU and IO Utilization numbers need to be carefully considered before virtualizing.
The general rule is:
For High IO Utilization you may wish to consider using VMDirectPath SCSI and Network connections within VMware ESX.
Requirements for VMDirectPath devices could limit the number of VMs per Host and increase the number of required PCI-e/PCI-X slots required per host.
Each VM added to a Virtualization Host affects the CPU, Network, and Disk performance of every other VM on the host.
Virtualization using ESX or ESXi is 90% planning and 10% doing. A good design/plan is required. Things to consider for your Design/Plan are:
Consolidation ratios depend entirely on the workload to be used within the virtual environment.
No two companies' workloads are the same, run your own tests. On average a host can run 6-8 single vCPU Server VMs per core and 11-12 single vCPU virtual desktops per core. For light loads more is possible.
Consolidation ratios in many ways depend on the number of VMs you feel comfortable having in one host. For example, the average could be 30 VMs to 1 host.
Depending on your high availability and redundancy requirements you may need 2-3 x the standard number of hosts. Virtualization hosts should contain enough memory so that workloads do not swap and therefore impact performance It should be noted that the performance of each individual VM affects every other VM. The use of VMware Fault Tolerance (FT) will reduce consolidation ratios as FT runs a shadow copy of VMs on other nodes within the VMware Clusters.
|CPU||The amount of CPU assigned to any one virtual machine. When there is CPU contention VMware Dynamic Resource Scheduling can move the VM between hosts in the same VMware Cluster. Controlled by the number of CPU shares, MHz Reservation, and MHz Limit assigned to any one VM.|
|Memory||The amount of memory assigned to any one virtual machine. When there is memory contention VMware Dynamic Resource Scheduling can move the VM between hosts in the same VMware Cluster. Controlled by the number of Memory shares, RAM, and RAM Limit assigned to any one VM.|
|Disk||The amount of Disk IO assigned to any one virtual machine. Controlled by the number of Disk shares assigned to any one VM.|
|Network||The amount of network IO assigned to any one virtual switch portgroup. Controlled by Quality of Service controls per virtual switch portgroup.|
VMware Shares, Reservations, Limits
Each VM starts with a set number of shares:
|CPU||Normal - 1000 times # of vCPU shares - 0 MHz (unlimited) Reserved, Unlimited Ex: 1 vCPU = Shares: 1000; Reservation 0 MHz; Limit: Unlimited|
|Memory||Normal - 10 times RAM assigned shares - 0 MBs Reserved, Limit in MB of RAM assigned Ex: 512MB = Shares: 5120; Reservation 0 MBs; Limit: 512|
|Disk||1000 Shares Ex: 1 Disk = Shares: 1000|
|Network||Per Portgroup NOT VM|
The Service Console also has shares, reservations, and limits: Service Console = CPU Shares: 500; CPU Reservation: 233MHz; CPU Limit: Unlimited; Memory Shares: 500; Memory Reservation: 0 MB; Memory Limit: Unlimited.
More VMs adds to the total number of shares for a given host: Ex. 10 1 vCPU VMs each with 1 GB of Memory and 1 disk given the following number of shares:
|Total CPU Shares:||10500 = 10 * 1000 (VMs) + 500 (SC)|
|Per VM CPU Shares:||1000 or 1000/10500 of system = ~9.5% of the host|
|Total Memory Shares:||102900 = 10 * (10 * 1024) (VMs) + 500 (SC)|
|Per VM Memory Shares:||10240 or 10240 / 102900 of system = ~9.95% of the system|
|Per VM Disk Shares:||1000 = 1000/10000 = 10% of the system disk IO|
Resource Pools are containers for VMs that can limit CPU and Memory that the contained VMs can use in total. There exists a Top Level Resource Pool that is the full amount of CPU and Memory resources available to the VMware Cluster.
Resource Pools may apply CPU or Memory Limits to a pool of VMs and can also be nested as long as the definitions for CPU and Memory limits are reduced in the inner pools. For example, Pool C within Pool B can be defined with less CPU and Memory resources than Pool B.
Lower Resource Pools can be expanded on the fly by borrowing resources from a parent pool.
Each Resource Pool also has Shares, Reservations, and Limits. The default settings are as follows:
|CPU Shares:||Normal - 4000|
|CPU Reservation:||0 MHz (unlimited, expandable)|
|CPU Limit:||Unlimited (Available MHz)|
|Expandable:||Memory Shares: 163840|
|Memory Reservation:||0 MBs (unlimited, expandable)|
|Memory Limit:||Unlimited (Available MBs), Expandable|
A VM placed outside Resource Pools will have more available resources than any VM within a Resource Pool if expandable is not selected. This confuses share calculation as total number of CPU and Memory Shares cannot exceed limits of resource pool into which the VMs and other Resource Pools have been placed except when pools are expandable.
Ex. 2 default non-expandable Resource Pools each containing 6 default 1 vCPU, 1024MB VMs:
|Total CPU Shares within Resource Pools:||6000 = 6 * 1000 (VMs)|
|Total CPU Shares:||8500 = 2 x 4000 (RPs) + 500 (SC)|
|Per Resource Pool CPU Shares:||4000 = 4000/8500 = ~47% of Host|
|Per VM CPU Shares per Resource Pools:||~666.66 = 4000/6 VMs or ~16.6% of the Resource Pool which is ~7.8% of the host|
Base component is the virtual switch (vSwitch). A vSwitch is a very simple Layer-2 device with a software CAM table. Virtual Distributed Switch is superset of the vSwitch. A container for identical vSwitches across all vSphere ESX/ESXi Hosts. Virtual Distributed Switches also add the ability to use Private VLANs and other security constructs.
Virtual Networking Design Considerations
Virtual Networking encompasses redundancy, security, and performance. There should be 2 physical NICs per network/ trunk entering a VMware ESX or ESXi host for redundancy and performance.
For the best security results, each virtual network security zone should be connected to different physical switching networks. Less than 4 physical NICs is considered insecure, lacking in redundancy, and will suffer performance degradations.
Virtual Switches cannot be layered or connected directly to each other. There needs to be a VM between two virtual switches acting as a router, gateway, bridge, or firewall to connect to vSwitches.
Common Virtual Networking Examples
The most common question is how to configure virtual networking with either 4 or 6 physical NICs (pNICs). These examples will assume the following:
With 4 pNICs there is quite a bit of overlap between all the various networks and network performance would need to be considered. Each pNIC on vSwitch0 has several networks running over it so VLANs or subnets will be necessary.
It is best when using less than 6 pNICs to choose to not use this configuration for a DMZ UNLESS pNIC2 and pNIC3 connect to separate physical switching networks due to the hostile nature of this network.
With 6 pNICs some of the overlap disappears and you have enough pNIC to have both a standard virtual machine network as well as a DMZ.
With 6 pNICs there are now 3 vSwitches and each pNIC has a specific duty:
In general, you want to have separate pSwitches to increase security from Layer-2 network attacks instead of using VLANs. Use of VLANs is a trust that your pSwitches are not susceptible to Layer-2 attacks.
When you introduce a DMZ into the mix for 6 pNICS our virtual network looks like our standard 4 pNIC case for all but the DMZ which uses the extra 2 pNICs.
What makes this configuration work with a DMZ is the use of separate pSwitches for each pair of pNICs. This gives the virtual network security and redundancy. However, IP Storage performance can suffer.
Bridging between vSwitches
The last example network is one where you have a need to bridge between two vSwitches using a virtual appliance. The use of this is for:
There is a major Caveat however:
The second Portgroup to be bridged (vPG2 in Figure 8) does not need to be:
In addition to bridging between two Portgroups, if the VLAN ID of a portgroup is 4095, it acts as a SPAN port which receives all network packets on a given vSwitch.
VLAN Methods within Virtual Network
There are three 802.1q VLAN implementation methods within the vSwitch. They are defined by where the trunk of VLANs ends.
|External Switch Tagging (EST)||The Trunk ends at the pSwitch and each pNIC on the Virtualization Host has a single network cable representing a separate VLAN/Network|
|Virtual Switch Tagging (VST)||The Trunk ends at the vSwitch and each portgroup on the vSwitch represents a different VLAN|
|Virtual Guest Tagging (VGT)||The Trunk ends at a specific VM. This only works if the VLAN ID of the portgroup to which the VM is connected has a VLAN ID of 4095.|
Shared Storage is a requirement for the clustered attributes of VMware ESX/ESXi.
Local Storage is often required for Business Continuity reasons.
Storage for VMware ESX/ESXi implies where VMs may live aka Data Stores. Data Stores can live on:
Data Stores cannot live on:
Storage per Virtual Machine can be of these types:
Concerns with Storage
The major concerns when using storage is Disk Utilization which could translate into Network Utilization when using IP Storage.
Disk IO Utilization issues depend on the following:
|Disk IO within the VMs||As measured from outside the VM. While the in-VM numbers could match the out-VM numbers, the out-VM numbers depend on external clocks. Counters made within a VM expect a CPU to constantly be running within the VM, this is not necessarily the case.|
|Number of VMs per LUN||The standard thought is no more than 12 VMs per LUN. However, this may change if there are low utilization VMs or high utilization VMs.|
|Size of the LUN||The size of the LUN will determine the number of VMs that can fit per data store. The common thought is many smaller LUNs (600-700GB) instead of 1 large LUN.|
|Settings on the HBA||If you are using a FC-SAN it is important to have the proper Q-Depth number used for each LUN.|
|Number of Spindles used for each RAID Set||More spindles per RAID set imply better performance. However, for some SANs over 7 spindles has a performance decrease.|
|Type of RAID used||RAID-1 is faster than RAID-5 which is faster than RAID-6 (ADG)|
|Underlying Disk Type and Speeds||SSD is faster than FC, which is faster than SCSI, which is faster than SAS which has a higher mean time before failure (MTBF) than SATA.|
Higher speed drives are very important; try to use 15K drives. SSD can have Stutter so it is important to get high quality SSD devices.
SCSI Reservation Conflicts/p>
When using FC-SAN or iSCSI Servers the most important item to avoid is SCSI Reservation Conflicts.
SCSI Reservations occur when there are modifications to the metadata for any non-NFS data store.
Metadata is changed when a file on the data store is created, modified, accessed (access time), or deleted.
ESX alleviates this by limiting the number of simultaneous vMotions and Storage vMotions that are possible.
It is possible to perform more than 4 simultaneous actions that create SCSI Reservations. Actions include, but are not limited to:
The day to day operational issues will impact your virtualization host. These include the following tasks per VM:
|Anti-Virus Scanning||Full Disk scans should be staggered across all VMs within a VMware Cluster or Host. This is IO Intensive.|
|Spyware Scanning||Full Disk scans should be staggered across all VMs within a VMware Cluster or Host. This is Disk IO Intensive.|
|Vulnerability Scanning||Vulnerability Scanning without a VM implies using the Network and that could be Network IO intensive; from within a VM such a scan could be Disk IO Intensive.|
|Configuration Management||Initial computations of checksums for every file on a system could be CPU and Disk IO Intensive. Full Disk Scans could be Disk IO and CPU Intensive as well.|
|Disk Defragmentation||This is an IO intensive application and not recommended when using Linked Clones, as a Disk Defragmentation will touch every block of a disk, even those NOT in use. Thereby allocating the full disk and destroying the benefit of Linked Clones.|
|Patching within a VM||Patching within a VM has two issues. The first is that it can be Disk IO Intensive. The second is that patches often require a reboot which could cause Disk, Memory, and CPU issues if many VMs were to reboot at once. Stagger such reboots.|
|Patching an ESX/ESXi host||Without VMotion this task requires that VMs be powered off, moved off the host to be patched and then powered back on. For rolling updates without VMotion this could be many reboots.|
|Patching Storage Arrays||Without Storage VMotion, patching storage arrays requires powering off all VMs, migrating to other storage, and maybe powering the VMs back on. Use of Storage VMotion could alleviate the need for downtime, if there is a secondary data store available from another array.|
|Package Installation||Package Installation within a VM for many VMs at the same time could cause Disk IO issues. Stagger such package installations. Package Installation within a VMware ESX host does not require anything special.|
|Creation, Modification, Deletion of VMs||Creation of a VM has a Disk IO intensive component and will cause a SCSI Reservation to be requested. Once the VMDK is created, Disk IO issues cease. Modification of a VM may require powering off a VM and then powering it back on to gain the benefit of the change. This will cause a SCSI Reservation to be requested. Deletion of a VM could be Disk IO intensive for many VMs at the same time. This will cause a SCSI Reservation to be requested.|
Monitoring of VMs within an ESX host falls into different categories and will have different operational concerns. Here are some of the more prevalent concerns:
|Performance||Performance monitoring depends on CPU counters which depend on the full number of cycles within a VM's vCPU which is not always the case which implies that performance numbers from within a VM is more a gauge then true numbers. Gather performance data from without the VM.|
|State||The state of the VM, its network devices, etc. can be found within or without a VM.|
|Inventory/Assess Management||This would occur from within a VM.|
|Security||Security monitoring or auditing must take place from within the VM for Guest OS Hardening, and without the VM for VM specific security settings.|
Sphere APIs can be used to improve overall monitoring. The APIs are:
|vNetwork||Virtual Network API used by the Cisco Nexus 1000V switch. Network monitoring would occur at the vSwitch level and be performed without the VM.|
|vStorage||Virtual Storage API used by the EMC PowerPath VE product allows storage vendors to use the multipath plugin functionality to provide functionality and gather better disk IO statistics. Anti-virus and Anti-spyware vendors can also directly read a VMDK without the VM needing to be powered on.|
|vCompute||Virtual Compute API can be used to gather better performance numbers for individual VMs or vCPUs.|
|vMemory||Virtual Memory API will be used by Anti-Virus and Anti-Spyware vendors to quarantine reads and writes to parts of the memory of a VM.|
|VMsafe||VMsafe moves Security monitoring into the VMkernel. The most common use as of now is to provide virtual firewall at the vSwitch level and not as an inline device between virtual switches.|
Installing ESX is a 19 step process with only 1 step being the actual installation.
|File system||Minimal Size (MBs)|
At this time you have enough information to install VMware ESX or ESXi.
There are some post install steps to take as well:
Virtual Machines are comprised of the following virtual hardware devices:
|IDE Controllers||Used for IDE Disks and CD-ROMs. Up to 4 devices available.|
|IDE or SCSI Disks||Per the formats given under the storage subsection. Up to 4 IDE CD-ROM or Disks available. Up to 60 SCSI Devices available 4 controllers w/15 devices each.|
|SCSI Devices||Using SCSI Passthru or VMDirectPath to access attached SCSI devices like Tape Libraries, Tape Drives, and other SCSI devices. SCSI ID within VM should match SCSI ID of SCSI Passthru device.|
|IDE CD-ROMs||Up to 4 CD-ROM and IDE disks allowed.|
|Floppy||Up to 2 Floppy devices allowed.|
|Memory||Up to 64GBs available.|
|CPU||1, 2, 4, or 8 vCPUs available. 8 vCPUs requires Enterprise Plus licensing for vSphere.|
|Video Monitor||Memory for Video graphics adapter shared with Guest operating system assigned memory.|
|Serial Ports||Passthru to the Virtualization Host serial port using service console serial port pipeline.|
|USB Devices (vSphere Only, does NOT work in ESX/ESXI 4.0)||Passthru to the Virtualization Host USB devices.|
Virtual Machines also contain the VMware Backdoor
Devices not in the list of available devices can be connected to a VM through other means:
Any network method to transfer a device from one host to another host is supported over IP. This depends on the Guest OS not ESX or ESXi.
Security is incredibly important. Work with your Security Team.
Please follow one of these security guidelines:
It is also recommended that you read the following: