Cisco UCS Blade Bios Configuration

I have been working on a BIOS policy for our recently purchased UCS B250-M2 units so I can take advantage of the Cisco Fabric and VMware ESXi 4.1 U1. Today I finally completed the policy which we will use moving forward. I will post the PDF of the settings I used very soon.

I came across an excellent Cisco Document with all of the settings and descriptions. You might find this help in the future.

NOTE: Cisco has advised that C6 and C3 combinations may cause problems. I found this also on an IBM site with a better description of the problem:

IBM Support On a subset of Intel Xeon 5600 series processors, during package C6 (ACPI C3) transitions, an internal circuit marginality may cause JEDEC DDR3 specification violations. These violations may result in control and data signal errors upon exiting from package C6 state. This could create an invalid DRAM state, system hang, reboot, and memory ECC errors. For systems with ECC memory, correctable/uncorrectable ECC errors may be logged in the IA32_MC8_Status MSR with memory errors resulting in a machine check exception.

NOTE: The ACPI C3 setting in SETUP equates to the Intel processor C6 state. This issue affects only the Intel Xeon 5600 series processors in systems that are running with ACPI C3-state enabled, and the memory errors reported in run-time. Memory failures reported in POST are not affected, and should be treated as the normal memory fallout.

Basically turn off C3 and C6 on Cisco B250-M2 units, it is not just Cisco with this problem.

vSphere 4.1 Linux Tools on Ubuntu Server 10.4

I use the server version of Ubuntu for a lot of projects. I prefer to use the non-graphical version since the resources are low. This allows me to keep my environment secure and overhead low. Most people can easily install the Linux tools in the graphical version by using a mouse. When it comes to the terminal version (SSH) they will find that it is not that easy for a newbie. Here is a very quick set of directions for making this work. Please note that I am doing this on a fresh machine and one that if something goes wrong I can simply delete the machine and start over. Also to note that I typically use the VMNET3 drivers which install perfectly with this version out of the box. Even Windows can do that.

Let's get started. I am assuming that you have setup your Ubuntu 64 or 32 properly. When you created the machine you chose the correct version of the OS. Why is this important? Well the tools are mounted to the machine based on the choice at the beginning of creation. Let's say you said Solaris Unix and then install Ubuntu. When the tools mount they will provide your virtual machine with Solaris tools and not linux tools. Word to the wise is choose the closest description that matches your machine. Since you did this right I will continue.

Step 1. Once you have powered on the virtual machine and have the prompt. Right click on the machine on the inventory to the left. Then goto Guest > Install/Upgrade VMware Tools.

Step 2. Now log into the machine. Now I typically do these command as root but you can also do "sudo" as long as you have the rights to run sudo. Again it might be easier just to log in as root just for this exercise. Copy the below line into the terminal window and then press enter.

mkdir -p /mnt/cdrom && mount -t iso9660 -o ro /dev/cdrom /mnt/cdrom

Step 3. Nothing happens? Well it did, remember your in the terminal window but the system did/should have looked for the VMware Linux tools and mounted them in the proper directory. Now we have to go and copy the tools to a folder and begin the install.

Step 4. Navigate to the tools. Type the below command

cd /mnt/cdrom

Note: You should be able to now type "ls" to see what files are in this directory.

Step 5. Now copy the tools from the CD ROM to a folder so that we can begin to install.

cp VMwareTools-8.3.2-299420.tar.gz /home/USERNAME

Note: USERNAME is a folder that you probably created when you setup Ubuntu so replace this with your username

Step 6. UnTar the file to begin the install.

tar -xzf VMwareTools-8.3.2-299420.tar.gz

Step 7. Run the installer

cd /home/USERNAME/vmware-tools-distrib
./vmware-install.pl

Step 8. Check vCenter/ESXi. Once the tools have been installed your vCenter or VI Client should show information about the virtual machine like IP, hostname, etc. If not then try rebooting. If it still doesn't work then reply to this post.

Why should I have VMware tools? Your machine will run better, vMotion better and also backup better with the ability for vCenter to directly communicate with the virtual machine, good luck and hope this helps.

ESXi 4.x host network config scripts

I have been working on a bunch of iSCSI ESXi host installs and this information has proven very valuable. Time to share with everyone else.

Using Putty or terminal log into your ESX/ESXi 4.x host and run the following commands. Read the descriptions before you run the commands

NOTE: I ran these commands after I install and setup the iSCSI switch in order verify MTU 9000 (Jumbo Frames)

To list all the kernels and information

esxcfg-vmknic -l

Change the MTU of the kernel

esxcfg-vmknic -m 9000 KERNEL_NAME

Check to make sure everything work as you wanted.

esxcfg-vmknic -l

List all of the vSwitches

esxcfg-vswitch -l

Change the MTU of the vSwitch

esxcfg-vswitch -m 9000 SWITCH_NAME

List the switch to verify changes were made.

esxcfg-vswitch -l

NOTE: even though everything looks correct I usually reboot the server so that it loads all o the changes on boot, you might find it helpful to recheck if problems arise later.

If you want to check the MTU on everything run the following:

vmkping -s 8000 <ip address of iSCSI storage>

This checks the whole path is working with jumbo frames (9000 will not work as a few bits are added making it over 9000)

ESXi 4.1 + Dell iSCSI Mem Plugin

I just recently installed a new UCS B250-M2 into the environment and since it was a fresh install we decided to jump into ESXi 4.1 instead of the discontinued ESX 4.1. I am very familiar with ESXi as I have been working with it for many years and have a very fluent Linux/Unix background. Anyways, this was extremely easy. The below information should work for any server but best to always take precautions.

Server: Cisco UCS B250-M2 (firmware 1.4.2b)
Network: Jumbo frames (MTU 9000) end to end + (4) 10GB Storage NICS
Storage: EqualLogic (4) nodes

Step 1. Before you begin make sure that you have the following pieces of  information + software:

1) Dell EqualLogic Mem Plugin from EqualLogic.com, I used version 1.0.1.
2) VMware ESXi 4.1 clean install. This can be done to an existing host but my directions are for an unconfigured ESXi 4.1 machine with (4) unassigned physical nics.
3) (4) ip addresses on the same network as the Dell Storage Array.
4) VMware vMA (vSphere 4.1 Management Assistant) which can be obtained from VMware's site

Step 2. Once you have the files on your desktop from Dell unzip the single file and you will find multiple files. We need to move (2) files from your desktop to the vMA virtual machine. You can use WindowsSCP or Unix/Linux SCP. Here is an example of the transfer script:

scp setup.pl username@x.x.x.x:/~
scp dell-eql-mem-1.0.1.168651.zip username@x.x.x.x:/~

This will move the setup.pl and Dell mem file to the vMA in order to run the script.

Step 3. Run this script from the vMA which will configure everything for you. I kept open the VI Client the entire time while the script was building. It updated everything on the fly.

./setup.pl --server=192.168.125.210 --configure --ips=10.0.0.190,10.0.0.191,10.0.0.192,10.0.0.193 --nics=vmnic3,vmnic4,vmnic5,vmnic6 --mtu=9000 --enableswiscsi

192.168.125.210 = ESXi 4.1 Server
10.0.0.190~10.0.0.193 = The iSCSI IPs for storage are network
vmnic3~vmnic6 = These are NICs available for storage area network

Now check everything making sure it is running properly. I am going to post some scripts for checking the information on the Kernel and vSwitch.

UPDATE: 5/25/2011
So now that I am testing all of the settings I noticed that this script above is just to configure the server and not to actually install the MEM Cache here is the command to install it. Path will be the same as above

NOTE: You will need to REBOOT this server so be prepared.

./setup.pl --install --server=x.x.x.x

Once you have everything completed you can check the mem Cache by running this command.

./setup.pl --listparam --server=x.x.x.x

To Enable the plugin

./setup.pl --enable --server=x.x.x.x

To Disable the plugin

./setup.pl --disable --server=x.x.x.x

Ubuntu Wireless Driver Fix

During Spring break I helped out our friends by installing Ubuntu 10.x Netbook Edition on some Dell Minis. One of the laptops was hit with a really bad virus. I ran into a problem with the wireless card not working. Here is how I got it working.

NOTE: You will need an internet cable to connect to the laptop until the wireless card is working.

Open the Terminal application and type the following:

sudo apt-get update

This will update the repositories and tell your system where to get the drivers from.

Now type:

sudo apt-get --reinstall install bcmwl-kernel-source

This will install the driver.

Now reboot the laptop and try and check the system to see if you can connect to the internet with the wireless.

VCAP-DCA Information

As I said in a previous post I will be working on my VCAP-DA. Here are some great pieces of information to help you prepare and I will post more information as I continue my voyage to the prize.

TheSaffaGeek Prep Page: Link

VMware's Introduction to how the test works: Link

ESX/ESXi Disabling Balloon Driver

This is not recommended by VMware and in fact they removed the ability to do this through the VI Client. In case you really need this turned on leave it automatic.

To disable the balloon driver in an ESX host:

1. Access the ESX host on which the virtual machine resides using the vSphere Client connected to the ESX host or vCenter Server.
2. Log in to the ESX Server host as a user with administrative rights.
3. Shut down the virtual machine.
4. Right-click on the virtual machine listed on the Inventory panel and click Edit Settings.
5. Click the Options tab and select General.
6. Click Configuration Parameters.
7. Click Add row and add the parameter sched.mem.maxmemctl in the text box.
8. Click on the row next to it and add 0 in the text box.
9. Click OK to save changes.

Note: To re-enable the balloon driver, perform the preceding steps and remove the sched.mem.maxmemctl parameter.

VMware vMA console

I am getting ready to take the VCAP-DCA and while in a performance class we were using the vMA to get information from ESX/ESXi. I will post a bunch of information on how to use this tool but in the mean time download it and put it in your infrastructure.

VMware's official page for : vSphere Management Assistant

Additional information for how to interrupt the information from ESX/rESXtop : Community

VMware vCMA for iPad

I was working on getting my iPad vCenter client working and wanted to post some important information about the app and virtual machine which needs to be added to the environment in order to work. The directions are also listed in the first posting for getting it running.

http://labs.vmware.com/flings/vcma