Painting yourself into a corner... VMware Style

Forgive me, I have not posted. Now I appear to have some time tonight. VMware used to be ESX.  Then it became ESXi then View (excuse me.. Horizon View) on top of ESXi.  You get the picture.

I was planning on patching 4 ESXi hosts used for VMware View..  I moved the VMs off, placed the host into maintenance mode and waited for the VMs to move.  Once in maintenance mode go to Update Manager and then go to the patches and apply. The host reboots and NORMALLY comes back. You exit maintenance mode and move on to the next host.  No muss, no fuss.

NOT THIS TIME. The host did not reconnect with vCenter. Was it down? I could ping it and connect to the host with a direct client connection. Now what?!?

Off to the Google for some suggestions. Apparently in VMware 5.5 you cannot get an ESXi host with Update 3 to connect to a vCenter with LESS than Update 3 on it also.

Now lets go to the remote console (iDRAC in this case) to remedy the situation. NO. I did not configure it.  This is not good...

Off to the customer site at 9PM.

Once on site at the customer, I did save my bacon with something else I came across in my searching. The "SHIFT+R". The SHIFT+R at the boot screen allows you to revert to the previous build. So I:

1. Booted
2. Shift+R
3. Said "Y" to confirm
4. Rebooted and all was good.

What did we learn:

1. RESEARCH the patches
2. HAVE THE iDRAC SET UP
3. KNOW YOUR ESCAPE ROUTE

P2V and the GPT Disk.. Part 2

In the previous post I had installed the Double Take - Move product and synchronized the servers. Then on 12/27 I was able to execute the fail-over to the VM. I logged into the software and it gave the option of using one of the "point in time" replications or using "live data". Since I wanted to make sure that the copy was the most current possible I chose the "live data" option. Double Take then shut down the source server, finalized the target server and then rebooted the target and then was done. It was almost anti-climactic. I then installed VMware Tools (which the mouse driver gave me an issue.. not sure why) and then turned it over to the customer. They logged into the server and checked. The databases were online and a user checked the application. All was operating as advertised. Thank you DoubleTake for making me look good!

P2V and the GPT Disk

For the better part of 2010 I have been working on performing P2V migrations. Taking a Physical server and turning it into a Virtual Machine for the unwashed masses. For the most part the process of turning a physical server into a VM is fairly painless.

However, I did most recently run across an issue with a server with a dreaded GPT (GUID Partition Table) boot disk and NONE of the P2V utilities that I would normally use would work. This GPT boot disk is a fairly recent phenomenon since it requires a EFI BIOS in order to understand what to do with it. To make matters worse the RAID group was 2095 or so GB in size (there was only 350GB used however).

As I stated previously, I did try everything I could think of in my arsenal. VMware Converter, Vizioncore vConverter, Novell PlateSpin.. NOTHING was working. I even thought of using Windows System Backup.

What was our superhero to do...

To the rescue came Vision Solutions - DoubleTake Move . It is a bit different in that it requires that there be a SOURCE and a TARGET rather than a copy of the SOURCE BECOMING the TARGET. So I loaded Windows 2008 onto the target and installed the software. I then installed the software onto the source server and created a mirror set... they connected and began to synchronize. After they are in sync you select "Failover" and the source server shuts down and the target is renamed to the source name. I have not done this part yet.. stay tuned for part 2 of our exciting adventure (This is where Max my dog would yawn).

Merry Christmas to all..

Long Time Coming

Ok.. So it has been a while since I have entered a post. It does not mean that I have not been doing anything. Quite the contrary. Now that I have had a chance to breathe I will begin entering some posts that will hopefully be of interest. They will probably be more on the lines of VMware ESX 4 and ESXi 4 since that is what I have been working with as of late. Without any further waste of time, here is the first post:

Setting up your own iSCSI...

The average person is probably thinking "What is iSCSI and why would I care" but the VMware folks in our midst are saying "Tell me more...". I needed to set up a VMware ESX environment at home to test ESXi 4 and some other VMware products. Working on a budget, I sourced a used Dell PE2850 and downloaded the free ESXi from VMware. Now back to the budget. I managed to get the server with 2 - 18GB disks but I did not have much in the way of money left for additional space. So out with the iSCSI solution. Since ESXi will connect to iSCSI over a normal Ethernet network I could add an larger SATA drive to a normal PC and then present it via the iSCSI software to the ESXi host and BANG.. more ESXi storage.

Can I speak with root?

Once again... this is not a very exciting post. Unless you are having to re-install a Linux system because you do not have the root password, then it could be like sex. Maybe just a piece of candy but you get the idea.

First get the installation media for said system. In this example it is a Red Hat 4.6 or 5.1 system. Then perform the following:
1. Boot from the media. When it comes up with the installation prompt type "linux rescue"
2. The system will continue to boot and ask about the network interface - it is not needed
3. When it asks about mounting the installation allow it to mount with read-write

Now for the fun part.. ok as fun as it gets

4. The hard drive installation will be under /mnt/sysimage
5. Change to /mnt/sysimage/etc and then edit shadow with vi
6. Find the root entry in the shadow file it will look like:

root:Ep6mckrOLChF.:10063:0:99999:7:::

7. Change the line in the file to look like the following:

root::10063:0:99999:7:::

8. Save and exit vi, and then reboot the system
9. Voila, no root password.
10. Log in as root and set the password with the passwd command ASAP!

What to expect....

Ok when it comes to programming ANYTHING I am pretty slow. I needed to change the firewall ruleset on quite a few ESX servers and it seemed like a script was in order to make this change for me. However, since I could not get the Opsware client to run a script remotely (that was part of the problem) i needed to figure out an alternative method.

Enter EXPECT.. I had heard it mentioned before but not really understood what it did. Basically you can open a connection to another system and if you can tell the script what to "expect" then it will know what to send it in return.. pretty cool huh.. This is the part where my wife's eyes glaze over.

Basically what I was doing is this...

spawn ssh some_user@somemachine.com
expect "someuse@somemachine's password:"
send password\r
expect "someuser@somesmachine"

And so on.. you get the idea. BUT.. is there not always a but... the issue I was having was that the prompt was on a VMware ESX server and looked like [usename@hostname username]$. I entered the text verbatim so that the expect showed the brackets etc. It puked. But by removing the brackets and only going to the end of the first word it works. Example..
expect "someuser@hostname" works but expect "[usename@hostname username]$" does not. Go figure. So I am going to stop while I am ahead today and revel in my glory....

Just rember... You Stay Classy San Diego.

Backup a minute..

Ok.. another yawner of a post but hey that is me. I am working on process of backing up the VM's in an environment and making it as quick to recover as possible. In my search I ran across a product called esXpress v3.1. It is pretty cool (ok as cool as VM software can be). It does a couple things:

1. It will put a VM into "Snapshot Mode" and then perform a backup of the VMDK.
2. It can put the VMDK backup to SMB/FTP or SSH.
3. After it is done it will remove the snapshot.

THIS IS DONE WHILE THE VM IS RUNNING! HOW COOL IS THAT!

Oh and I did not mention the best part. The software is FREE! Now there are additonal features such as being able to make backups local and remote at the same time, having additional helper machines, encryption and other things.

Check it out at www.esxpress.com

My Friend CRON...

OK another dry post but hey.. this is not the Life and Times of Someone Exciting.. sorry. I was at work minding my own business when I realized that my little automated email was not there waiting in my inbox from several servers on Mondays when I came in. (Maybe it had a "Case of the Mundays" as described in the movie Office Space) Computers are like Marines. They are not supposed to choose what orders they follow and what ones they do not. I began to investigate.


Looking into the cron log (/var/log/cron) I noticed that NONE of the CRON jobs were running. I am on to something. I looked into the messages log and sure enough there was an entry "Authentication token is not longer valid; new one required". Clear as mud, of course. So I went to the worlds User Manual.. Google. It showed that there was an issue with the root user id. Huh? I just used it. Logging into the console with the root user id, I was greeted with a message that the id was aged out and would have to be changed! I DID NOT GET THIS WHEN I SU to the ROOT ACCOUNT! I changed the password and restarted the crond service. All good now..

As usual, I hope someone benefits from my stupidity.

Don't Go Away Mad...

Ok, it has been a little while since the last post, but hey, I don't create magic that often (never if you ask my wife).

One of the things we were dealing with is an automated flow that deleted our Virtual Machines. This is a good thing if you want the VM's gone but in our case we kind of wanted them there but they kept getting deleted. Needless to say this was NOT a good thing. I was able to come up with a solution. We modified the delete script to a rename script.

Here is how it works....

The script was using "rm vm_name" and with a little trial and error I came up with...



mv vm_name vm_name.`date +"%Y%m%d"`_`date +"%H%m"`

Now that seems complex but it moves (hence the mv command" from the vm_name directory to vm_name.yearmonthday_hourminute .. for example a VM named vm01 decomissioned on December 19th at 11:15 am would be...

vm01.20081219_1115

Now it it would still be resident on the disk but it would show when it was decomissioned. This will help if you want to keep VM's around for 30 days or so before deleting them.

Also some other steps were not covered.. the vmware-cmd was used for all the shutdown and unregistering of the VM's..

Hope this helps somone..

Perl and the Array..

Ok this is not going to be much fun for the average person but I found it helpful. Figured I would put it out here for everyone to ponder.. much like the meaning of life, why are we here, how my belly button fills with lint. The REALLY important stuff.

I was working on a small script ( I really do not call it a program). In this script, I needed to find the most recent log file on a Linux box and do something with it. How I accomplished this was a rather cumbersome Perl script but here goes:

1. I was able to sort the data by using "ls -c". This put the newest log file on top.
2. The I was able to pipe that data into an array. If you are not familiar with an array, it stores information in separate entries kind of like of a database.
3. Since I wanted the first entry in the array I was able to pull it using the following code:

@item_wanted = $array_i_am_using[0];

Then I can print the array item out if needed etc.

Hope this helps someone.

I might be reading this later in the future :)

So this would make it easier..

I am in the process of creating a script that will look @ a text file (from and AV scan in this case )and then email a portion of it to someone. We have five Red Hat servers and in the time it has taken for me to beat this into submission I could MANUALLY check AV scan file.

However, I digress. In the process I am learning some PERL, a little HTML and a program called sendEmail-v1.55.tar.gz. This program looks like a god-send and will come in handy in the future I can see.

The broad strokes of the process is this:
1. AV scans system (I know scanning for Viruses on a Linux box .. don't get me started)
2. Perl script runs and parses the AV scan log and then creates an HTML file.
3. sendEmail uses the HTML file as the body of the email to the admin folks to allow them to quickly see if there is a problem or not.

The sendEmail program will communicate directly to an SMTP server (if you do not know what this is... might want to read up a little bit) and it is all done @ a command line and easy to script.

Well all for now. Hope everyones Thanksgiving goes well.

Another Day Another Patch..

Life in Missouri is always great on the health. (no not really) The weather likes to change quickly and the mold count is always high. So right now I have a head cold. It makes this work more challenging that it should.

Yesterday I set up a Web Server to hold the ESX patches. Once the ESX-1003525 (for version 3.0.2) is loaded the rollup packages work pretty great. Here is a crib notes version on how to set this up.

1. Take your everyday Linux build and load Apache.
2. Once Apache is loaded, find the directory that the web content goes into. (/var/www/html for RedHat EL 5)
3. Create a sub directory for the ESX patches. I used the crazy name of "esx"
4. Edit to the /etc/httpd/conf/httpd.conf file.. Find the section "Listen" and change the port after Listen to something you like.. I used 8080 so that it is not as vunerable.
5. After saving the httpd.conf file, type "service httpd restart" to restart Apache
6. You should now be able to open a web browser to the http://my_web_server:8080/esx directory.
7. Use WinSCP or something else to move the packages to the esx directory on the web server.
8. Un-TAR the patch files
9. Check the connection from the web browser again, you should be able to see the patch files.

Now on the ESX server to be patched:
1. Open the ESX firewall type "esxcfg-firewall -o port_number,tcp,out,httpConnection" this will open the firewall to allow the esxupdate to connect to the web server.
2. Now type "esxupdate -d http://my_web_server:8080/esx/ESX302-Rollup-01 update"
3. The esxupdate will check for the contents.xml file in the directory and will install all the files listed in it.

Note: More details on the esxupdate command can be found by typing esxupdate without anything else.

Hope this helps someone!

Time for Maintenance

ESX Maintenance Mode.. this mode is important for applying the all important updates. This can be done easily via Virtual Center but in the event you need to patch oh I don't know.. 40+ ESX hosts in an automated fashon this can be a pain in the rear.

My new best friend is VIMSH. The vimsh can be used to process many thing on the command line of the console of an ESX server. The details can be found in this doc..Here.



An example of the code I am going to use is:



vimsh -n -e hostsvc/maintenance_mode_enter

vimsh -n -e hostsvc/maintenance_mode_exit



Running vimsh --help shows that the "-n" runs the command in non-interactive mode and the "-e" runs the commands specifed. THIS IS COOL! Now I can set up the Opsware client to place an ESX host into maintence mode, patch it, reboot it, and then exit maintence mode.



Whoo Hoo! Its a great day to be a Geek!

Software Upgrade

Last night was the "Mans Night" at the Pub. I have not missed Mans Night in MONTHS. However, I was at work last night working on a "Software Upgrade". Working at a Government facility which requires 15 character passwords and changing from one network to another is cumbersome to say the least. So back to the upgrade. It takes 15 min to download, I have to burn a CD, transfer it to the server and then walk back to the office to begin the install process.

I begin the upgrade. It is on Tomcat. It is not working. They suggest D/L the file again. I say "NO". I checked the permissions on the D/L file that Tomcat was supposed to be working with. It is set to ROOT only. Duh. How can the Tomcat Service get to the file.
I change the permissions and restart Tomcat and Pow.. it works.

I am cutting out much cussing and other things that took place. I left @ 12:10 in the morning.

Needless to say, I will make sure I check the permissions from now on.

That is my tip for the day.

Back at the Salt Mine..

Well, I was out of circulation for a while.. and we won't go into that now. I have been back @ work for 2 weeks now. I am working for a small computer company (HP..joke get it.. they are not small) as a consultant on a Government Project. That is all I can say about that.

However, I have been working with OpsWare, VMware and Linux (2 of the 3 are new to me) and figured I would put down some thoughts as I come across them.

YUM - yum is another package manager in the Linux world. I am not sure when it came into existance or where from but it is cool! I am working on a project where we wanted to install all packages with the exception of BIND and SENDMAIL. Ok.. I try to remove the BIND with rpm..nothing doing. There are all kinds of other packages that rely on it. However you can go "yum -y remove sendmail" and BAM, it removes all the associated packages.

In the words of Ricky Bobby "That Just Happened!"

I am able to move on for the day.