This Blog is to share our knowledge and expertise on Linux System Administration and VMware Administration

Tuesday, June 14, 2016

VSphere 6.0 important log files and its locations

Tuesday, June 14, 2016 0
VSphere 6.0 has made some significant changes to the logging locations for its contained vCenter and PSC services. Everything has been condensed into a common area of the directory structure and labeled with the service name. In short, it makes a LOT more sense now than it did before.

This is an overview of what the structure looks like now and where to find what you need.

Windows Log Locations


vCenter Appliance Log Locations


vCenter Service

Use this to troubleshoot issues with issues relating directly operation of the vCenter. Everything from DB connectivity problems to vCenter crashes are in here. This log will have a LOT of information in it and is a good place to start on many issues.

Inventory Service

Formally the ds.log in 5.x. The format and location has changed.
Used to troubleshoot why the inventory service will not start.

Single Sign on

This is a good log to use as a “one-stop-shop” for SSO authentication issues. Authentication requests/failures as well as problems with an identity source will post here. 

Contains installation errors during configuration of vmdir. Especially useful for errors when adding another PSC to the same SSO domain.

Has information concerning the SSO LDAP instance named vmdir. Problems with ldap operations and replication within SSO can be found here.

vPostgres Service

Operational information about the local vPostgres instance. This is just a renamed version on pg_log in normal Postgresql.

vSphere Web Client

An excellent source of information when troubleshooting errors within the Web Client. If you receive errors from simply clicking on objects, you begin chasing them down here!

Entries in here can help determine why your vSphere Web Client service won’t start, or if it suddenly crashes. This log will not have as much on issues received while inside the Web Client

VMware System and Hardware Health Manager

This service is used to poll ESXi hosts for IPMI information for the Hardware Status tab. Entries in here can determine why the service won’t start, is malfunctioning, or if it suddenly crashes.

Performance Charts

Has information on the Performance Charts section of the vCenter. If the charts fail to load, look here first.

Explain about vCenter operation times out with the error: Operation failed since another task is in progress

Tuesday, June 14, 2016 0

The default time-outs in VMware Infrastructure (VI) Client may not be long enough for certain long operations, such as deleting snapshots. This article provides information on how to prevent these timeouts. 

vCenter Server has a default 15 minute timeout for any task. Starting with the vCenter 2.5 Update 4 release, to prevent vSphere Client from displaying unnecessary timeout error messages, you can configure the timeout values by editing the vpxd.cfg file and the vpxa.cfg file of the source and destination ESXi/ESX host.
Note: If you are using VCB and your backup failed due to a timeout, check your virtual machine for a backup snapshot that has been left behind.

Lengthy Tasks which Time Out

When a task is reported to timeout within vCenter, the task may continue to run in the ESXi/ESX host level. Certain tasks (such as a snapshot consolidation) may take a long time to complete and should not be interrupted.
Note: In the case of snapshot consolidation, even though the vSphere Client timeout occurs, the operation on the ESXi/ESX host is still running. You can verify by observing the .vmdk file for the virtual machine. It is updated every minute which means the delta files are being committed to the .vmdk file.

vCenter Server Timeout Settings

To change the timeout value in the vCenter Server, update vpxd.cfg on vCenter and vpxa.cfg on the ESXi/ESX:
  1. Log in to the vCenter Server with the appropriate permissions.
  2. Open the vpxd.cfg file in a text editor. The default location for the file is:

    C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg
  3. For Windows 7 and Windows 2008, the default location for the file is:

    C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg
  4. To increase the timeout values for the virtual machine migration task, add the following timeout parameter in the vpxd.cfg file:


    Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
  5. To increase the SOAP layer blocking call timeout, add the following values in the vpxd.cfg file:

    Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours. This line may not be present in ESX 4.0.
  6. Restart the vCenter Server service. 

ESXi/ESX timeout settings

  1. Log in to the ESXi/ESX host as root via the console or an SSH session.
  2. Open the vpxa.cfg file in a text editor.

    By default, this file is located at:
    • ESX - /etc/opt/vmware/vpxa/vpxa.cfg
    • ESXi - /etc/vmware/vpxa/vpxa.cfg
  3. To increase the timeout values for the virtual machine migration task (both source and destination hosts), add the following timeout parameter in the vpxa.cfg file:

  4. To increase the SOAP layer blocking call timeout, add these values in the vpxa.cfg file :

    Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds or 3 hours.
  5. Configure the timeout value for the time that vCenter Server waits to capture the virtual machine's ID at ESX/ESXi destination. Add a new configurable parameter in the vpxa.cfg file:

    Note: The value 600 can be changed based on your requirements. This example uses 600 seconds or 10 minutes.
  6. Restart the vmware-vpxa service on the ESXi/ESX host. 

How to change the database password in Update Manager 4.1 Update 1, 5.x and 6.0

Tuesday, June 14, 2016 0

To change the database password in Update Manager 4.1 Update 1, 5.x and 6.0

    Navigate to the directory where Update Manager is installed. The default location is:
  •     C:\Program Files (x86)\VMware\Infrastructure\Update Manager\.
  •     Launch VMwareUpdateManagerUtility.exe.
  •     Use an account with administrator privileges on vCenter Server to log in to the utility.
  •     Click Database Settings.
  •     Type your new username and password in the appropriate fields.
  •     Click Apply.
  •     Close the VMware vCenter Update Manager utility.
  •     Restart the VMware vCenter Update Manager service.

Changing the vCenter Server database user ID and password

Tuesday, June 14, 2016 0

To change the vCenter Server user ID for SQL database connections for vCenter Server 5.x and earlier: 

Note: Before making any registry modifications, ensure that you have a current and valid backup of the registry.

    Take a full backup of the registry prior to editing it. Do not skip this step.
    Click Start > Run, type regedit and click OK.
    In the Windows Registry Editor, navigate to:
        HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter\DB (under My Computer)
        For 32-bit versions of vCenter Server running on 64-bit versions of Windows:

        HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\VMware, Inc.\VMware VirtualCenter\DB

For vCenter Server 5.0

        HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter\DB

        Note: To see these keys in a 32-bit version of the Registry Editor in a 64-bit operating system, click Start > Run, type %systemroot%\syswow64\regedit, and click OK.

    Right-click 2 and click Modify.
    Enter the database user ID in the Value data field.
    Click OK.

To change the vCenter Server user ID for SQL database connections for vCenter Server 6.0:

    Stop the vCenter Server service.
    Navigate to: C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx.
    Take a backup copy of vpxd.cfg.
    Open vpxd.cfg in a text editor.
    Locate the <DB> element and modify the value for <key_2> to reflect the new database user ID.     ...
    If you are using Windows Integrated Authentication to connect to the vCenter Server database, remove the value for <key_3>.
    Save vpxd.cfg.
    Start the vCenter Server service.

    Note: If the vCenter Server database password has changed, follow the steps below to update the vCenter Server database password before starting the vCenter Server service.

To update the password used by the vCenter Server for database connections to the SQL Database, use one of these options:

Follow these steps in VMware Infrastructure (VI) Client for VirtualCenter 2.5 only

        Click Administration > VirtualCenter Management Server Configuration.
        Click Database.
        In the Database Setting page, enter the new password in the Password field.
        Click OK.
    For VirtualCenter 2.5 Update 2 and later, the -p command line flag sets the database password in vCenter Server:
        Click Start, right-click Command Prompt, and select Run as administrator to open a command prompt as an administrator.
        Run this command: 

For vCenter Server 5.5 and earlier

        C:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe -p
        For vCenter Server 6.0:

        C:\Program Files\VMware\vCenter Server\vpxd\vpxd.exe -p

        Note:This is the default path to the vCenter Server installation directory. Change the path appropriately, if required.
        Enter a new password when prompted.

        Note: If changing any SQL authentication modes or credentials (for example, changing from SQL to Windows authentication), ensure that the ODBC System DSN utilized for the vCenter Server database connection is also updated to reflect the credential changes.

        Restart the vCenter Server service.

Stages of Linux Boot Process

Tuesday, June 14, 2016 0

The following are the 6 high level stages of a typical Linux boot process.


    BIOS stands for Basic Input/Output System
    Performs some system integrity checks
    Searches, loads, and executes the boot loader program.
    It looks for boot loader in floppy, cd-rom, or hard drive. You can press a key (typically F12 of F2, but it depends on your system) during the BIOS startup to change the boot sequence.
    Once the boot loader program is detected and loaded into the memory, BIOS gives the control to it.
    So, in simple terms BIOS loads and executes the MBR boot loader.

2. MBR

    MBR stands for Master Boot Record.
    It is located in the 1st sector of the bootable disk. Typically /dev/hda, or /dev/sda
    MBR is less than 512 bytes in size. This has three components 1) primary boot loader info in 1st 446 bytes 2) partition table info in next 64 bytes 3) mbr validation check in last 2 bytes.
    It contains information about GRUB (or LILO in old systems).
    So, in simple terms MBR loads and executes the GRUB boot loader.


    GRUB stands for Grand Unified Bootloader.
    If you have multiple kernel images installed on your system, you can choose which one to be executed.
    GRUB displays a splash screen, waits for few seconds, if you don’t enter anything, it loads the default kernel image as specified in the grub configuration file.
    GRUB has the knowledge of the filesystem (the older Linux loader LILO didn’t understand filesystem).
    Grub configuration file is /boot/grub/grub.conf (/etc/grub.conf is a link to this). The following is sample grub.conf of CentOS.

    title CentOS (2.6.18-194.el5PAE)
              root (hd0,0)
              kernel /boot/vmlinuz-2.6.18-194.el5PAE ro root=LABEL=/
              initrd /boot/initrd-2.6.18-194.el5PAE.img

    As you notice from the above info, it contains kernel and initrd image.
    So, in simple terms GRUB just loads and executes Kernel and initrd images.

4. Kernel

    Mounts the root file system as specified in the “root=” in grub.conf
    Kernel executes the /sbin/init program
    Since init was the 1st program to be executed by Linux Kernel, it has the process id (PID) of 1. Do a ‘ps -ef | grep init’ and check the pid.
    initrd stands for Initial RAM Disk.
    initrd is used by kernel as temporary root file system until kernel is booted and the real root file system is mounted. It also contains necessary drivers compiled inside, which helps it to access the hard drive partitions, and other hardware.

5. Init

    Looks at the /etc/inittab file to decide the Linux run level.
    Following are the available run levels
        0 – halt
        1 – Single user mode
        2 – Multiuser, without NFS
        3 – Full multiuser mode
        4 – unused
        5 – X11
        6 – reboot
    Init identifies the default initlevel from /etc/inittab and uses that to load all appropriate program.
    Execute ‘grep initdefault /etc/inittab’ on your system to identify the default run level
    If you want to get into trouble, you can set the default run level to 0 or 6. Since you know what 0 and 6 means, probably you might not do that.
    Typically you would set the default run level to either 3 or 5.

6. Runlevel programs

    When the Linux system is booting up, you might see various services getting started. For example, it might say “starting sendmail …. OK”. Those are the runlevel programs, executed from the run level directory as defined by your run level.
    Depending on your default init level setting, the system will execute the programs from one of the following directories.
        Run level 0 – /etc/rc.d/rc0.d/
        Run level 1 – /etc/rc.d/rc1.d/
        Run level 2 – /etc/rc.d/rc2.d/
        Run level 3 – /etc/rc.d/rc3.d/
        Run level 4 – /etc/rc.d/rc4.d/
        Run level 5 – /etc/rc.d/rc5.d/
        Run level 6 – /etc/rc.d/rc6.d/
    Please note that there are also symbolic links available for these directory under /etc directly. So, /etc/rc0.d is linked to /etc/rc.d/rc0.d.
    Under the /etc/rc.d/rc*.d/ directories, you would see programs that start with S and K.
    Programs starts with S are used during startup. S for startup.
    Programs starts with K are used during shutdown. K for kill.
    There are numbers right next to S and K in the program names. Those are the sequence number in which the programs should be started or killed.
    For example, S12syslog is to start the syslog deamon, which has the sequence number of 12. S80sendmail is to start the sendmail daemon, which has the sequence number of 80. So, syslog program will be started before sendmail.

That is what happens during the Linux boot process. Thanks for  Reading.

How Traceroute Works?

Tuesday, June 14, 2016 0
Traceroute utility uses the TTL field in the IP header to achieve its operation. For users who are new to TTL field, this field describes how much hops a particular packet will take while traveling on network.

So, this effectively outlines the lifetime of the packet on network. This field is usually set to 32 or 64. Each time the packet is held on an intermediate router, it decreases the TTL value by 1. When a router finds the TTL value of 1 in a received packet then that packet is not forwarded but instead discarded.

After discarding the packet, router sends an ICMP error message of “Time exceeded” back to the source from where packet generated. The ICMP packet that is sent back contains the IP address of the router.

So now it can be easily understood that traceroute operates by sending packets with TTL value starting from 1 and then incrementing by one each time. Each time a router receives the packet, it checks the TTL field, if TTL field is 1 then it discards the packet and sends the ICMP error packet containing its IP address and this is what traceroute requires. So traceroute incrementally fetches the IP of all the routers  between the source and the destination.

Tuesday, June 7, 2016

When up2date/yum fail with "Error Class Code 31" - Solved.

Tuesday, June 07, 2016 0
Whenever Running up2date or yum update fails with below error

Error Message    : Service not enabled for system profile: ""

Error Class Code: 31
Error Class Info   :This system does not have a valid entitlement for Red Hat Network.

    Please visit https://rhn-server/rhn/systems/SystemEntitlements. or 

    login at https://rhn-server, and from the "Overview" tab,
    select "Subscription Management" to enable Redhat Network service for this system.

  • System registration fails with above error.
  • Redhat Network entitlements missing after Redhat contract renewal.
  • After executing rhn_register, the system appears in host list, but as unentitled.
  • Cannot entitle system.
  • System does not have a valid entitlement for Red Hat Network.
  • When trying to install a package, an error was received that said the system does not have a valid entitlement.
  • No longer able to update system.
  • Satellite certificate activation is failing with "Error Class Code 31"?


If the system is not registered with rhn-server, follow the below steps to have an entitlement.

  • Log in to Satellite Customer Portal
  • Click on My Subscriptions
  • Under Redhat Network Classic select All Registered Systems
  • Click on system name
  • Click on Edit These Properties beside System Properties
  • Ensure either Update or Management is selected for Base Entitlement.
  • Click the Update Properties button located in the bottom-right corner.

Root Cause

  • Error Class Code: 31 means that a valid entitlement is not assigned to your system profile.
  • When you register a system, the base entitlement gets assigned to either Update / Management (as per the free entitlement in account) along with the base channel. But if the base entitlement is removed for the system profile then while updating the system it fails with Error Class Code: 31

Saturday, April 23, 2016

Find command with more example

Saturday, April 23, 2016 0
1. Find Files Using Name in Current Directory

[root@nsk nskselvan]# find . -name kal.txt

2. Find Files Under Home Directory

[root@nsk nskselvan]# find /home -name kal.txt

3. How to run the last executed find command?

[root@nsk nskselvan]# !find
find /home -name kal.txt

4. Find Files Using Name and Ignoring Case sensitive

[root@nsk nskselvan]# find /home -iname Kal.txt

5. Find Directories under / directory by using particular name.

[root@nsk nskselvan]# find / -type d -name kal

6. Find all txt Files in Directory

[root@nsk nskselvan]# find / -type f -name "*.txt"

7. Find Files With 777 Permissions

[root@nsk nskselvan]# find . -type f -perm 0777 -print

8. Find Files Without 777 Permissions

[root@nsk nskselvan]# find / -type f ! -perm 777 

9. Find Sticky Bit Files with 551 Permissions

[root@nsk nskselvan]#  find / -perm 1551

10. Find SUID Files

[root@nsk nskselvan]# find / -perm /u=s

11. Find SGID Files

[root@nsk nskselvan]# find / -perm /g+s


12. Find Read Only Files

[root@nsk nskselvan]# find / -perm /u=r 

13. Find Executable Files

[root@nsk nskselvan]# find / -perm /a=x | more

14. Find Files with 777 Permissions and Chmod to 644 (dont use untill not checked)

# find / -type f -perm 0777 -print -exec chmod 644 {} \;

15. Find Directories with 777 Permissions and Chmod to 755 (dont use untill not checked)

# find / -type d -perm 777 -print -exec chmod 755 {} \;

16. Find and remove single File  

# find . -type f -name "kal.txt" -exec rm -f {} \;

17. Find and remove Multiple File ( list before using rm command)

# find . -type f -name "*.txt" -exec rm -f {} \;

18. Find all Empty Files under /tmp Directory

[root@nsk nskselvan]# find /tmp -type f -empty

19.  Find all Empty Directories under /tmp Directory

[root@nsk nskselvan]#  find /tmp -type d -empty

20. File all Hidden Files 

[root@nsk nskselvan]# find / -type f -name ".*" | more

21. Find Single File Based on User

[root@nsk nskselvan]# find / -user nskselvan -name kal.txt

22. Find all Files Based on User

[root@nsk nskselvan]# find /  -user nskselvan


23. Find all Files Based on Group

[root@nsk nskselvan]# find / -group ntp


24. Find Particular Files of User

[root@nsk nskselvan]# find / -user nskselvan -iname "*.txt"

25. Find Last 5 Days Modified Files

[root@nsk nskselvan]# find / -mtime 5 

26. Find Last 5 Days Accessed Files

[root@nsk nskselvan]#  find / -atime 5 | more

27. Find Last 1-5 Days Modified Files

[root@nsk nskselvan]#  find / -mtime +1 -mtime -5

28. Find Changed Files in Last 1 Hour

[root@nsk nskselvan]# find / -cmin -60


29. Find Modified Files in Last 1 Hour

[root@nsk nskselvan]# find / -mmin -60


30. Find Accessed Files in Last 1 Hour

[root@nsk nskselvan]# find / -amin -60 | more

31. Find 5MB Files

[root@nsk nskselvan]# find / -size 5M


32. Find Size between 5MB – 7MB

[root@nsk nskselvan]# find / -size +5M -size -7M

33. Find and Delete 100MB Files (list before deleting the file)

# find / -size +100M -exec rm -rf {} \;

34. Find Specific Files and Delete (list before deleting the file)

# find / -type f -name *.mp3 -size +10M -exec rm {} \;