This Blog is to share our knowledge and expertise on Linux System Administration and VMware Administration

Monday, November 27, 2017

Integrated Lights Out Manager ILOM Command line Linvirtshell

Monday, November 27, 2017 0

Integrated Lights Out Manager ILOM Command line

Login with ILOM root password
[root@testserver ~]# ssh test-ilom.example.com
Oracle(R) Integrated Lights Out Manager
Version 3.2.8.25 r114493
Copyright (c) 2016, Oracle and/or its affiliates. All rights reserved.
Warning: HTTPS certificate is set to factory default.
Hostname: test-ilom

Below targets are available to check in console. Once you run ls command, will get the property information of hardware. In that property we can get ILOM IP address, MAC ID, Server Model, Serial No & Operating system detail as well.

-> ls

 /System
    Targets:
        Open_Problems (0)
        Processors
        Memory
        Power
        Cooling
        Storage
        Networking
        PCI_Devices
        Firmware
        BIOS
        Log
    Properties:
        health = OK
        health_details = -
        open_problems_count = 0
        type = Rack Mount
        system_fw_version = 3.2.8.25
        locator_indicator = Off
        power_state = On
        actual_power_consumption = 305 watts
        action = (none)

a. To check power supply status
-> cd /System/power
/System/Power

-> show

 /System/Power
    Targets:
        Power_Supplies

    Properties:
        health = OK
        health_details = -
        actual_power_consumption = 383 watts
        max_permitted_power = 788 watts
        installed_power_supplies = 2
        max_power_supplies = 2

->
Login as root & run ipmitool command.
[root@test-ilom~]# ipmitool sunoem cli
Connected. Use ^D to exit.
-> show -l all /System/Power/Power_Supplies location health -t
Target             | Property              | Value
-------------------+-----------------------+-----------------------------------
/System/Power/     | health                | OK
Power_Supplies/   |                       |
Power_Supply_0   |                       |
/System/Power/     | location              | PS0 (Power Supply 0)
Power_Supplies/   |                       |
Power_Supply_0   |                       |
/System/Power/     | health                | OK
Power_Supplies/   |                       |
Power_Supply_1   |                       |
/System/Power/     | location              | PS1 (Power Supply 1)
Power_Supplies/   |                       |
Power_Supply_1   |                       

b. To check Processors status
-> cd Processors
/System/Processors

-> show

 /System/Processors
    Targets:
        CPUs

    Properties:
        health = OK
        health_details = -
        architecture = x86 64-bit
        summary_description = Two Intel Xeon Processor E5 Series
        installed_cpus = 2
        max_cpus = 2

c. To check BIOS version
-> cd BIOS
/System/BIOS

-> show

 /System/BIOS
    Targets:
        Config

    Properties:
        system_bios_version = 17120100
        boot_mode = Legacy
        reset_to_defaults = none

Sunday, November 26, 2017

Ignore first 4 lines of cat command output and display first word of remaining lines Unix

Sunday, November 26, 2017 0

Below awk command is very useful to run small scripts in Unix.

Ignore first 4 lines of cat command output and display first word of remaining lines  Unix

Purpose Ignore first 4 lines of cat command output and display first word of remaining lines.

[root@nsk-linux tmp]# cat test.txt
Public cloud is future
Automation never ends
Infra as a Service
Private Cloud
Baremetal Cloud Service
Amazon Cloud service

[root@nsk-linux tmp]# cat test.txt |  awk '{if(NR>4)print $1}'
Baremetal
Amazon

Here, as per our need we can change the value.

Saturday, November 25, 2017

Virtual machines show warning messages when starting the udev daemon Linux

Saturday, November 25, 2017 0

Virtual machines show warning messages when starting the udev daemon.

After upgrading VMware Tools,  Linux virtual machines show warnings when starting the udev daemon.

dmesg shows the below messages.

Starting udev:
udevd[572]: add_to_rules: unknown key 'SUBSYSTEMS'
udevd[572]: add_to_rules: unknown key 'ATTRS{vendor}'
udevd[572]: add_to_rules: unknown key 'ATTRS{model}'
udevd[572]: add_to_rules: unknown key 'SUBSYSTEMS'
udevd[572]: add_to_rules: unknown key 'ATTRS{vendor}'
udevd[572]: add_to_rules: unknown key 'ATTRS{model}'

Ctrl+C will bypass udev daemon to finish the boot process.

To disable the warning message, comment out unused lines (ubuntu  & other type of unix entries) in the  /etc/udev/rules.d/99-vmware-scsi-udev.rule file

For linux we need to modify the below line from

ACTION=="add", BUS=="scsi", SYSFS{vendor}=="VMware, " , SYSFS{model}=="VMware Virtual S", RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"

To

ACTION=="add", BUS=="scsi", SYSFS{vendor}=="VMware " , SYSFS{model}=="Virtual disk ", RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"

Save the modifiation and reboot the virtual machine.

Friday, November 24, 2017

Phpldapadmin Renew the expired openldap user

Friday, November 24, 2017 0

Phpldapadmin  Renew the expired openldap user 

Login to phpldapadmin admin console by using Admin id & password.
Explore the ldap directory & search the user in left side
Select the user and click
Right side we can see the user property.
Update the password directly in password section.

Before changing the Shadow Last Change, the ShadowMax & ShadowWarning settings will be like below 


Update the date in Shadow change as per your policy.

 
Once changed the ShadowLastChange setting to current date, ShadowMax & ShadowWarning will change.

Note: Once you changed the password & shadow Last Change settings click update objects. Then only above change will take effect.

Update or Recover root password using Host Profiles VMware

Friday, November 24, 2017 0
VMware Update or Recover root password using Host Profiles.
Create the Host Profile


1. Right click any host in your inventory (it doesn’t matter which), choose Host Profile, then Create Profile from Host.


2. Name the Host Profile and give it a description. Click Next.

3. Click Finish to create the Host Profile.

Configure the Host Profile
I. Open Host Profiles.


II. Right-click the Host Profile and choose Edit Profile.

III. Expand Security Configuration, choose Administrator password
    Choose Configure a fixed administrator password, type in the password twice, then click OK.


IV. Right-click the Host Profile and choose Enable/Disable Profile Configuration.

V. Uncheck everything in the Host Profile configuration except the Security Configuration, click OK.

a. Apply the Host Profile to the ESXi Host (or vSphere Cluster)

b. Right click any host in your inventory, choose Host Profile, then Manage Profile.

c. Choose the Password_Reset Host Profile, click OK.

d. Right click any host in your inventory, choose Host Profile, then Apply Profile.

e. Recheck the changes to be applied, click Finish

Log In
     Log in with the new password. If the setting has been changed successfully, you should be able to log in with the new password immediately.


Thursday, November 23, 2017

for loop command for grep the particular file system from list of servers.

Thursday, November 23, 2017 0

Double For loop command for grep the particular file system (mount point) from list of servers.

Here,
 servers       -  contains server list
 fmount        -  mount point

Run the below command. It will login to the list of server and search the mount point and print the output.

#for j in `cat servers`; do for i in `cat fmount`; do echo $j; ssh $j df -hP | grep -i $i; done; done

Modify the variable as per your needs.

Openldap user password reset by using Phpldapadmin

Thursday, November 23, 2017 0
Follow the below steps to Reset Openldap User password by using Phpldapadmin

Login to phpldapadmin admin console by using Admin id & password.

Explore the ldap directory & search the user in left side
Select the user and click
Right side we can see the user property.


Update the password directly here.

Finally update the object.

Tuesday, November 21, 2017

LVM Interview questions and answers Linvirtshell

Tuesday, November 21, 2017 0
1.Is it possible to increase the logical volume?

 Yes. LVM has the feature to increase the volume without unmount it.


2.How to reduce the logical volume? is it possible to reduce an online?

No. we can't reduce the logical volume an online. Here is the steps to reduce the logical volume.

Un-mount the filesystem
Run e2fsck on the volume device
Reduce the Filesystem using resize2fs
Reduce the logical Volume using lvreduce
Mount the filesystem back.


3.How do you scan the new LUN or disk for LVM physical volume?

Use pvscan to scan existing physical volume from newly connected SAN or DISKS.


4.How to scan disks for existing volume group?

vgscan is used to scan existing volume group from newly connected SAN or DISKS.
But we should use pvscan prior to executing this command.


5.How to scan a logical volume from exising volume group?

lvscan command is used to scan.


6.How to stop the logical volume? or deactivate the logical volume?

#lvchange -an /dev/vg_name/lv_name

-a  - activate
 n   - no


7.How to activate the logical volume which is in deactivated state?

#lvchange -ay /dev/vg_name/lv_name

-a - activate
 y - yes


8.How to disable the volume group? or Deactivate the volume group?

#vgchange -an volume_group_name

-a  - activate
 n   - no

9.How to enable the volume group? or Activate the volume group?

#vgchange -ay volume_group_name

-a - activate
 y - yes

10.How do you find, what are the disks are used for logical volume mirroring?
 

[root@testserver archive]# lvs -a -o +devices
  LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert Devices
  LogVol00 VolGroup00 -wi-ao 32.00G                                       /dev/xvda2(0)
  LogVol01 VolGroup00 -wi-ao 17.88G                                       /dev/xvda2(1024)




11. What are steps to perform in order to increase the logical volume?

  Extend the logical volume
   Increase the Filesystem size
   Verify the status using df command or lvs command.

12.How to list the imported volume groups?

 vgs command to display the imported volume group.


13.How to list the available logical volumes on the system?

  lvs command to list the available logical volumes on the system.

14.How to list the available physical volumes in LVM?

  pvs command to list the available physical volumes.


15.How to see the detailed volume group information?

  [root@testserver archive]# vgdisplay VolGroup00
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                     lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access               read/write
  VG Status                resizable
  MAX LV                    0
  Cur LV                      2
  Open LV                   2
  Max PV                    0
  Cur PV                     1
  Act PV                      1
  VG Size                49.88 GB
  PE Size                32.00 MB
  Total PE                 1596
  Alloc PE / Size       1596 / 49.88 GB
  Free  PE / Size       0 / 0
  VG UUID               ZfQCQ1-suTc-ykV9-TwvN-ACpB-XcEM-NuWlnE

  

16.How to see the detailed logical volume information?

 #lvdisplay /dev/centos/root

17.How to see the detailed physical volume information?

  pvdisplay /dev/disk_name

18.How to rename volume Group? can we rename the VG an online?

Its possible to rename the volume group an online. But the mounted volumes will not reflect the same unless you re-mount the volume with new VG name. Need to update the /etc/fstab with new VG name.

19.How to take a LVM configuration backup?

 vgcfgbackup vg_name  to take the latest configuration backup of volume group. The default volume group backup location is  /etc/lvm/backup.

20.How to re-create the device files for LVM volumes?

 vgmknodes to recreate the LVM devices files

21.Brief about lvmdump.

 lvmdump is tool for LVM2 to collect the various information for diagnostic purposes.By default, it creates a tarball suitable for submission along with a problem report

22.How are snapshots in LVM2 different from LVM1 in Redhat Linux?

LVM1 snapshots are readonly by default where LVM2 snapshots were read/write.
 

23.What are the steps involved to create the logical volume from scratch?
 

Create a physical volume using pvcreate command.
   #pvcreate /dev/sdc

Create a volume group using "vgcreate" command
   #vgcreate centos /dev/sdc

Create a logical volume using "lvcreate" command
   #lvcreate -L 100M -n vol1 centsos

Create a filesystem on logical volume using mkfs command.
    #mkfs -t ext4 /dev/centos/vol1

Mount the filesystem using mount command for use.
    #mount -t ext4 /dev/centos/vol1 /vol1
 

24.How to extend the volume group?

 vgextend we can increase the volume group.



25.Assume Volume group "centos" is already exists. How do you extend the volume group with 100GB? Provide all the steps with commands.

   1.Get the 100GB lun from storage team.(/dev/sdd)
   2.Create physcical volume ( # pvcreate /dev/sdd )
   2.Extend the volume group (# vgextend centos /dev/sdd)

26.If the centos has two physical volumes called /dev/sdc/ & /dev/sdd. How do you remove /dev/sdd from centos.

 vgreduce command will help to reduce, but we need to unmount the lv related to this vg & deactivate the same. Then we can use below command.
 vgreduce centos /dev/sdd/


27.How to decommission/remove LVM completely from the server?

     1.Un-mount all the logical filesystems
     2.Remove the logical volumes using "lvremove" command.
     3.Destroy the volume group using "vgremove" command.
     4.Use "pvremove" command remove the physical volumes from the system.


28. Why LVM is required in Linux Server?

LVM stands for Logical Volume Manager , to resize filesystem's size online we required LVM partition in Linux. Size of LVM partition can be extended and reduced using the lvextend & lvreduce commands respectively.


29. How create partition from the raw disk - Brief.

Using fdisk utility we can create partitions from the raw disk.Below are the steps to create partition from the raw disk :
- fdisk /dev/hd* (IDE) or /dev/sd* (SCSI)
- Type n to create a new partition
- After creating partition , type w command to write the changes to the partition table.

Monday, November 20, 2017

lsof command with example - Linvirtshell

Monday, November 20, 2017 0

The lsof is an excellent utility that can be used to identify process or user that is locking system resource such as file or network socket.

Basically, the /etc/services system files is usually used to map a numeric port number to a descriptive port name defined by user.

In the absence of any options, lsof lists all open files belonging to all active processes.

NAME

       lsof - list open files

To report all processes that are accessing the TCP sockets found on the system
#lsof -i TCP

To find out what process is holding TCP port 22
#lsof -i tcp:22

To show all resources that are being held by process id 2057
#lsof -p 2057

To find out what are the resources held by all cron processes. 
#lsof -o crond

To confirm resources that are being held by user id user linuser
#lsof -u linuser| more

To find out what are the processes that are locking the specify file /usr/sbin/anacron
# lsof /usr/sbin/anacron

Listing the files, any of whose IPv4 & IPv6 Internet address matches
# lsof -i4 -i6

Checking the count of open files
# lsof | wc -l

Listing of NFS files
#lsof -N

Hope it helps

Saturday, November 18, 2017

VM Tools version by using Powercli command - Vmware

Saturday, November 18, 2017 0
Use the below command to get the VM tools version.

get-vm |% { get-view $_.id } | select Name, @{ Name=";ToolsVersion";;

Expression={$_.config.tools.toolsVersion}}




Friday, November 17, 2017

Error "system was unable to find a physical volume" SOLVED -Step by Step

Friday, November 17, 2017 0

If we get Error  "system was unable to find a physical volume" . It needs  to restore  the corrupted Volume Group


Situation :

If the volume group metadata area of a physical volume is accidentally overwritten or otherwise destroyed, you will get an error message indicating that the metadata area is incorrect, or that the system was unable to find a physical volume with a particular UUID. You may be able to recover the data the physical volume by writing a new metadata area on the physical volume specifying the same UUID as the lost metadata.

Solution:

The following example shows the sort of output you may see if the metadata area is missing or corrupted.

[root@test]# lvs -a -o +devices

  Couldn't find device with uuid 'zhtUGH-1N2O-tHdu-b14h-gH34-sB7z-NHhkdf'.
  Couldn't find all physical volumes for volume group VG.
  Couldn't find device with uuid 'zhtUGH-1N2O-tHdu-b14h-gH34-sB7z-NHhkdf'.
  Couldn't find all physical volumes for volume group VG.

  ...

You may be able to find the UUID for the physical volume that was overwritten by looking in the /etc/lvm/archive directory. Look in the file VolumeGroupName_xxxx.vg for the last known valid archived LVM metadata for that volume group.

Alternately, you may find that deactivating the volume and setting the partial (-P) argument will enable you to find the UUID of the missing corrupted physical volume.

[root@test]# vgchange -an --partial

  Partial mode. Incomplete volume groups will be activated read-only.
  Couldn't find device with uuid 'zhtUGH-1N2O-tHdu-b14h-gH34-sB7z-NHhkdf'.
  Couldn't find device with uuid 'zhtUGH-1N2O-tHdu-b14h-gH34-sB7z-NHhkdf'.

  ...

Use the --uuid and --restorefile arguments of the pvcreate command to restore the physical volume. The following example labels the /dev/sdh1 device as a physical volume with the UUID indicated above, zhtUGH-1N2O-tHdu-b14h-gH34-sB7z-NHhkdf. This command restores the physical volume label with the metadata information contained in centos_00000-1802035441.vg, the most recent good archived metatdata for volume group .

The restorefile argument instructs the pvcreate command to make the new physical volume compatible with the old one on the volume group, ensuring that the the new metadata will not be placed where the old physical volume contained data (which could happen, for example, if the original pvcreate command had used the command line arguments that control metadata placement, or it the physical volume was originally created using a different version of the software that used different defaults).

The pvcreate command overwrites only the LVM metadata areas and does not affect the existing data areas.

[root@test]# pvcreate --uuid "zhtUGH-1N2O-tHdu-b14h-gH34-sB7z-NHhkdf" --restorefile /etc/lvm/archive/centos_00000-1802035441.vg /dev/sdh1
  Physical volume "/dev/sdh1" successfully created

You can then use the vgcfgrestore command to restore the volume group's metadata.

[root@test]# vgcfgrestore VG
  Restored volume group VG 

You can now display the logical volumes.

[root@test]# lvs -a -o +devices

  LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Devices

  stripe VG   -wi--- 300.00G                               /dev/sdh1 (0),/dev/sda1(0)
  stripe VG   -wi--- 300.00G                               /dev/sdh1 (34728),/dev/sdb1(0) 

The following commands activate the volumes and display the active volumes.

[root@test]# lvchange -ay /dev/VG/stripe
[root@test]# lvs -a -o +devices

  LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Devices
  stripe VG   -wi-a- 300.00G                               /dev/sdh1 (0),/dev/sda1(0)
  stripe VG   -wi-a- 300.00G                               /dev/sdh1 (34728),/dev/sdb1(0)

If the on-disk LVM metadata takes as least as much space as what overrode it, this command can recover the physical volume. If what overrode the metadata went past the metadata area, the data on the volume may have been affected. You might be able to use the fsck command to recover that data


Thursday, November 16, 2017

How to add temporary & permanent route in Linux box?

Thursday, November 16, 2017 0

Follow the below steps for adding the route as  temporary or permanent . 

Temporary Route

We just need to add the ip route add command in command line. like below

[root@testserver ~]#ip route add 10.217.156.0/25 via 10.194.32.1 dev eth0

[root@testserver ~]# route -n | grep -i 10.217.156.0

10.217.156.0    10.194.32.1     255.255.255.128 UG    0      0        0 eth0

Above route entry will be invicible post reboot the linux box.

Permanent Route

We need to create a file route-eth0 (since it is pointing eth0 device) under /etc/sysconfig/network-scripts/ path & add the below entry.

[root@testserver network-scripts]# cat route-eth0
10.217.156.0/25 via 10.194.32.1 dev eth0

save & exit.

Route will be persistent after every reboot of the Linux box.


Note : Permanent route need  network interface.reload for immediate effect.

Hope it helps.




Tuesday, November 14, 2017

Server hang at GRUB during boot - SOLVED

Tuesday, November 14, 2017 0
If a RHEL server hangs on boot with nothing more than the word GRUB in the upper left hand corner of the screen, this means that GRUB is unable to read its configuration file. If you actually get a GRUB menu, but the server does not boot then you have different and potentially more complex issue.

The most common reason for GRUB being unable to read its configuration is caused by a discrepancy between how the BIOS enumerated the hard drives and what GRUB expects to be its boot disk.


To correct this issue, boot the server in rescue mode.


Once booted into rescue mode and your root disk filesystems have been mounted. Check the /boot/grub/device.map file to ensure it has correctly identified the boot disk. hd0 should point to the disk that contains /boot. On an HP Proliant system you should see the following line:


(hd0) /dev/cciss/c0d0


If it does not, correct the file and then update GRUB by issuing the following command:


/sbin/grub --batch --device-map=/boot/grub/device.map --config-file=/boot/grub/grub.conf --no-floppy


And then from the GRUB prompt enter the following commands:


grub> root (hd0,0)
grub> setup (hd0)
grub> quit


You can now eject the ISO and reboot the server normally.

What is NUMA (Non-Uniform Memory Access)?

Tuesday, November 14, 2017 0
NUMA stands for Non-Uniform Memory Access,describes a system with more than one system bus. CPU resources and memory resources are grouped together into a “NUMA node.

The memory in a NUMA node is thus much more easily accessed by an associated CPU. A CPU that needs to access memory in a different node (“Remote Access”) will experience much higher latency, and thus reduced application performance. 


NUMA is, in short, an alternative approach to server architecture that links several small, high-performing nodes together inside a single server case.  


So long as the memory and CPU being used falls within the bounds of the NUMA node, local communication within a NUMA node allows a CPU much faster access to memory than in an ordinary system layout. This is especially important in the multi-GHz era of today, when CPUs operate significantly faster than the memory they are using. NUMA helps keep CPUs from entering a stalled state, waiting for data, by allowing the fastest possible access to memory. 



 Numactl is a Linux tool that you can use to view NUMA topology on Linux.


numactl -- hardware

[root@nsk ~]# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0
node 0 size: 2047 MB
node 0 free: 1454 MB
node distances:
node   0
  0:  10

numactl -- show


[root@nsk ~]# numactl --show
policy: default
preferred node: current
physcpubind: 0
cpubind: 0
nodebind: 0
membind: 0
 

Sunday, November 12, 2017

BUG: soft lockup - CPU#0 stuck for 10s!

Sunday, November 12, 2017 0

•Soft lockups are situations in which the kernel's scheduler  subsystem has not been given a chance to perform its job for more than  10 seconds; they can be caused by defects in the kernel, by hardware  issues or by extremely high workloads.

Run following command and check whether you still encounter these "soft lockup" errors on the system:

# sysctl -w kernel.softlockup_thresh=30

To make this parameter persistent across reboots by adding following line in /etc/sysctl.conf file:

 kernel.softlockup_thresh=30


Note: The softlockup_thresh kernel parameter was introduced in Red Hat Enterprise Linux 5.2 in kernel-2.6.18-92.el5 thus it is not possible to modify this on older versions

SOLVED : Buffer I/O error on boot

Sunday, November 12, 2017 0
Situation:

•After upgrading from Red Hat Enterprise Linux (RHEL) 5.1 to RHEL 5.5 (kernel 2.6.18-53.el5 to 2.6.18-194.8.1.el5), a system started to show IO errors on boot.

•The boot process took more time than before, but there are otherwise no significant problems occuring.


SCSI device sdc: 419430400 512-byte hdwr sectors (214748 MB)

sdc: Write Protect is off
sdc: Mode Sense: 77 00 10 08
SCSI device sdc: drive cache: write back w/ FUA
SCSI device sdc: 419430400 512-byte hdwr sectors (214748 MB)
sdc: Write Protect is off
sdc: Mode Sense: 77 00 10 08
SCSI device sdc: drive cache: write back w/ FUA
sdc:end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
Dev sdc: unable to read RDB block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
unable to read partition table
sd 1:0:0:1: Attached scsi disk sdc
 Vendor: SUN       Model: LCSM100_S         Rev: 0735
 Type:   Direct-Access                      ANSI SCSI revision: 05

Solution:


follow below solution to remediate above issue.


•Switching the controller to active/active mode would allow the devices to be probed through both controller ports and prevent the errors.

•An option to speed up the boot process is to rebuild the initrd without the HBA driver kernel modules and then probe the devices post boot, ie

# mkinitrd -v -f --omit-scsi-modules /boot/initrd-2.6.18-194.8.1.el5.img 2.6.18-194.8.1.el5

Performance collection tools to gather data for fault analysis in VMware

Sunday, November 12, 2017
This article explains how to use performance collection tools to gather data for analysis of faults such as:
    Unresponsive ESX hosts
    Unresponsive virtual machines
    ESX host purple diagnostic screens

Why gather performance data for a fault?

If the diagnostic logs do not help you determine the cause of a fault, you may need to use performance collection tools to gather further data for analysis. Set up performance collections tools to gather data about faults that may occur.

Performance gathering tools

VMware recommends the following tools for gathering performance data: 

top
The top utility provides a list of CPU-intensive tasks for the ESX host Service Console.
Use top in batch mode for Fault troubleshooting by directing the output to a file so that it can be reviewed after a recurrence.


Note: The top command is not available for ESXi.
To run the top utility, run the command:


# top –bc –d <delay in seconds> [–n <iterations>] > output-perf-stats-file.txt

 
Use the information in the output file to identify any trends before the fault. 


esxtop
The esxtop tool provides performance statistics for the entire ESX/ESXi host. It provides details of network, storage, CPU, and memory load from the VMkernel perspective. It provides details on a VMkernel world basis.
esxtop
To collect the data over long periods of time, run esxtop in batch mode. Direct the output to a file so that it can be reviewed after the fault.


To run the esxtop tool, run the command:


# esxtop –b –d <delay in seconds> [-n <iterations>] > output-perf-statistics-file.csv

 
Like esxtop, the resxtop tool provides performance statistics for a specified ESX host in the network. It provides the same performance information as esxtop and may be used either after deploying the VMware vSphere Management Assistant (vMA) virtual appliance or installing the VMware Command-Line Interface (vCLI). 


To run the resxtop tool and collect batch performance data, log into the vMA or open the vCLI, and execute the command:


# resxtop [server] [vihost] [portnumber] [username] -b -d <delay in seconds> [-n <interations>] > output-perf-statistics-file.csv


vm-support -s

 
Use the vm-support command with the -s parameter to collect performance statistics, system configuration information, and logging. Submit the file generated by this command to VMware Support for further assistance, if required. 


Performance Monitor (PERFMON.EXE)

 
Microsoft's Performance Monitor is a utility that comes with every Microsoft Windows NT-based Operating System. This utility can be used to monitor local and remote Microsoft Windows machines. It can log performance data and display data from logs or real-time data.


This utility is useful when reviewing data collected from the esxtop tool and for troubleshooting virtual machine unresponsiveness. When using Performance Monitor for virtual machine unresponsiveness, collect the data remotely from another Microsoft Windows machine so that the utility does not affect the data being gathered.
For more information about Performance Monitor on your specific version of Windows, refer to Microsoft support sites.

Friday, November 10, 2017

Time command in Linux Server - Brief explanation

Friday, November 10, 2017
NAME
       time - time a simple command or give resource usage
      
Format
       time [options] command [arguments...]

The time command runs the specified program command with the given arguments.  When command finishes, time writes a message to standard error giving timing statistics about this program run.  These statistics consist of 


(i) the elapsed real time between invocation and termination,
(ii) the user  CPU time (the sum of the tms_utime and tms_cutime values in a struct tms as returned by times.
(iii) the system CPU time (the sum of the tms_stime and tms_cstime values in a struct tms as returned by times.

real %e
user %U
sys %S

%e - Elapsed real time (in seconds).
%U - Total number of CPU-seconds that the process spent in user mode.
%S - Total number of CPU-seconds that the process spent in kernel mode.

Ex:

[root@nsk-linux ~]# time route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.2.0        0.0.0.0         255.255.255.0   U     1      0        0 eth4
0.0.0.0         10.0.2.2        0.0.0.0         UG    0      0        0 eth4

real    0m0.001s
user    0m0.000s
sys     0m0.001s

[root@nsk-linux ~]# time uptime

 08:34:58 up 57 min,  2 users,  load average: 0.04, 0.12, 0.08

real    0m0.003s
user    0m0.002s
sys     0m0.001s

For more option, please read man time.

Thursday, November 9, 2017

How to find the file in between some days & delete the same?

Thursday, November 09, 2017
Follow the below steps to find out the particular modified file in between 20 to 30 days & delete the same.

Command to find and list the file

#find / -mtime +20 -mtime -30 -type f -name test.* -exec ls -al {} \;   

Command to delete the listed file.

#find / -mtime +20 -mtime -30 -type f -name test.* -exec rm {} \;

As per your needs, change the file name.

Hope it helps.

Default Queue Depth values for QLogic HBAs for various ESXi/ESX versions

Thursday, November 09, 2017
This table lists the default Queue Depth values for QLogic HBAs for various ESXi/ESX versions:



The default Queue Depth value for Emulex adapters has not changed for all versions of ESXi/ESX released to date. The Queue Depth is 32 by default, and because 2 buffers are reserved, 30 are available for I/O data.

The default Queue Depth value for Brocade adapters is 32.

Wednesday, November 8, 2017

Enabling Intel VT-x and AMD-V Virtualization Hardware Extensions in BIOS of ESXI

Wednesday, November 08, 2017
This section describes how to identify hardware virtualization extensions and enable them in your BIOS if they are disabled. The Intel VT-x extensions can be disabled in the BIOS. The virtualization extensions cannot be disabled in the BIOS for AMD-V. 

Procedure for Enabling virtualization extensions in BIOS


1.    Reboot the computer and open the system's BIOS menu. This can usually be done by pressing the delete key, the F1 key or Alt and F4 keys or F10 key depending on the Harware.


2.    Enabling the virtualization extensions in BIOS


        a.    Open the Processor submenu The processor settings menu may be hidden in the Chipset, Advanced CPU Configuration or Northbridge.
         b.    Enable Intel Virtualization Technology (also known as Intel VT-x). AMD-V extensions cannot be disabled in the BIOS and should already be enabled. The virtualization extensions may be labeled Virtualization Extensions, Vanderpool or various other names depending on the OEM and system BIOS.
         c.    Enable Intel VT-d or AMD IOMMU, if the options are available. Intel VT-d and AMD IOMMU are used for PCI device assignment.
         d.    Select Save & Exit.


3.    Reboot the machine.


4.    When the machine has booted, run cat /proc/cpuinfo |grep -E "vmx|svm". Specifying --color is optional, but useful if you want the search term highlighted. If the command outputs, the virtualization extensions are now enabled. If there is no output your system may not have the virtualization extensions or the correct BIOS setting enabled.

Tuesday, November 7, 2017

How to Determin if Intel Virtualization Technology or AMD Virtualization is enabled in the BIOS without reboot

Tuesday, November 07, 2017
When troubleshooting vMotion, Enhanced VMotion Capability (EVC), or 64bit virtual machine performance, you may need to determine if the Intel Virtualization Technology (VT) or AMD Virtualization (AMD-V) are enabled in the BIOS.

This section describes how to identify hardware virtualization extensions and enable them in your BIOS if they are disabled.

Log in to the ESX host as the root user.

    Run this command:

    esxcfg-info|grep "\----\HV Support"

    The output of the HV Support command indicates the type of Hyper-visor support available. These are the descriptions for the possible values:

    0 - VT/AMD-V indicates that support is not available for this hardware.
    1 - VT/AMD-V indicates that VT or AMD-V might be available but it is not supported for this hardware.2 - VT/AMD-V indicates that VT or AMD-V is available but is currently not enabled in the BIOS.3 - VT/AMD-V indicates that VT or AMD-V is enabled in the BIOS and can be used.

How to check the Listening Ports on Linux Server - Linvirtshell

Tuesday, November 07, 2017

We can check the listening ports on Linux Server by below ways

fuser - identify processes using files or sockets (Refer- Man Pages for more information)

Linvirtshell.com

netstat -  Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships (Refer- Man Pages for more information)

Linvirtshell.com

ssh  - OpenSSH SSH client (remote login program)  (Refer- Man Pages for more information)

[root@nsk-linux ~]# ssh -vv 10.0.2.15 25

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.0.2.15 [10.0.2.15] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/identity-cert type -1
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
....
...

lsof - list open files (Refer- Man Pages for more information)

Linvirtshell.com

  nmap - Network exploration tool and security / port scanner (Refer- Man Pages for more information) 

Here for ex, i took output from localhost, you can change any ip or server name.
Linvirtshell.com

telnet -  user interface to the TELNET protocol  (Refer- Man Pages for more information)

Here for ex, i took output from localhost, you can change any ip or server name.

Linvirtshell.com

Monday, November 6, 2017

How to Avoid & Solve the Disk Limitations error with KVM guests?

Monday, November 06, 2017
There are some limitations specific to the virtio-blk driver that will be discussed in this kbase article. Please note, these are not general limitations of KVM, but rather relevant only to cases where virtio-blk is used.

Disks under KVM are para-virtualized block devices when used with the virtio-blk driver. All para-virtualized devices (e.g. disk, network, balloon, etc.) are PCI devices. Presently, guests are limited to a maximum of 32 PCI devices. Of the 32, 4 are required by the guest for minimal baseline functionality and are
therefore reserved.


When adding a disk to a KVM guest, the default method assigns a separate virtual PCI controller for every disk to allow hot-plug support (i.e. the ability to add/remove disks from a running VM without downtime). Therefore, if no other PCI devices have been assigned, the max number of hot-pluggable disks is 28.


If a guest requires more disks than the available PCI slots allow, then there are three possible work-arounds.


1. Use PCI pass-through to assign a physical disk controller (i.e. FC HBA, SCSI controller, etc.) to the VM and subsequently use as many devices as that controller supports.
2. Forego the ability to hot-plug and assign the virtual disks using multi-function PCI addressing.
3. Use the virtio-scsi driver, which creates a virtual SCSI HBA that occupies a single PCI address and supports thousands of hot-plug disks.


Here i have used the option 2 for correcting this problem.

Multi-function PCI addressing allows up to 8 virtual disks per controller. Therefore you can have n * 8 possible disks, where n is the number of available PCI slots.

On a system with 28 free PCI slots, you can assign up to 224 virtual disks to that VM. However, as previously stated you will not be able to add and/or remove the multi-function disks from the guest without a reboot of the guest.

Any disks assigned without multifunction can however continue to use
hot-plug.

The XML config below, demonstrates how to configure a multi-function PCI controller:

<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/rootvg/lvtest01'/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/rootvg/lvtest02'/>
<target dev='vdc' bus='virtio'/>
<alias name='virtio-disk2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x1' multifunction='on'/>


In the above, we defined a new controller in slot 7. And then attached 2 disks (vdb, vdc) to that controller. So that only one PCI slot is used, slot 7. And 2 disks are attached to it. We could add 8 more disks to the controller before having to create a new controller in slot 8 (assuming 8 is the next available slot).

You can check a guests config from the virtualization host, by using "virsh dumpxml <guest>". This will show you which slots are in use and therefore which are available.


To add one or more multifunction controllers you would use "virsh edit <guest>" and then add the appropriate XML entries modeled after the example above. Remember, the guest must be rebooted before the changes to the XML config are applied.