This Blog is to share our knowledge and expertise on Linux System Administration and VMware Administration

Showing posts with label Linux Troubleshooting. Show all posts
Showing posts with label Linux Troubleshooting. Show all posts

Tuesday, November 14, 2017

Server hang at GRUB during boot - SOLVED

Tuesday, November 14, 2017 0
If a RHEL server hangs on boot with nothing more than the word GRUB in the upper left hand corner of the screen, this means that GRUB is unable to read its configuration file. If you actually get a GRUB menu, but the server does not boot then you have different and potentially more complex issue.

The most common reason for GRUB being unable to read its configuration is caused by a discrepancy between how the BIOS enumerated the hard drives and what GRUB expects to be its boot disk.


To correct this issue, boot the server in rescue mode.


Once booted into rescue mode and your root disk filesystems have been mounted. Check the /boot/grub/device.map file to ensure it has correctly identified the boot disk. hd0 should point to the disk that contains /boot. On an HP Proliant system you should see the following line:


(hd0) /dev/cciss/c0d0


If it does not, correct the file and then update GRUB by issuing the following command:


/sbin/grub --batch --device-map=/boot/grub/device.map --config-file=/boot/grub/grub.conf --no-floppy


And then from the GRUB prompt enter the following commands:


grub> root (hd0,0)
grub> setup (hd0)
grub> quit


You can now eject the ISO and reboot the server normally.

Sunday, November 12, 2017

BUG: soft lockup - CPU#0 stuck for 10s!

Sunday, November 12, 2017 0

•Soft lockups are situations in which the kernel's scheduler  subsystem has not been given a chance to perform its job for more than  10 seconds; they can be caused by defects in the kernel, by hardware  issues or by extremely high workloads.

Run following command and check whether you still encounter these "soft lockup" errors on the system:

# sysctl -w kernel.softlockup_thresh=30

To make this parameter persistent across reboots by adding following line in /etc/sysctl.conf file:

 kernel.softlockup_thresh=30


Note: The softlockup_thresh kernel parameter was introduced in Red Hat Enterprise Linux 5.2 in kernel-2.6.18-92.el5 thus it is not possible to modify this on older versions

SOLVED : Buffer I/O error on boot

Sunday, November 12, 2017 0
Situation:

•After upgrading from Red Hat Enterprise Linux (RHEL) 5.1 to RHEL 5.5 (kernel 2.6.18-53.el5 to 2.6.18-194.8.1.el5), a system started to show IO errors on boot.

•The boot process took more time than before, but there are otherwise no significant problems occuring.


SCSI device sdc: 419430400 512-byte hdwr sectors (214748 MB)

sdc: Write Protect is off
sdc: Mode Sense: 77 00 10 08
SCSI device sdc: drive cache: write back w/ FUA
SCSI device sdc: 419430400 512-byte hdwr sectors (214748 MB)
sdc: Write Protect is off
sdc: Mode Sense: 77 00 10 08
SCSI device sdc: drive cache: write back w/ FUA
sdc:end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
Dev sdc: unable to read RDB block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
unable to read partition table
sd 1:0:0:1: Attached scsi disk sdc
 Vendor: SUN       Model: LCSM100_S         Rev: 0735
 Type:   Direct-Access                      ANSI SCSI revision: 05

Solution:


follow below solution to remediate above issue.


•Switching the controller to active/active mode would allow the devices to be probed through both controller ports and prevent the errors.

•An option to speed up the boot process is to rebuild the initrd without the HBA driver kernel modules and then probe the devices post boot, ie

# mkinitrd -v -f --omit-scsi-modules /boot/initrd-2.6.18-194.8.1.el5.img 2.6.18-194.8.1.el5

Friday, November 3, 2017

SOLVED : pam_ldap: error trying to bind as user

Friday, November 03, 2017
If you are getting error below after giving the correct password

Nov  2 03:56:42 testserver sshd[30173]: pam_ldap: error trying to bind as user "uid=testuser,ou=People,dc=test,dc=testdomain,dc=com" (Invalid credentials)

Nov  2 03:56:43 testserver sshd[30173]: Failed password for testuser from 10.17.0.3 port 51306 ssh2

Reason: Password is not syncing properly to all client server during the scheduled window

Solution : Restart the slapd service on LDAP server & it will sync to all server.

#/etc/init.d/slapd restart

Hope it helps.

Wednesday, November 1, 2017

How do I exclude Kernel or other packages from getting updated in RHEL while updating system via yum?

Wednesday, November 01, 2017 0

Excluding  Kernel or other packages from getting updated in RHEL while updating system via yum

The up2date command in Red Hat Enterprise Linux 4 excludes kernel updates by default. The yum in Red Hat Enterprise Linux 5  includes kernel updates by default.

 To skip installing or updating kernel or other packages while using the yum update utility in Red Hat Enterprise Linux 5 and 6 use following options:

 Temporary solution via Command line:

 # yum update --exclude=PACKAGENAME

For example, to exclude all kernel related packages:

# yum update --exclude=kernel*

To make permanent changes, edit the /etc/yum.conf file and following entries to it:

[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exclude=kernel* redhat-release*                           <====

NOTE: If there are multiple package to be excluded then separate them using a single space or comma. Also, do not add multiple 


exclude= lines in the configuration file because yum only considers the last exclude entry.

To exclude 32 bit packages edit /etc/yum.conf file.

exclude=*.i?86 *.i686