Issue
After updating the system, no one (including root) can use su without getting bash: /bin/su: Permission denied errors, but ssh and terminal logins work as normal. SELinux is disabled and no new log entries are generated in /var/log/secure or /var/log/messages when root tries to run /bin/su, which has appropriate permissions, file size, and md5sum. The /etc/nsswitch.conf, /etc/pam.d/system-auth, and /etc/pam.d/su files have all been replaced with default versions and still the problem remains.
Raw
[root@localhost ~]# ls -l /bin/su
-rwsr-xr-x 1 root root 28336 May 11 2011 /bin/su
[root@localhost ~]# /bin/su -
bash: /bin/su: Permission denied
[root@localhost ~]# strace -tvfs 2048 -o su_strace_root.log su -lc exit
strace: exec: Permission denied
<truncacted strace output:>
28530 15:17:38 execve("/bin/su", ["su", "-lc", "exit"], ... "_=/usr/bin/strace"]) = -1 EACCES (Permission denied)
Resolution
The one customer that reported this issue eventually resolved it by realizing that the system had a 3rd-party LDAP application installed that was no longer being used but which hadn't been permanently disabled and was therefore initialized after the reboot. Customer quote:
Issue was basically with an LDAP client called TAMOS running on the system was causing this authorization issues.
I hit that because after reboot all the disabled TAMOS processes started and not allowing us to authorize to su -.
Now TAMOS is been uninstalled from the system and we are good now.
If seeing a similar problem, this particular cause could be confirmed or ruled out by checking if the kail kernel module is loaded or if there are TAMOS log entries on the system.
Raw
$ lsmod | grep kail
kail 124328 1 kaznmod,[permanent]
$ egrep 'TAMOS|kail' /var/log/dmesg
kail: no version for "struct_module" found: kernel tainted.
kail: no version magic, tainting kernel.
TAMOS: INFO kail_init_module kernel module initializing
TAMOS INFO: kail_kernel.c: init_module OK: Perm2rw
TAMOS: INFO kail_kernel kailPerm2rw
kail_kernel change_perm loop: 0
kail_kernel change_perm loop: 1
kail_kernel change_perm loop: 0
kail_kernel change_perm loop: 1
TAMOS INFO: nct_async.c TIMEDWAIT_THREAD NOT enabled
TAMOS INFO: nct_async.c TRACE_AREXIT enabled
TAMOS INFO: nct_asyncInit pre: FFFFFFFF88512A50:FFFFFFFF88512A68:FFFFFFFF88512A60
TAMOS INFO nct_init.. scanning procs
TAMOS INFO: nct_init LSM enabled
TAMOS INFO: ignoring NR call index: 1000 at kail index: 3
kail_kernel kailPerm2ro
kail_kernel change_perm loop: 0
kail_kernel change_perm loop: 1
TAMOS INFO:kail_kernel perm2ro done: 0
kail_kernel change_perm loop: 0
kail_kernel change_perm loop: 1
TAMOS INFO:kail_kernel perm2ro done: 0
TAMOS INFO: setting up as a security module
TAMOS INFO: nct_arThread entering base counter local: 0: FFFF810234A75DEC global: 0: FFFFFFFF88512A48
TAMOS INFO: nct_arThread entering base counter local: 0: FFFF810234A73DEC global: 1000: FFFFFFFF88512A48
TAMOS INFO: nct_arThread entering base counter local: 0: FFFF810234A71DEC global: 2000: FFFFFFFF88512A48
TAMOS INFO: nct_arThread entering base counter local: 0: FFFF810234A6FDEC global: 3000: FFFFFFFF88512A48
TAMOS INFO: kaznmod successfully inserted (30:(10:0) of 6) as a security framework and overlaying.
TAMOS INFO: Initialization complete