This Blog is to share our knowledge and expertise on Linux System Administration and VMware Administration

Wednesday, December 20, 2017

ssh_exchange_identification: Connection closed by remote host - Password less authentication setup

Wednesday, December 20, 2017 0

ssh_exchange_identification: Connection closed by remote host - Password less authentication setup 


Situation:
While setup passwordless authentication from testserver2 (192.181.166.55) to testserver1 (192.181.130.55)  server, getting  error "ssh_exchange_identification: Connection closed by remote host"

[kanachim@testserver2 .ssh]$ ssh -vv testserver1
OpenSSH_5.3p1-hpn13v7, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.181.166.55 [192.181.166.55] port 22.
debug1: Connection established.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /home/kanachim/.ssh/id_rsa type 1
debug1: identity file /home/kanachim/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host

Solution:
Check /etc/hosts.deny on server testserver1

root@host testserver1# grep sshd /etc/hosts.deny 
# DenyHosts: Mon Dec 18 22:10:38 2017 | sshd: 192.181.166.55
sshd: 192.181.166.55

Remove the sshd entry from hosts.deny on testserver1

Login to testserver2 Switch to user kanachim & create new key

-bash-3.2$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/kanachim/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/kanachim/.ssh/id_rsa.
Your public key has been saved in/home/kanachim/.ssh/id_rsa.pub.
The key fingerprint is:
37:6a:69:c1:a4:01:4b:c5:34:be:32:42:90:0b:20:a0 kanachim@192.181.166.55
The key's randomart image is:
+--[ RSA 2048]----+
|B.  o++          |
|=. . +..         |
|E.. . o .        |
|..     *         |
|  . o o S o      |
|   . o   = .     |
|        =        |
|       o         |
|                 |
+-----------------+
-bash-3.2$
-bash-3.2$ ssh-copy-id kanachim@192.181.130.55
Password:
Now try logging into the machine, with "ssh 'kanachim@192.181.130.55'", and check in:
  .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

Now login from testserver2 to testserver1

-bash-3.2$ ssh 192.181.130.55
Last login: Thu Feb 27 00:30:38 2014 from 10.217.230.145
-bash-3.2$ hostname
testserver1

Tuesday, December 19, 2017

Recover RPM DB from a corrupted RPM database in RHEL 7

Tuesday, December 19, 2017 0

Rebuilding corrupted rpm database in RHEL7

Situation:
Although everything is done to ensure that your RPM databases are intact, your RPM database may become corrupt and unuseable. This happens mainly if the filesystem on which the rpm db resides is suddenly inaccessible (full, read-only, reboot, or so on).

Solution:
1.Start by creating a backup of your corrupt rpm db, as follows:
[root@nsk ~]# tar zcvf rpm-db.tar.gz /var/lib/rpm/*
tar: Removing leading `/' from member names
/var/lib/rpm/Basenames
/var/lib/rpm/Conflictname
/var/lib/rpm/__db.001
/var/lib/rpm/__db.002
/var/lib/rpm/__db.003
/var/lib/rpm/Dirnames
/var/lib/rpm/Group
/var/lib/rpm/Installtid
/var/lib/rpm/Name
/var/lib/rpm/Obsoletename
/var/lib/rpm/Packages
/var/lib/rpm/Providename
/var/lib/rpm/Requirename
/var/lib/rpm/Sha1header
/var/lib/rpm/Sigmd5
/var/lib/rpm/Triggername

2.Remove stale lock files if they exist through the following command:
[root@nsk ~]# rm -f /var/lib/rpm/__db*

3.Now, verify the integrity of the Packages database via the following:
[root@nsk ~]# /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages; echo $?
BDB5105 Verification of /var/lib/rpm/Packages succeeded.
0

If it prints 0, proceed to next step.

4. Rename the Packages file (don't delete it, we'll need it!), as follows:
[root@nsk ~]# mv /var/lib/rpm/Packages  /var/lib/rpm/Packages.org

5. Now, dump the Packages db from the original Packages db by executing the following command:
[root@nsk ~]# cd /var/lib/rpm/
 [root@nsk rpm]# /usr/lib/rpm/rpmdb_dump Packages.org | /usr/lib/rpm/rpmdb_load Packages
rpmdb_load: BDB1540 configured environment flags incompatible with existing environment

6.Verify the integrity of the newly created Packages database. Run the following:
[root@nsk rpm]#  /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages; echo $?
BDB5105 Verification of /var/lib/rpm/Packages succeeded.
0

If the exit code is not 0, you will need to restore the database from backup.

7. Rebuild the rpm indexes, as follows:
[root@nsk ~]# rpm -vv --rebuilddb
[root@nsk rpm]# rpm -vv --rebuilddb
D: rebuilding database /var/lib/rpm into /var/lib/rpmrebuilddb.1312
D: opening  db environment /var/lib/rpm private:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db environment /var/lib/rpmrebuilddb.1312 private:0x401
D: opening  db index       /var/lib/rpmrebuilddb.1312/Packages (none) mode=0x42
D: opening  db index       /var/lib/rpmrebuilddb.1312/Packages 0x1 mode=0x42
D: disabling fsync on database
....
...
D: adding "5f7fd424d0773a4202731bff4901d449699b0929" to Sha1header index.
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm
D: closed   db index       /var/lib/rpmrebuilddb.1312/Sha1header
D: closed   db index       /var/lib/rpmrebuilddb.1312/Sigmd5
D: closed   db index       /var/lib/rpmrebuilddb.1312/Installtid
D: closed   db index       /var/lib/rpmrebuilddb.1312/Dirnames
D: closed   db index       /var/lib/rpmrebuilddb.1312/Triggername
D: closed   db index       /var/lib/rpmrebuilddb.1312/Obsoletename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Conflictname
D: closed   db index       /var/lib/rpmrebuilddb.1312/Providename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Requirename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Group
D: closed   db index       /var/lib/rpmrebuilddb.1312/Basenames
D: closed   db index       /var/lib/rpmrebuilddb.1312/Name
D: closed   db index       /var/lib/rpmrebuilddb.1312/Packages
D: closed   db environment /var/lib/rpmrebuilddb.1312

8. Use the following command to check the rpm db with yum for any other issues (this may take a long time):
[root@nsk rpm]# yum check
Loaded plugins: fastestmirror
....
...

9. Restore the SELinux context of the rpm database through the following command:
[root@nsk rpm]# restorecon -R -v /var/lib/rpm

Monday, December 18, 2017

How to reboot the Xen Virtual Machine when was at hung state.

Monday, December 18, 2017 0

How to reboot the Xen Virtual Machine when was at hung state.

If console is not working from the host(Dom0), the console was opened in order to use sysrq magic packages
#xm console xenvm006

From other terminal on the node, run the below  comamnd one by one.
#xm sysrq xenvm006  h
#xm sysrq xenvm006  m    #should show the amount of memory been used
#xm sysrq xenvm006  t      #should show the current tasks

the output is  "console is opened and was not good"  also   "vm was with out resources"
Check the xentop, if  a high cpu usage is shown like 200%

Then reboot the virtual machine.

Open one more terminal & run the below command.

#xm destroy  xenvm006

if the virtual machine is running under cluster,  first disable the vm from cluster to avoid failover.

#clusvcadm -d vm:xenvm006

Then destroy the virtual machine.

Because as the vm is hung state, clusvcadm -d wont do a clean shutdown, so we need to destroy it manually.

#xm create xenvm006  (will start the vm)
#clusvcadm -e vm:xenvm006   (with cluster)

Sunday, December 17, 2017

Understanding the LVM Configuration Files in Linux

Sunday, December 17, 2017 0

Understanding the LVM Configuration Files in Linux

The following files are part of LVM configuration:

/etc/lvm/lvm.conf
    Central configuration file read by the tools.

etc/lvm/lvm_hosttag.conf
    For each host tag, an extra configuration file is read if it exists: lvm_hosttag.conf. If that file defines new tags,
    then further configuration files will be appended to the list of tiles to read in.
In addition to the LVM configuration files, a system running LVM includes the following files that affect LVM system setup:

/etc/lvm/cache
    Device name filter cache file (configurable).

/etc/lvm/backup/
    Directory for automatic volume group metadata backups (configurable).

/etc/lvm/archive/
    Directory for automatic volume group metadata archives (configurable with regard to directory path and archive history depth).

/var/lock/lvm/
    In single-host configuration, lock files to prevent parallel tool runs from corrupting the metadata; in a cluster, cluster-wide DLM is used.

How to monitor the permission change and ownership change of a particular directory or file in Linux Server

Sunday, December 17, 2017 0

Monitoring the permission change and ownership change of a particular directory or file in Linux Server

1. Use the audit package to accomplish this task.2. Ensure the auditd service is running, and set to start on boot chkconfig auditd on3. Set a watch on the required file to be monitored by using the auditctl command:

# auditctl -w /etc/passwd -p war -k monitor-passwd

here:
    auditctl   :the command used to add entries to the audit database.
    -w            :Insert a watch for the file system object at path, i.e. /etc/shadow.
    -p             :Set permissions filter for a file system watch. r=read, w=write, x=execute, a=attribute change.
    -k             : Set a filter key on an audit rule. The filter key is an arbitrary string of text that can be up to 31 bytes long. 

It can uniquely identify the audit records produced by a rule.

Note that you must add the rule to /etc/audit/audit.rules on RHEL5 or RHEL6  in order for them to persist after reboot.
    You can place the watch rule in the /etc/audit/audit.rules file to set permanently:
-w /etc/passwd -p a -k monitor-passwd
       Check auditd service and if it is stopped, start it.

# service auditd status
# chkconfig --list auditd
# chkconfig auditd on

In this example, a watch is placed on the /etc/passwd file for any syscalls which perform a write, read, or attribute change
    (-p war). This is logged with the key monitor-passwd. This key can be used to search through the audit logs to find these actions, 
 using the ausearch command:

# ausearch -ts today -k monitor-passwd
----
time->Sat May  3 07:32:20 2009
type=PATH msg=audit(117045140.872:34): item=0 name="/etc/passwd" inode=1308742 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0
type=CWD msg=audit(117045140.872:34): cwd="/root"
type=SYSCALL msg=audit(117045140.872:34): arch=40000003 syscall=226 success=yes exit=0 a0=867c4b8 a1=458bcc4f a2=8686800 a3=1c
items=1 ppid=3544 pid=3558 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 comm="vim" exe="/usr/bin/vim"
subj=root:system_r:unconfined_t:s0-s0:c0.c1023 key="monitor-passwd"

    From this trace, it can be seen that the file /etc/resolve.conf was edited using the /usr/bin/vim command. The user that ran the  command was running with the root:system_r:unconfined_t:s0-s0:c0.c1023 SELinux context. And, the timestamp can be converted into readable form.

# date -d @117045140
Sat May  3 05:32:20 CST 2009

    You can search for an event based on the given key string
# ausearch -k monitor-passwd
    For a clearly view, you can generate report base on audit rule keys
# aureport -k

Friday, December 15, 2017

Identify the differentiation between Oracle Enterprise Linux from Redhat Enterprise Linux

Friday, December 15, 2017 0

Identify the differentiation between Oracle Enterprise Linux from Redhat Enterprise Linux

We can differentiate the OEL from RHEL by checking below things

1. Kernel  - uname -mrs command

RHEL runs with normal Redhat Kernel
[root@nsk ~]# uname -mrs
Linux 3.10.0-693.5.2.el7.x86_64 x86_64

OEL runs with Unbreakable Enterprise Kernel
[root@testserver ~]# uname -mrs
Linux 3.8.13-118.16.2.el6uek.x86_64 x86_64

2.  Yum repositiry

RHEL points to  RHEL repo name (only 3)
ex: RHEL-7 - Base 
      RHEL-7 - Extras
      RHEL-7 - Updates

OEL points to Oracle repo name ( customized many repo)
ex: Oracle Linux 6Server Latest (x86_64)
      OL 6Server - x86_64

3. MTA

RHEL: comes with sendmail and postfix
OEL : supports sendmail as a preferred MTA

4. Patching

RHEL gets update from  RHN satellite server
OEL gets update from  Oracle Spacewalk server

5. Linux Standard Base (lsb_release -a)
RHEL
Distributor ID : RHEL
Description    : RHEL Linux release 7.4.1708 (Core)
Codename     : Core
LSB Version   : core-4.1-amd64:core-4.1-noarch

OEL
Distributor ID : OracleServer
Description    : Oracle Linux Server release 6.8
Codename    : n/a
LSB Version:  :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch

6. Linux Release by version

[root@nsk ~]# cat /proc/version
Linux version 3.10.0-693.5.2.el7.x86_64 (brewbuilder@xx-xx.build.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Fri Oct 20 20:32:50 UTC 2017

[root@testserver ~]# cat /proc/version
Linux version 3.8.13-118.16.2.el6uek.x86_64 (mockbuild@x86-ol6-builder-04) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #2 SMP Wed Jan 11 17:19:24 PST 2017

Thursday, December 14, 2017

Understanding the Redhat Virtualization - Log files

Thursday, December 14, 2017 0

Understanding the Redhat Virtualization -  Log files

Red Hat Virtualization features the xend daemon and qemu-dm process, two utilities that write the multiple log files to the /var/log/xen/ directory:

xend.log is the log file that contains all the data collected by the xend daemon, whether it is a normal system event, or an operator initiated action. All virtual machine operations (such as create, shutdown, destroy, etc.) appears here. The xend.log is usually the first place to look when you track down event or performance problems. It contains detailed entries and conditions of the error messages.

xend-debug.log is the log file that contains records of event errors from xend and the Virtualization subsystems (such as framebuffer, Python scripts, etc.)

Xen-hotplug-log is the log file that contains data from hotplug events. If a device or a network script does not come online, the event appears here.

qemu-dm. [PID].log is the log file created by the qemu-dm process for each fully virtualized guest. When using this log file, you must retrieve the given qemu-dm process PID, by using the ps command to examine process arguments to isolate the qemu-dm process on the virtual machine. Note that you must replace the [PID] symbol with the actual PID qemu-dm process.

If you encounter any errors with the Virtual Machine Manager, you can review the generated data in the virt-manager.log file that resides in the /.virt-manager directory. Note that every time you start the Virtual Machine Manager, it overwrites the existing log file contents. Make sure to backup the virt-manager.log file, before you restart the Virtual Machine manager after a system error.

Wednesday, December 13, 2017

Unable to switch to root using sudo su – effective uid is not 0, is sudo installed setuid root

Wednesday, December 13, 2017 0

Unable to switch to root using sudo su –  effective uid is not 0, is sudo installed setuid root 

Error – While switching to root using sudo su - , throwing the below Error.

[root@test ~]# ssh testserver.local.com -l nsk
nsk@testserver.local.com's password:
Last login: Tue Dec 12 09:24:48 2017 from 10.0.0.16
-bash-3.2$ sudo su -
sudo: effective uid is not 0, is sudo installed setuid root?

Reason:
/usr/bin/sudo executable don’t have the setuid permission 

Solution:
Set the setuid permission for /usr/bin/sudo command. #chmod u+s /usr/bin/sudo

Example:-
[root@testserver ~]# ls -al /usr/bin/sudo
-rwxr-xr-x 1 root root 697605 Mar  5  2014 /usr/bin/sudo
[root@testserver ~]# chmod u+s /usr/bin/sudo
[root@testserver ~]# ls -al /usr/bin/sudo
-rwsr-xr-x 1 root root 697605 Mar  5  2014 /usr/bin/sudo