This Blog is to share our knowledge and expertise on Linux System Administration and VMware Administration

Sunday, December 31, 2017

Configuring logrotate in RHEL7

Sunday, December 31, 2017 0

Configuring logrotate in RHEL7

The logrotate tool allows you to rotate the logs that are generated by applications and scripts
It keeps your log directories clutter-free and minimizes disk usage when correctly configured.

The logrotate tool is installed by default.This document will show you how to rotate logs for rsyslog. We will rotate the logs everyday, add an extension based on the date, compress them with a one-day delay, and keep them for 365 days. Perform the following steps:
First, to check logrotate is installed, perform the following command:
[root@nsk ~]# rpm -qa | grep -i logrotate
logrotate-3.8.6-14.el7.x86_64

Ensure that it's enabled through the following:
[root@nsk ~]# systemctl restart crond

Open /etc/logrotate.d/syslog with your favorite editor. The contents of this file are the following, by default:
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Now, replace this with the following code:
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    compress
    daily
    delaycompress
    dateext
    missingok
    rotate 365
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}
Finally, save the file.

The logrotate tool is a script that is launched by cron everyday.
The directives added to the default logrotate definition are compress, daily, delaycompress, dateext, missingok, and rotate.

The compress directive compresses old versions of the log files with gzip. This behavior is somewhat changed by specifying delaycompress. This causes us to always have the most recently rotated log file available uncompressed.

The daily directive makes logrotate execute the definition every day. The rotate directive only keeps x rotated log files before deleting the oldest. In this case, we have specified this to be 365, which means that while rotating daily, the logs are kept for 365 days.
The missingok directive makes it alright for syslog to not create a file, which, however unlikely, is possible.

The dateext directive appends a date to the rotated file in the form of yyyymmdd instead of a number, which is the default.

The /etc/logrotate.conf file contains the defaults directives for all definitions. If you don't specifically use a directive within a definition for a file, the values in this file will be used if specified.

Yum, for instance, doesn't generate a lot of messages, and it keeps this log file readable for much longer than your syslog files. This, by the way, is reflected in the definition for yum.
If you want to debug your new configuration, this can be achieved by executing the following to test just one configuration:

# /usr/sbin/logrotate -v /etc/logrotate.d/<config file>

[root@nsk ~]# /usr/sbin/logrotate -v /etc/logrotate.d/syslog
reading config file /etc/logrotate.d/syslog
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
....
....
renaming /var/log/spooler to /var/log/spooler-20171231
disposeName will be /var/log/spooler-20171231.gz
running postrotate script
removing old log /var/log/cron-20171231.gz
error: error opening /var/log/cron-20171231.gz: No such file or directory
set default create context
[root@nsk ~]#

Alternatively, you can use the following to test everything:

[root@nsk ~]# /usr/sbin/logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file bootlog
reading config file chrony
reading config file numad
reading config file syslog
reading config file wpa_supplicant
reading config file yum
reading config file yum_24dec2017
error: yum_24dec2017:1 duplicate log entry for /var/log/yum.log
....
..

  rotating pattern: /var/log/btmp  monthly (1 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/btmp
  log does not need rotating (log has been rotated at 2017-12-14 11:40, that is not month ago yet)
set default create context
[root@nsk ~]#

Friday, December 29, 2017

Enabling EPEL repository by using ansible playbook

Friday, December 29, 2017 0

Enabling EPEL repository by using ansible playbook

EPEL is the most important repository for Enterprise Linux and it contains a lot of additional packages. It's also a safe repository since no package in EPEL will conflict with packages in the base repository. To enable EPEL in RHEL/CentOS 7, it is enough to just install the epel-release package. To do so in Ansible, we will use:

$vi setup_epel.yaml

- name: Ensure EPEL is enabled 
  yum: 
    name: epel-release 
    state: present 
    become: True 

As you can see, we have used the yum module, specifying the name of the package and that we want it to be present.

Running this playbook
To instruct Ansible to execute a playbook instead of a module, we will have to use a command ansible-playbooks.

Syntax :  ansible-playbook -i HOST setup_epel.yaml

Command

$ ansible-playbook -i test01.fale.io setup_epel.yaml

here,

i   - group of server or server name (Inventory)

Thursday, December 28, 2017

Installing Ansible from source in RHEL7

Thursday, December 28, 2017 0

Installing Ansible from source in RHEL7

By default git package is installed in RHEL 7.

You can install Ansible directly from the source. Installing from source does not require any root permissions. Let's clone a repository and activate virtualenv, which is an isolated environment in Python where you can install packages without interfering with the system's Python packages. The command and the resulting output for the repository is as follows:

[root@nsk nsk]# git clone git://github.com/ansible/ansible.git
Cloning into 'ansible'...
remote: Counting objects: 282962, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 282962 (delta 2), reused 2 (delta 0), pack-reused 282953
Receiving objects: 100% (282962/282962), 91.03 MiB | 113.00 KiB/s, done.
Resolving deltas: 100% (183996/183996), done.
[root@nsk nsk]#
.
[root@nsk nsk]# cd ansible/
[root@nsk ansible]# source ./hacking/env-setup
Ansible now needs setuptools in order to build. Install it using your package manager (usually python-setuptools) or via pip (pip install setuptools).

Setting up Ansible to run out of checkout...

PATH=/home/nsk/ansible/bin:/home/nsk/ansible/test/runner:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
PYTHONPATH=/home/nsk/ansible/lib:
MANPATH=/home/nsk/ansible/docs/man:
Remember, you may wish to specify your host file with -i
Done!
[root@nsk ansible]#

Ansible needs a couple of Python packages, which you can install using pip. If you don't have pip installed on your system, install it using the following command. If you don't have easy_install installed, you can install it using Python's setuptools package on Red Hat systems, or by using Brew on the Mac:

[root@nsk ansible]# yum -y update
[root@nsk ansible]# cd ..
[root@nsk nsk]# curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1558k  100 1558k    0     0   104k      0  0:00:14  0:00:14 --:--:--  118k
[root@nsk nsk]# python get-pip.py
Collecting pip
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 113kB/s
Collecting setuptools
  Downloading setuptools-38.2.5-py2.py3-none-any.whl (489kB)
    100% |████████████████████████████████| 491kB 115kB/s
Collecting wheel
  Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB)
    100% |████████████████████████████████| 51kB 125kB/s
Installing collected packages: pip, setuptools, wheel
Successfully installed pip-9.0.1 setuptools-38.2.5 wheel-0.30.0

Once you have installed pip, install the paramiko, PyYAML, jinja2, and httplib2 packages using the following command lines:

[root@nsk nsk]# pip install paramiko PyYAML jinja2 httplib2
Collecting paramiko
  Downloading paramiko-2.4.0-py2.py3-none-any.whl (192kB)
    100% |████████████████████████████████| 194kB 127kB/s
Collecting PyYAML
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% |████████████████████████████████| 256kB 125kB/s
Collecting jinja2
  Downloading Jinja2-2.10-py2.py3-none-any.whl (126kB)
    100% |████████████████████████████████| 133kB 126kB/s
Collecting httplib2
  Downloading httplib2-0.10.3.tar.gz (204kB)
    100% |████████████████████████████████| 204kB 131kB/s
Collecting pyasn1>=0.1.7 (from paramiko)
  Downloading pyasn1-0.4.2-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 117kB/s
Collecting bcrypt>=3.1.3 (from paramiko)
  Downloading bcrypt-3.1.4-cp27-cp27mu-manylinux1_x86_64.whl (57kB)
    100% |████████████████████████████████| 61kB 174kB/s
Collecting cryptography>=1.5 (from paramiko)
  Downloading cryptography-2.1.4-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)
    100% |████████████████████████████████| 2.2MB 75kB/s
Collecting pynacl>=1.0.1 (from paramiko)
  Downloading PyNaCl-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl (696kB)
    100% |████████████████████████████████| 706kB 106kB/s
Collecting MarkupSafe>=0.23 (from jinja2)
  Downloading MarkupSafe-1.0.tar.gz
Collecting six>=1.4.1 (from bcrypt>=3.1.3->paramiko)
  Downloading six-1.11.0-py2.py3-none-any.whl
Collecting cffi>=1.1 (from bcrypt>=3.1.3->paramiko)
  Downloading cffi-1.11.2-cp27-cp27mu-manylinux1_x86_64.whl (405kB)
    100% |████████████████████████████████| 409kB 124kB/s
Collecting enum34; python_version < "3" (from cryptography>=1.5->paramiko)
  Downloading enum34-1.1.6-py2-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.5->paramiko)
  Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
    100% |████████████████████████████████| 102kB 123kB/s
Collecting idna>=2.1 (from cryptography>=1.5->paramiko)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 134kB/s
Collecting ipaddress; python_version < "3" (from cryptography>=1.5->paramiko)
  Downloading ipaddress-1.0.19.tar.gz
Collecting pycparser (from cffi>=1.1->bcrypt>=3.1.3->paramiko)
  Downloading pycparser-2.18.tar.gz (245kB)
    100% |████████████████████████████████| 256kB 138kB/s
Building wheels for collected packages: PyYAML, httplib2, MarkupSafe, ipaddress, pycparser
  Running setup.py bdist_wheel for PyYAML ... done
  Stored in directory: /root/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
  Running setup.py bdist_wheel for httplib2 ... done
  Stored in directory: /root/.cache/pip/wheels/ca/ac/5f/749651f7925b231103f5316cacca82a487810c22d30f011c0c
  Running setup.py bdist_wheel for MarkupSafe ... done
  Stored in directory: /root/.cache/pip/wheels/88/a7/30/e39a54a87bcbe25308fa3ca64e8ddc75d9b3e5afa21ee32d57
  Running setup.py bdist_wheel for ipaddress ... done
  Stored in directory: /root/.cache/pip/wheels/d7/6b/69/666188e8101897abb2e115d408d139a372bdf6bfa7abb5aef5
  Running setup.py bdist_wheel for pycparser ... done
  Stored in directory: /root/.cache/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built PyYAML httplib2 MarkupSafe ipaddress pycparser
Installing collected packages: pyasn1, six, pycparser, cffi, bcrypt, enum34, asn1crypto, idna, ipaddress, cryptography, pynacl, paramiko, PyYAML, MarkupSafe, jinja2, httplib2
Successfully installed MarkupSafe-1.0 PyYAML-3.12 asn1crypto-0.24.0 bcrypt-3.1.4 cffi-1.11.2 cryptography-2.1.4 enum34-1.1.6 httplib2-0.10.3 idna-2.6 ipaddress-1.0.19 jinja2-2.10 paramiko-2.4.0 pyasn1-0.4.2 pycparser-2.18 pynacl-1.2.1 six-1.11.0
[root@nsk nsk]#

Note
By default, Ansible will be running against the development branch. You might want to check out the latest stable branch. Check what the latest stable version is using the following command line:

[root@nsk ansible]# git branch -a
* devel
  remotes/origin/HEAD -> origin/devel
  remotes/origin/devel
  remotes/origin/release1.5.0
  remotes/origin/release1.5.1
..
  remotes/origin/stable-2.0-network
  remotes/origin/stable-2.0.0.1
  remotes/origin/stable-2.1
  remotes/origin/stable-2.2
  remotes/origin/stable-2.3
  remotes/origin/stable-2.4
  remotes/origin/temp-staging-post-2.3.3
  remotes/origin/threading_instead_of_forking
  remotes/origin/threading_plus_forking
[root@nsk ansible]#

Copy the latest version you want to use. Version 2.4 was the latest version available at the time of writing. Check the latest version using the following command lines:

[root@nsk ansible]# git checkout stable-2.4
Branch stable-2.4 set up to track remote branch stable-2.4 from origin.
Switched to a new branch 'stable-2.4'

[root@nsk nsk]# ansible --version
ansible 2.4.3.0 (stable-2.3 8708105616) last updated 2017/12/28 08:55:50 (GMT +550)
  config file = None
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/nsk/ansible/lib/ansible
  executable location = /home/nsk/ansible/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]

You now have a working setup of Ansible ready.

Note : One of the benefits of running Ansible from source is that you can enjoy the new features immediately, without waiting for your package manager to make them available for you.

Wednesday, December 27, 2017

Monitoring services using journalctl in RHEL7

Wednesday, December 27, 2017 0

Monitoring services using journalctl in RHEL7


Monitoring services using journalctl in RHEL7

Systemd's journal has the added advantage that its controls allow you to easily narrow down on messages generated by specific services.

1. First, display all the messages generated by your system.
This will show all the messages generated on the system; run the following commands:

[root@nsk ~]# journalctl
-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:32:09 IST. --
Dec 26 07:31:23 nsk systemd-journal[89]: Runtime journal is using 8.0M (max allowed 91.9M, trying to leave 137.9M free of 911.6M available → current limit 91.
Dec 26 07:31:23 nsk kernel: Initializing cgroup subsys cpuset
Dec 26 07:31:23 nsk kernel: Initializing cgroup subsys cpu
Dec 26 07:31:23 nsk kernel: Initializing cgroup subsys cpuacct
Dec 26 07:31:23 nsk kernel: Linux version 3.10.0-693.5.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #
Dec 26 07:31:23 nsk kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.5.2.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root
Dec 26 07:31:23 nsk kernel: e820: BIOS-provided physical RAM map:
Dec 26 07:31:23 nsk kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Dec 26 07:31:23 nsk kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved…...
…………..
~]#
2. Now, display all system-related messages.
This command shows all the messages related to the system and not its users:
[root@nsk ~]# journalctl –-system

3. Display all the current user messages.
This command shows all messages related to the user that you are logged on with:
[root@nsk ~]# journalctl –-user

4. Next, display all messages generated by a particular service using the following command line:
journalctl --unit=<service>

[root@nsk ~]# journalctl --unit=sshd
-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --
Dec 26 07:31:29 nsk systemd[1]: Starting OpenSSH server daemon...
Dec 26 07:31:29 nsk sshd[944]: Server listening on 0.0.0.0 port 22.
Dec 26 07:31:29 nsk sshd[944]: Server listening on :: port 22.
Dec 26 07:31:29 nsk systemd[1]: Started OpenSSH server daemon.
Dec 26 07:33:37 nsk sshd[1238]: Accepted password for root from 10.0.2.2 port 60698 ssh2
Dec 26 07:34:11 nsk sshd[1261]: Accepted password for root from 10.0.2.2 port 60702 ssh2
Dec 26 08:30:19 nsk systemd[1]: Stopping OpenSSH server daemon...
……..

6. Now, display messages by priority.

Priorities can be specified by a keyword or number, such as debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0). When specifying a priority, this includes all the lower priorities as well. For example, err implies that crit, alert, and emerg are also shown. Take a look at the following command line:
journalctl -p <priority>

[root@nsk ~]# journalctl -p err
-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --
Dec 26 08:26:15 nsk rsyslogd[613]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
Dec 26 08:30:21 nsk lvmetad[483]: Failed to accept connection errno 11.

7. Next, display messages by time.
You can show all messages from the current boot through the following commands:

[root@nsk ~]# journalctl -b
-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --
Dec 26 08:30:45 nsk systemd-journal[86]: Runtime journal is using 8.0M (max allowed 91.9M, trying to leave 137.9M free of 911.6M available → current limit 91.
Dec 26 08:30:45 nsk kernel: Initializing cgroup subsys cpuset
Dec 26 08:30:45 nsk kernel: Initializing cgroup subsys cpu
Dec 26 08:30:45 nsk kernel: Initializing cgroup subsys cpuacct
Dec 26 08:30:45 nsk kernel: Linux version 3.10.0-693.5.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #
Dec 26 08:30:45 nsk kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.5.2.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root
Dec 26 08:30:45 nsk kernel: e820: BIOS-provided physical RAM map:
You can even show all the messages within a specific time range by running the following:
[root@nsk ~]# journalctl --since="2017-12-26 08:30:00" --until="2017-12-26 09:00:00"
-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --
Dec 26 08:30:19 nsk polkitd[619]: Registered Authentication Agent for unix-process:1587:353648 (system bus name :1.40 [/usr/bin/pkttyagent --notify-fd 5 --fal
Dec 26 08:30:19 nsk ntpd[1551]: ntpd exiting on signal 15
Dec 26 08:30:19 nsk systemd[1]: Stopped target Network is Online.
Dec 26 08:30:19 nsk sshd[944]: Received signal 15; terminating.
Dec 26 08:30:19 nsk systemd[1]: Stopping Network is Online.
Dec 26 08:30:19 nsk crond[628]: (CRON) INFO (Shutting down)
…….

For instance, if you want to show all the error messages between 8:30 and 9:00 on 2017-12-26, your command would be the following:

[root@nsk ~]# journalctl -p err --since="2017-12-26 08:30:00" --until="2017-12-26 09:00:00"
-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --
Dec 26 08:30:21 nsk lvmetad[483]: Failed to accept connection errno 11.
[root@nsk ~]#

The journalctl binary is an executable one, so it is impossible to use the traditional "following" techniques such as tail –f or using less and pressing CTRL + F. Simply add -f or --follow as an argument to the journalctl command.

[root@nsk ~]# journalctl -f
-- Logs begin at Tue 2017-12-26 07:31:23 IST. --
Dec 26 08:30:53 nsk systemd[1]: Started Crash recovery kernel arming.
Dec 26 08:30:53 nsk systemd[1]: Startup finished in 392ms (kernel) + 1.702s (initrd) + 6.681s (userspace) = 8.777s.
Dec 26 08:32:08 nsk sshd[1259]: Accepted password for root from 10.0.2.2 port 63824 ssh2
Dec 26 08:32:09 nsk systemd[1]: Created slice User Slice of root.
Dec 26 08:32:09 nsk systemd[1]: Starting User Slice of root.
Although most environments are used to create syslog messages to troubleshoot, the journal does provide the added value of being able to create simple filters that allow you to monitor their messages live.

Tuesday, December 26, 2017

Configuring the systemd journal for persistence in RHEL7

Tuesday, December 26, 2017 0

Configuring the systemd journal for persistence in RHEL7

By default, the journal doesn't store log files on disk, only in memory or the /run/log/journal directory. This is sufficient for the recent log history (with the journal) but not for long-term log retention should you decide to go with journal only and not with any other syslog solution.

Configuring journald to keep more logs than memory allows is fairly simple, as follows:
Open /etc/systemd/journald.conf with your favorite text editor with root permissions by executing the following command:
vim /etc/systemd/journald.conf

Ensure that the line containing Storage is either remarked or set to auto or persistent and save it, as follows:

[root@nsk ~]# vim /etc/systemd/journald.conf
#  This file is part of systemd.
...
..
[Journal]
Storage=auto

If you select auto, the journal directory needs to be manually created. The following command would be useful for this:

[root@nsk ~]# mkdir -p /var/log/journal
[root@nsk ~]#

Now, restart the journal service by executing the following command:

[root@nsk ~]# systemctl restart systemd-journald

There are many other options that can be set for the journal daemon.
By default, all the data stored by journald is compressed, but you could disable this using Compress=no.
It is recommended to limit the size of the journal files by either specifying a maximum retention age (MaxRetentionSec), a global maximum size usage (SystemMaxUse), or a maximum size usage per file (SystemMaxFileSize).


Monday, December 25, 2017

Screen command in Linux

Monday, December 25, 2017 0

Screen command in Linux


NAME
       screen - screen manager with VT100/ANSI terminal emulation

Screen  is  a full-screen window manager that multiplexes a physical terminal between several processes. When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can  use the  program  as  you  normally  would.  Then,  at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill existing windows, view a list of windows, turn output logging on and off, copy-and-paste text between windows, view  the  scrollback history,  switch  between  windows in whatever manner you wish, etc. All windows run their programs completely independent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the user’s  terminal.   When  a program  terminates,  screen  (per default) kills the window that contained it.  If this window was in the foreground, the display switches to the previous window; if none are left, screen exits.

#screen -h 2000 -t SERVER_UPDATE -S SERVER_UPDATE

Here
  -h  num - Specifies the history scrollback buffer to be num lines high.
  -t  name - sets the title for the default shell or specified program. 
  -S  sessionname  -  When  creating  a  new  session,  this  option can be used to specify a meaningful name for the session. This name identifies the session for
            "screen -list" and "screen -r" actions. It substitutes the default [tty.host] suffix.

Prints a list of screen session
#screen -ls
There is a screen on:
        27137.pts-1.testserver       (Attached)
1 Socket in /var/run/screen/S-root.

Detach the screen session
#screen -D 27137.pts-1.testserver
[27137.pts-1.testserver power detached.]

Attach the screen session
#screen -r 27137.pts-1.testserver

Please check man screen page for more options.

Sunday, December 24, 2017

Configuring logrotate for yum in RHEL7

Sunday, December 24, 2017 0

Configuring logrotate for yum in RHEL7


Every time you use yum to install and/or update packages, it logs to /var/log/yum.log. 

I do recommend keeping your complete yum history as it doesn't grow a lot, unless you reinstall packages a lot.

For a rich interface to your yum history, I suggest you use yum history.

By default, your yum log file is rotated yearly, and even then, it only rotates if the size of your log file exceeds 30 KB, and your logs are only kept for 4 years. 

virtual servers have the potential to stay "alive" beyond these 3-4 years.

How to do it…
Modify /etc/logrotate.d/yum to the following:

/var/log/yum.log {
    missingok
    notifempty
    size 30k
    rotate 100
    yearly
    create 0600 root root
}

How it works…
This configuration will only rotate the yum log when it exceeds 30 KB in size on a yearly basis, and it will keep 100 rotated logs, which is basically log files for 100 years!

Friday, December 22, 2017

Install KVM in RHEL7

Friday, December 22, 2017 0

Install KVM in RHEL7

By default, a RHEL 7 system doesn't come with a KVM or libvirt preinstalled. This can be installed in three ways:

Through the graphical setup during the system's setupVia a kickstart installationThrough a manual installation from the command line


To install a KVM, you will require at least 6 GB of free disk space, 2 GB of RAM, and an additional core or thread per guest.

Check whether your CPU supports a virtualization flag (such as SVM or VMX). Some hardware vendors disable this in the BIOS, so you may want to check your BIOS as well. Run the following command:

# grep -E 'svm|vmx' /proc/cpuinfo
flags    : ... svm ...
Check whether the hardware virtualization modules (such as kvm_intel and kvm) are loaded in the kernel using the following command:

# lsmod | grep kvm
kvm_intel             155648  0
kvm                      495616  1 kvm_intel

Manual installation
This way of installing a KVM is generally done once the base system is installed by some other means. 

Install the software needed to provide an environment to host virtualized guests with the following command:
# yum -y install qemu-kvm qemu-img libvirt

The installation of these packages will include quite a lot of dependencies.

Install additional utilities required to configure libvirt and install virtual machines by running this command:
# yum -y install virt-install libvirt-python python-virthost libvirt-client

By default, the libvirt daemon is marked to autostart on each boot. Check whether it is enabled by executing the following command:
# systemctl status libvirtd

libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: inactive
   Docs: man:libvirtd(8)
    http://libvirt.org

If for some reason this is not the case, mark it for autostart by executing the following:
# systemctl enable libvirtd
To manually stop/start/restart the libvirt daemon, this is what you'll need to execute:
# systemctl stop libvirtd
# systemctl start libvirtd
# systemctl restart libvirtd

Kickstart installation
Installing a KVM during kickstart offers you an easy way to automate the installation of KVM instances. 

Add the following package groups to your kickstarted file in the %packages section:
@virtualization-hypervisor
@virtualization-client
@virtualization-platform
@virtualization-tools
Start the installation of your host with this kickstart file.

Graphical setup during the system's setup
This is probably the least common way of installing a KVM. The only time I used this was during the course of writing this recipe. Here's how you can do this:

Boot from the RHEL 7 Installation media.
Complete all steps besides the Software selection step.
Go to Software Selection to complete the KVM software selection.
Select the Virtualization host radio button in Base Environment, and check the Virtualization Platform checkbox in Add-Ons for Selected Environment:
Finalize the installation.
On the Installation Summary screen, complete any other steps and click on Begin Installation.

Wednesday, December 20, 2017

ssh_exchange_identification: Connection closed by remote host - Password less authentication setup

Wednesday, December 20, 2017 0

ssh_exchange_identification: Connection closed by remote host - Password less authentication setup 


Situation:
While setup passwordless authentication from testserver2 (192.181.166.55) to testserver1 (192.181.130.55)  server, getting  error "ssh_exchange_identification: Connection closed by remote host"

[kanachim@testserver2 .ssh]$ ssh -vv testserver1
OpenSSH_5.3p1-hpn13v7, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.181.166.55 [192.181.166.55] port 22.
debug1: Connection established.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /home/kanachim/.ssh/id_rsa type 1
debug1: identity file /home/kanachim/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host

Solution:
Check /etc/hosts.deny on server testserver1

root@host testserver1# grep sshd /etc/hosts.deny 
# DenyHosts: Mon Dec 18 22:10:38 2017 | sshd: 192.181.166.55
sshd: 192.181.166.55

Remove the sshd entry from hosts.deny on testserver1

Login to testserver2 Switch to user kanachim & create new key

-bash-3.2$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/kanachim/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/kanachim/.ssh/id_rsa.
Your public key has been saved in/home/kanachim/.ssh/id_rsa.pub.
The key fingerprint is:
37:6a:69:c1:a4:01:4b:c5:34:be:32:42:90:0b:20:a0 kanachim@192.181.166.55
The key's randomart image is:
+--[ RSA 2048]----+
|B.  o++          |
|=. . +..         |
|E.. . o .        |
|..     *         |
|  . o o S o      |
|   . o   = .     |
|        =        |
|       o         |
|                 |
+-----------------+
-bash-3.2$
-bash-3.2$ ssh-copy-id kanachim@192.181.130.55
Password:
Now try logging into the machine, with "ssh 'kanachim@192.181.130.55'", and check in:
  .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

Now login from testserver2 to testserver1

-bash-3.2$ ssh 192.181.130.55
Last login: Thu Feb 27 00:30:38 2014 from 10.217.230.145
-bash-3.2$ hostname
testserver1

Tuesday, December 19, 2017

Recover RPM DB from a corrupted RPM database in RHEL 7

Tuesday, December 19, 2017 0

Rebuilding corrupted rpm database in RHEL7

Situation:
Although everything is done to ensure that your RPM databases are intact, your RPM database may become corrupt and unuseable. This happens mainly if the filesystem on which the rpm db resides is suddenly inaccessible (full, read-only, reboot, or so on).

Solution:
1.Start by creating a backup of your corrupt rpm db, as follows:
[root@nsk ~]# tar zcvf rpm-db.tar.gz /var/lib/rpm/*
tar: Removing leading `/' from member names
/var/lib/rpm/Basenames
/var/lib/rpm/Conflictname
/var/lib/rpm/__db.001
/var/lib/rpm/__db.002
/var/lib/rpm/__db.003
/var/lib/rpm/Dirnames
/var/lib/rpm/Group
/var/lib/rpm/Installtid
/var/lib/rpm/Name
/var/lib/rpm/Obsoletename
/var/lib/rpm/Packages
/var/lib/rpm/Providename
/var/lib/rpm/Requirename
/var/lib/rpm/Sha1header
/var/lib/rpm/Sigmd5
/var/lib/rpm/Triggername

2.Remove stale lock files if they exist through the following command:
[root@nsk ~]# rm -f /var/lib/rpm/__db*

3.Now, verify the integrity of the Packages database via the following:
[root@nsk ~]# /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages; echo $?
BDB5105 Verification of /var/lib/rpm/Packages succeeded.
0

If it prints 0, proceed to next step.

4. Rename the Packages file (don't delete it, we'll need it!), as follows:
[root@nsk ~]# mv /var/lib/rpm/Packages  /var/lib/rpm/Packages.org

5. Now, dump the Packages db from the original Packages db by executing the following command:
[root@nsk ~]# cd /var/lib/rpm/
 [root@nsk rpm]# /usr/lib/rpm/rpmdb_dump Packages.org | /usr/lib/rpm/rpmdb_load Packages
rpmdb_load: BDB1540 configured environment flags incompatible with existing environment

6.Verify the integrity of the newly created Packages database. Run the following:
[root@nsk rpm]#  /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages; echo $?
BDB5105 Verification of /var/lib/rpm/Packages succeeded.
0

If the exit code is not 0, you will need to restore the database from backup.

7. Rebuild the rpm indexes, as follows:
[root@nsk ~]# rpm -vv --rebuilddb
[root@nsk rpm]# rpm -vv --rebuilddb
D: rebuilding database /var/lib/rpm into /var/lib/rpmrebuilddb.1312
D: opening  db environment /var/lib/rpm private:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db environment /var/lib/rpmrebuilddb.1312 private:0x401
D: opening  db index       /var/lib/rpmrebuilddb.1312/Packages (none) mode=0x42
D: opening  db index       /var/lib/rpmrebuilddb.1312/Packages 0x1 mode=0x42
D: disabling fsync on database
....
...
D: adding "5f7fd424d0773a4202731bff4901d449699b0929" to Sha1header index.
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm
D: closed   db index       /var/lib/rpmrebuilddb.1312/Sha1header
D: closed   db index       /var/lib/rpmrebuilddb.1312/Sigmd5
D: closed   db index       /var/lib/rpmrebuilddb.1312/Installtid
D: closed   db index       /var/lib/rpmrebuilddb.1312/Dirnames
D: closed   db index       /var/lib/rpmrebuilddb.1312/Triggername
D: closed   db index       /var/lib/rpmrebuilddb.1312/Obsoletename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Conflictname
D: closed   db index       /var/lib/rpmrebuilddb.1312/Providename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Requirename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Group
D: closed   db index       /var/lib/rpmrebuilddb.1312/Basenames
D: closed   db index       /var/lib/rpmrebuilddb.1312/Name
D: closed   db index       /var/lib/rpmrebuilddb.1312/Packages
D: closed   db environment /var/lib/rpmrebuilddb.1312

8. Use the following command to check the rpm db with yum for any other issues (this may take a long time):
[root@nsk rpm]# yum check
Loaded plugins: fastestmirror
....
...

9. Restore the SELinux context of the rpm database through the following command:
[root@nsk rpm]# restorecon -R -v /var/lib/rpm

Monday, December 18, 2017

How to reboot the Xen Virtual Machine when was at hung state.

Monday, December 18, 2017 0

How to reboot the Xen Virtual Machine when was at hung state.

If console is not working from the host(Dom0), the console was opened in order to use sysrq magic packages
#xm console xenvm006

From other terminal on the node, run the below  comamnd one by one.
#xm sysrq xenvm006  h
#xm sysrq xenvm006  m    #should show the amount of memory been used
#xm sysrq xenvm006  t      #should show the current tasks

the output is  "console is opened and was not good"  also   "vm was with out resources"
Check the xentop, if  a high cpu usage is shown like 200%

Then reboot the virtual machine.

Open one more terminal & run the below command.

#xm destroy  xenvm006

if the virtual machine is running under cluster,  first disable the vm from cluster to avoid failover.

#clusvcadm -d vm:xenvm006

Then destroy the virtual machine.

Because as the vm is hung state, clusvcadm -d wont do a clean shutdown, so we need to destroy it manually.

#xm create xenvm006  (will start the vm)
#clusvcadm -e vm:xenvm006   (with cluster)

Sunday, December 17, 2017

Understanding the LVM Configuration Files in Linux

Sunday, December 17, 2017 0

Understanding the LVM Configuration Files in Linux

The following files are part of LVM configuration:

/etc/lvm/lvm.conf
    Central configuration file read by the tools.

etc/lvm/lvm_hosttag.conf
    For each host tag, an extra configuration file is read if it exists: lvm_hosttag.conf. If that file defines new tags,
    then further configuration files will be appended to the list of tiles to read in.
In addition to the LVM configuration files, a system running LVM includes the following files that affect LVM system setup:

/etc/lvm/cache
    Device name filter cache file (configurable).

/etc/lvm/backup/
    Directory for automatic volume group metadata backups (configurable).

/etc/lvm/archive/
    Directory for automatic volume group metadata archives (configurable with regard to directory path and archive history depth).

/var/lock/lvm/
    In single-host configuration, lock files to prevent parallel tool runs from corrupting the metadata; in a cluster, cluster-wide DLM is used.

How to monitor the permission change and ownership change of a particular directory or file in Linux Server

Sunday, December 17, 2017 0

Monitoring the permission change and ownership change of a particular directory or file in Linux Server

1. Use the audit package to accomplish this task.2. Ensure the auditd service is running, and set to start on boot chkconfig auditd on3. Set a watch on the required file to be monitored by using the auditctl command:

# auditctl -w /etc/passwd -p war -k monitor-passwd

here:
    auditctl   :the command used to add entries to the audit database.
    -w            :Insert a watch for the file system object at path, i.e. /etc/shadow.
    -p             :Set permissions filter for a file system watch. r=read, w=write, x=execute, a=attribute change.
    -k             : Set a filter key on an audit rule. The filter key is an arbitrary string of text that can be up to 31 bytes long. 

It can uniquely identify the audit records produced by a rule.

Note that you must add the rule to /etc/audit/audit.rules on RHEL5 or RHEL6  in order for them to persist after reboot.
    You can place the watch rule in the /etc/audit/audit.rules file to set permanently:
-w /etc/passwd -p a -k monitor-passwd
       Check auditd service and if it is stopped, start it.

# service auditd status
# chkconfig --list auditd
# chkconfig auditd on

In this example, a watch is placed on the /etc/passwd file for any syscalls which perform a write, read, or attribute change
    (-p war). This is logged with the key monitor-passwd. This key can be used to search through the audit logs to find these actions, 
 using the ausearch command:

# ausearch -ts today -k monitor-passwd
----
time->Sat May  3 07:32:20 2009
type=PATH msg=audit(117045140.872:34): item=0 name="/etc/passwd" inode=1308742 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0
type=CWD msg=audit(117045140.872:34): cwd="/root"
type=SYSCALL msg=audit(117045140.872:34): arch=40000003 syscall=226 success=yes exit=0 a0=867c4b8 a1=458bcc4f a2=8686800 a3=1c
items=1 ppid=3544 pid=3558 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 comm="vim" exe="/usr/bin/vim"
subj=root:system_r:unconfined_t:s0-s0:c0.c1023 key="monitor-passwd"

    From this trace, it can be seen that the file /etc/resolve.conf was edited using the /usr/bin/vim command. The user that ran the  command was running with the root:system_r:unconfined_t:s0-s0:c0.c1023 SELinux context. And, the timestamp can be converted into readable form.

# date -d @117045140
Sat May  3 05:32:20 CST 2009

    You can search for an event based on the given key string
# ausearch -k monitor-passwd
    For a clearly view, you can generate report base on audit rule keys
# aureport -k