This Blog is to share our knowledge and expertise on Linux System Administration and VMware Administration

Showing posts with label Linux General. Show all posts
Showing posts with label Linux General. Show all posts

Monday, October 15, 2018

How to generate sosreport in different directory instead of default directory

Monday, October 15, 2018 0
Sosreport is a tool that collects configuration details, system information, diagnostic information and system logs from Linux server. Sosreport  generates  an archive report from the running system. The archive may be stored locally or centrally for recording or tracking purposes.

Normally sosreport will be saved under /tmp (RHEL6) /var/tmp (RHEL7) directory on linux server. Some time we need to save the sosreport to some other temporary directory because of space issue. 

Below steps are used to generate sosreport in different directory.

RHEL 5.6 and earlier

We need to set the temporary path environment before running sosreport.

[root@nsk ~]# env TMP=/path/to/target/directory sosreport

RHEL 5.7 and above

We have to use option --tmp-dir with sosreport to redirect the sosreport path 

[root@nsk ~]# sosreport --tmp-dir /path/to/target/directory

Sosreport may have hung because of a specific plugin. We can list all available sosreport plugin modules by below command

[root@nsk ~]# sosreport -l

Identify which plugin is hanging on by below command

[root@nsk ~]# sosreport -vvvv 

If a plugin times out or last plugin does not finish, rerun the sosreport and skip that plugin

[root@nsk ~]# sosreport -v -n plugin_name --tmp-dir /path/to/target/directory

Friday, October 5, 2018

Digging audit log with ausearch tool in RHEL 7

Friday, October 05, 2018 0
ausearch is a tool to search audit daemon logs based upon the events based on different search criteria.

 Audit system stores log entries in the /var/log/audit/audit.log file; if log rotation is enabled, rotated audit.log files are stored in the same directory.

Each event consists of three records, which share the same time stamp. Each record consists of several name=value pairs separated by a white space or a comma. 

type=SYSCALL msg=audit(1510471123.129:36): arch=c000003e syscall=175 success=yes exit=0 a0=1901a20 a1=1c5d a2=41a2d8 a3=18fe400 items=0 ppid=7 24 pid=725 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmo d" subj=system_u:system_r:insmod_t:s0 key=(null)
type=PROCTITLE msg=audit(1510471123.129:36): proctitle=2F7362696E2F6D6F6470726F6265002D71002D2D0069707461626C655F6D616E676C65
type=NETFILTER_CFG msg=audit(1510471123.276:37): table=nat family=2 entries=0

Here, ausearch utility allows us to search Audit log files for specific events. Also it is used to read the audit log timestamp to user readable timestamp by using -i (--interpret) option.

Example:

1.Search for hostname with given hostname specific.

[root@nsk ~]# ausearch -i -hn nsk.testingserver.internal
----
type=USER_AUTH msg=audit(10/04/2018 12:32:02.610:75) : pid=1617 uid=root auid=unset ses=unset subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_permit acct=gdm exe=/usr/libexec/gdm-session-worker hostname=nsk.testingserver.internal addr=? terminal=/dev/tty1 res=success'
----

2. Search only the first event that matches the search pattern.

[root@nsk ~]# ausearch -i -hn nsk.testingserver.internal --just-one
----
type=USER_AUTH msg=audit(10/04/2018 12:32:02.610:75) : pid=1617 uid=root auid=unset ses=unset subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_permit acct=gdm exe=/usr/libexec/gdm-session-worker hostname=nsk.testingserver.internal addr=? terminal=/dev/tty1 res=success'
[root@nsk ~]#

3. Search for all logged actions performed by user nsk, using the user's login ID (auid). 

[root@nsk log]# ausearch -ua 1001 -i
----
type=LOGIN msg=audit(12/04/2017 11:39:18.052:156) : pid=1433 uid=root subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=unset auid=nsk tty=(none) old-ses=4294967295 ses=2 res=yes
----
type=USER_ROLE_CHANGE msg=audit(12/04/2017 11:39:18.991:157) : pid=1433 uid=root auid=nsk ses=2 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 selected-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 exe=/usr/sbin/sshd hostname=nsk addr=fe80::f24d:eb49:d219:6b0 terminal=ssh res=success'

4. Search for an event matching the given Process ID.

[root@nsk ~]# ausearch -i -p 2162
----
type=CRYPTO_SESSION msg=audit(10/05/2018 13:07:54.052:118) : pid=2162 uid=root auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-server cipher=aes256-ctr ksize=256 mac=hmac-sha2-256 pfs=curve25519-sha256@libssh.org spid=2163 suid=sshd rport=50935 laddr=10.0.2.15 lport=22  exe=/usr/sbin/sshd hostname=? addr=10.0.2.2 terminal=? res=success'
----

5. Search the /var/log/audit/audit.log file for successful login attempts

[root@nsk log]# ausearch --message USER_LOGIN --success yes --interpret
----
type=USER_LOGIN msg=audit(11/12/2017 13:04:08.279:110) : pid=662 uid=root auid=root ses=2 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023
msg='op=login id=root exe=/usr/bin/login hostname=nsk addr=? terminal=tty1 res=success'
----
type=USER_LOGIN msg=audit(11/12/2017 13:05:46.297:107) : pid=1242 uid=root auid=root ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c
0.c1023 msg='op=login id=root exe=/usr/sbin/sshd hostname=10.0.2.2 addr=10.0.2.2 terminal=/dev/pts/0 res=success'
----

6. Search for all account and group activities.

[root@nsk log]# ausearch -m ADD_USER -m DEL_USER -m ADD_GROUP -m DEL_GROUP -i
----
type=ADD_GROUP msg=audit(11/15/2017 22:05:29.396:169) : pid=15764 uid=root auid=root ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-group id=dovecot exe=/usr/sbin/groupadd hostname=? addr=? terminal=? res=success'
----
type=ADD_USER msg=audit(11/15/2017 22:05:29.431:171) : pid=15769 uid=root auid=root ses=2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 msg='op=add-user id=dovecot exe=/usr/sbin/useradd hostname=? addr=? terminal=? res=success'

7. Search for all failed PROCTITLE from yesterday up until now

[root@nsk log]# ausearch --start yesterday --end now -m PROCTITLE -sv no -i
----
type=PROCTITLE msg=audit(10/03/2018 18:00:02.623:71) : proctitle=/usr/sbin/httpd -DFOREGROUND
type=SYSCALL msg=audit(10/03/2018 18:00:02.623:71) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x560fb5acc008 a1=O_RDONLY a2=0x1b6 a3=0x24 items=0 ppid=1 pid=1145 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=httpd exe=/usr/sbin/httpd subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(10/03/2018 18:00:02.623:71) : avc:  denied  { read } for  pid=1145 comm=httpd name=php-mapi.cfg dev="dm-0" ino=19498435 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:zarafa_etc_t:s0 tclass=file

Please refer man ausearch for more option.

Thursday, October 4, 2018

How to read/convert Audit log timestamp in RHEL7

Thursday, October 04, 2018 0
By default, audit log would be like below.

type=SYSCALL msg=audit(1510471123.129:36): arch=c000003e syscall=175 success=yes exit=0 a0=1901a20 a1=1c5d a2=41a2d8 a3=18fe400 items=0 ppid=7 24 pid=725 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmo d" subj=system_u:system_r:insmod_t:s0 key=(null)
type=PROCTITLE msg=audit(1510471123.129:36): proctitle=2F7362696E2F6D6F6470726F6265002D71002D2D0069707461626C655F6D616E676C65 type=NETFILTER_CFG msg=audit(1510471123.276:37): table=nat family=2 entries=0

Below commands are used to convert the auditlog timestamp to user readable timestamp.

1. AUSEARCH : 

ausearch utility allows us to search Audit log files for specific events. Also it is used to read the audit log epoch timestamp to user readable timestamp.By default, ausearch searches the /var/log/audit/audit.log file. We can specify a different file using the ausearch options -if file_name command

[root@nsk log]# ausearch -i | grep -i CONFIG
type=CONFIG_CHANGE msg=audit(11/12/2017 12:48:40.357:5) : audit_backlog_limit=8192 old=64 auid=unset ses=unset subj=system_u:system_r:unconfined_service_t:s0 res=yes
type=CONFIG_CHANGE msg=audit(11/12/2017 12:48:40.357:6) : audit_failure=1 old=1 auid=unset ses=unset subj=system_u:system_r:unconfined_service_t:s0 res=yes

-i, --interpret : Interpret numeric entities into text

2. DATE :  

A) Date command with %s is used to convert normal time to epoch time.
%s : seconds since 1970-01-01 00:00:00 UTC

Display the current time in the given FORMAT, or set the system date
[root@nsk audit]# date +%s
1538570770
[root@nsk audit]# date +%s
1538570773

B) Date command with -d@ is used to convert epoch time to normal time.
[root@nsk audit]# date -d@1538570776
Wed Oct  3 18:16:16 IST 2018
[root@nsk audit]#

C) If we want to convert the specific system time to epoch time.
[root@nsk audit]# date --date="Wed Oct  3 18:16:16 IST 2018" +%s
1538570776

3. PERL : Perl was originally a language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information

[root@nsk audit]# perl -pe 's/(\d+)/localtime/e' audit.log  | more
type=DAEMON_START msg=audit(Sun Nov Wed Oct  3 20:22:26 2018 12:48:40 2017.186:6974): op=start ver=2.7.6 format=raw kernel=3.10.0-693.el7.x86_64 auid=4294967295 pid=606 uid=0 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=success
type=CONFIG_CHANGE msg=audit(Sun Nov Wed Oct  3 20:22:26 2018 12:48:40 2017.357:5): audit_backlog_limit=8192 old=64 auid=4294967295 ses=429496 7295 subj=system_u:system_r:unconfined_service_t:s0 res=1

Here,
-p : Print out the pattern space
-e : command (This command allows one to pipe input from a shell command into pattern space)
localtime         : it is perl function, it is having 9 elements.
-d : Match 1 or more repetition of digits

Friday, September 14, 2018

python: SELinux is preventing httpd from connectto access on the unix_stream_socket /var/lib/mysql/mysql.sock - Mariadb running on RHEL 7

Friday, September 14, 2018 0
SELinux is preventing the httpd access by default.

Situation
While connecting mysql database from web, getting this error.
 python: SELinux is preventing httpd from connectto access on the unix_stream_socket /var/lib/mysql/mysql.sock

Solution
Run below command to allow httpd in Selinux.

[root@nsk ~]#  ausearch -c 'httpd' --raw | audit2allow -M my-httpd
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i my-httpd.pp

[root@nsk ~]# semodule -i my-httpd.pp

Hope it helps.

Tuesday, September 4, 2018

Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again - SOLVED

Tuesday, September 04, 2018 0
Situation : I have installed epel-release-latest-6.noarch.rpm on CentOS release 6.10 (Final) and trying to install 
python-pip, but getting the below error.

[root@puppetlabs tmp]# rpm -ivh epel-release-latest-6.noarch.rpm
warning: epel-release-latest-6.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing...                ########################################### [100%]
   1:epel-release         ###########################################  [100%]
[root@puppetlabs tmp]#
[root@puppetlabs tmp]# yum install python-pip
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again

Solution:

Edit both /etc/yum.repos.d/epel.repo and /etc/yum.repos.d/epel-testing.repo files, commenting all entries 
starting with mirrorlist=  and uncomment all the entries starting with baseurl= 

epel.repo base url is changed from http://download.fedoraproject.org/pub/epel/6/  to http://del-repos.extreme-ix.org/epel/6/x86_64/repodata/ update the same.
epel-testing.repo base url is changed from http://del-repos.extreme-ix.org/epel/testing/6/ to http://del-repos.extreme-ix.org/epel/testing/6 update the same.

Now run yum repolist command.

[root@puppetlabs yum.repos.d]# yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.far.fi
 * extras: centos.mirror.far.fi
 * updates: centos.mirror.far.fi
repo id                 repo name                                                                        status
base                    CentOS-6 - Base                                                               6,713
epel                     Extra Packages for Enterprise Linux 6 - x86_64           12,522
extras                  CentOS-6 - Extras                                                                 31
updates               CentOS-6 - Updates                                                             114
repolist: 19,380

Hope it helps.