LINUX AND VMWARE STUFF

Situation:
Yum is  failed with Errno 256, No more mirrors to try.

root@testserver# yum list available | grep erna
https://rhn.test.com/pulp/repos/shared/lce-lcs_6month_rhel7/ccv-lcs_6month_rhel7/content/dist/rhel/server/7/7Server/x86_64/extras/os/repodata/repomd. 

       xml: [Errno 14] curl#58 - "SSL peer rejected your certificate as expired."
Trying other mirror.


One of the configured repositories failed (Red Hat Enterprise Linux 7 Server - Extras (RPMs)),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

....
...
..

    yum-config-manager --save --setopt=rhel-7-server-extras-rpms.skip_if_unavailable=true

failure: repodata/repomd.xml from rhel-7-server-extras-rpms: [Errno 256] No more mirrors to try.
https://rhn.test.com/pulp/repos/shared/lce-lcs_6month_rhel7/ccv-lcs_6month_rhel7/content/dist/rhel/server/7/7Server/x86_64/extras/os/repodata/repomd. 

       xml: [Errno 14] curl#58 - "SSL peer rejected your certificate as expired."


Solution:
Becasue of corrupted yum cache, getting this error.  So refresh the subscription-manager clean the old cache & run the yum command

root@testserver# subscription-manager refresh
All local data refreshed
root@testserver# yum clean all
root@testserver#
root@testserver# yum list available | grep erna
AdaptermFernablesung_15_hq.noarch
DatentauschInternational_hq.noarch
Fernablesung_2.37_hq.noarch 2.0.0.RELEASE-8 tec-x86_64-server-software-release-7
Fernablesung_Application_hq.noarch
Fernablesung_Comm_3.42_hq.noarch 5.4.1.RELEASE-92 tec-x86_64-server-software-release-7
Fernablesung_Helper_hq.noarch 7.1.5.RELEASE-126 tec-x86_64-server-software-release-7
Fernablesung_IF_3.08_hq.noarch 2.1.0.RELEASE-4 tec-x86_64-server-software-release-7
Fernablesung_IF_3.10_hq.noarch 6.1.1.RELEASE-45 tec-x86_64-server-software-release-7
Fernablesung_Tss3_hq.noarch 1.1.1.RELEASE-315 tec-x86_64-server-software-release-7
ems-AdaptermFernablesung_15_hq.noarch
ems-Fernablesung_hq.noarch 5.2.0.RELEASE-28 tec-x86_64-server-software-release-7
gnome-shell-extension-alternate-tab.noarch
gnome-shell-extension-alternative-status-menu.noarch
root@testserver#

Here we are using nmcli to configure the network teaming. 

nmtui : Text user Interface
nmcli : Command line Interface
Teaming type : loadbalance
Interface         : team0, eth0, eth2

teamd package should be installed on server.

We can see the previous network configuration setup by using nmcli command.

root@nsk# nmcli con show
NAME  UUID                                                            TYPE      DEVICE
eth0     5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03     ethernet    eth0
eth1     8761e75a-0ba3-4f73-80e4-66df47023ed1  ethernet    eth1

Teaming configuration

Execute the below command on server to create team0 teaming interface.

root@nsk# nmcli con add type team con-name team0 ifname team0 config '{ "runner": {"name": "loadbalance"}}'
Connection 'team0' (e07596d2-59b6-406a-a742-07d5ff96901b) successfully added.

root@nsk# nmcli con show
NAME   UUID                                                              TYPE      DEVICE
eth0     5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03        ethernet    eth0
eth1     8761e75a-0ba3-4f73-80e4-66df47023ed1     ethernet    eth1
team0  e07596d2-59b6-406a-a742-07d5ff96901b     team         team0

Now, /etc/sysconfig/network-scripts/team0 contains below lines

root@nsk# cat ifcfg-team0
TEAM_CONFIG="{\"runner\": {\"name\": \"loadbalance\", \"tx_hash\": [\"eth\", \"ipv4\", \"ipv6\"]}}"
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=team0
UUID=e07596d2-59b6-406a-a742-07d5ff96901b
DEVICE=team0
ONBOOT=yes
DEVICETYPE=Team

Now add IPv4 address on team0 interface.

root@nsk# nmcli con mod team0 ipv4.addresses 172.27.0.169/28
root@nsk# nmcli con mod team0 ipv4.gateway 172.27.0.161
root@nsk# nmcli con mod team0 ipv4.method manual

Add eth0 interface to the teaming interface.

root@nsk# nmcli con add type team-slave con-name team0-slave0 ifname eth0 master team0
Connection 'team0-slave0' (5fa33d1b-3f25-4fda-b10d-a184e4260cbe) successfully added.

Add eth2 interface to the teaming interface.

root@nsk# nmcli con add type team-slave con-name team0-slave1 ifname eth2 master team0
Connection 'team0-slave1' (bbdb784b-2a9c-4fe3-9a19-11d64a9748ff) successfully added.

root@nsk# ls -al | grep -i team0

ifcfg-team0  
ifcfg-team0-slave0  
ifcfg-team0-slave1

Now bring up the team0 interface.

root@nsk# nmcli connection up team0
Connection successfully activated (master waiting for slaves) (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/11)

root@nsk# teamdctl team0 state
setup:
  runner: loadbalance

root@nsk# /etc/init.d/network restart
Restarting network (via systemctl):                        [  OK  ]

root@nsk# ifconfig | egrep -i "eth0|eth2|team0"

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether b0:83:fe:d8:44:02  txqueuelen 1000  (Ethernet)
        RX packets 2930895882  bytes 2216873481505 (2.0 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1694125755  bytes 312320131645 (290.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 56

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether b0:83:fe:d8:44:02  txqueuelen 1000  (Ethernet)
        RX packets 2705861  bytes 301266106 (287.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25  bytes 4159 (4.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 59

team0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.27.0.169  netmask 255.255.255.240  broadcast 172.27.0.175
        inet6 fe80::bbe8:90aa:4108:e728  prefixlen 64  scopeid 0x20<link>
        ether b0:83:fe:d8:44:02  txqueuelen 1000  (Ethernet)
        RX packets 35  bytes 2586 (2.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 28  bytes 4451 (4.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Check the configuration

root@nsk# nmcli con reload
root@nsk# nmcli con show
NAME                 UUID                                                             TYPE      DEVICE
eth1                    8761e75a-0ba3-4f73-80e4-66df47023ed1   ethernet    eth1
team0                 e07596d2-59b6-406a-a742-07d5ff96901b   team         team0
team0-slave0     5fa33d1b-3f25-4fda-b10d-a184e4260cbe     ethernet    eth0
team0-slave1     bbdb784b-2a9c-4fe3-9a19-11d64a9748ff     ethernet    eth2

Check the team0 ports status

root@nsk# teamnl team0 ports
 4: eth2: up 1000Mbit FD
 2: eth0: up 1000Mbit FD

We can use teamdctl command to check team0 configuration status.

root@nsk# teamdctl team0 state
setup:
  runner: loadbalance
ports:
  eth0
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
  eth2
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
root@nsk#

Note : If we need Activebackup setup, just replace the loadbalance with activebackup while creating team0 interface.

If storage team extended the existing LUN instead of creating new LUN, below steps need to follow.

Run multipath -ll command and search the device info which is mapped to the LUN.

root@nsk# multipath -ll | grep -A 6 -i 3600a09634224747a367d4b55357c4f87
3600a09634224747a367d4b55357c4f87 dm-6 NETAPP,LUN C-Mode
size=400G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 2:0:0:1 sdg 8:96  active ready running
  `- 1:0:0:1 sdc 8:32  active ready running
3600a09803830436a345d4b51506c4f43 dm-2 NETAPP,LUN C-Mode
size=110G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw

As per above output, sdg and sdc is the devices.  Now rescan the LUN.

root@nsk# echo "1" > /sys/block/sdg/device/rescan
root@nsk# echo "1" > /sys/block/sdc/device/rescan 

Reload multipathd service 

root@nsk# /etc/init.d/multipathd reload

Resize the PV (provide complete path like below)

root@nsk# pvresize /dev/mapper/3600a09634224747a367d4b55357c4f87  

Extend the LV

root@nsk# lvextend -L +199.95g /dev/mapper/oracle_vg-oracledata

Sosreport is a tool that collects configuration details, system information, diagnostic information and system logs from Linux server. Sosreport  generates  an archive report from the running system. The archive may be stored locally or centrally for recording or tracking purposes.

Normally sosreport will be saved under /tmp (RHEL6) /var/tmp (RHEL7) directory on linux server. Some time we need to save the sosreport to some other temporary directory because of space issue. 

Below steps are used to generate sosreport in different directory.

RHEL 5.6 and earlier

We need to set the temporary path environment before running sosreport.

[root@nsk ~]# env TMP=/path/to/target/directory sosreport

RHEL 5.7 and above

We have to use option --tmp-dir with sosreport to redirect the sosreport path 

[root@nsk ~]# sosreport --tmp-dir /path/to/target/directory

Sosreport may have hung because of a specific plugin. We can list all available sosreport plugin modules by below command

[root@nsk ~]# sosreport -l

Identify which plugin is hanging on by below command

[root@nsk ~]# sosreport -vvvv 

If a plugin times out or last plugin does not finish, rerun the sosreport and skip that plugin

[root@nsk ~]# sosreport -v -n plugin_name --tmp-dir /path/to/target/directory

ausearch is a tool to search audit daemon logs based upon the events based on different search criteria.

 Audit system stores log entries in the /var/log/audit/audit.log file; if log rotation is enabled, rotated audit.log files are stored in the same directory.

Each event consists of three records, which share the same time stamp. Each record consists of several name=value pairs separated by a white space or a comma. 

type=SYSCALL msg=audit(1510471123.129:36): arch=c000003e syscall=175 success=yes exit=0 a0=1901a20 a1=1c5d a2=41a2d8 a3=18fe400 items=0 ppid=7 24 pid=725 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmo d" subj=system_u:system_r:insmod_t:s0 key=(null)
type=PROCTITLE msg=audit(1510471123.129:36): proctitle=2F7362696E2F6D6F6470726F6265002D71002D2D0069707461626C655F6D616E676C65
type=NETFILTER_CFG msg=audit(1510471123.276:37): table=nat family=2 entries=0

Here, ausearch utility allows us to search Audit log files for specific events. Also it is used to read the audit log timestamp to user readable timestamp by using -i (--interpret) option.

Example:

1.Search for hostname with given hostname specific.

[root@nsk ~]# ausearch -i -hn nsk.testingserver.internal
----
type=USER_AUTH msg=audit(10/04/2018 12:32:02.610:75) : pid=1617 uid=root auid=unset ses=unset subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_permit acct=gdm exe=/usr/libexec/gdm-session-worker hostname=nsk.testingserver.internal addr=? terminal=/dev/tty1 res=success'
----

2. Search only the first event that matches the search pattern.

[root@nsk ~]# ausearch -i -hn nsk.testingserver.internal --just-one
----
type=USER_AUTH msg=audit(10/04/2018 12:32:02.610:75) : pid=1617 uid=root auid=unset ses=unset subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_permit acct=gdm exe=/usr/libexec/gdm-session-worker hostname=nsk.testingserver.internal addr=? terminal=/dev/tty1 res=success'
[root@nsk ~]#

3. Search for all logged actions performed by user nsk, using the user's login ID (auid). 

[root@nsk log]# ausearch -ua 1001 -i
----
type=LOGIN msg=audit(12/04/2017 11:39:18.052:156) : pid=1433 uid=root subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=unset auid=nsk tty=(none) old-ses=4294967295 ses=2 res=yes
----
type=USER_ROLE_CHANGE msg=audit(12/04/2017 11:39:18.991:157) : pid=1433 uid=root auid=nsk ses=2 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 selected-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 exe=/usr/sbin/sshd hostname=nsk addr=fe80::f24d:eb49:d219:6b0 terminal=ssh res=success'

4. Search for an event matching the given Process ID.

[root@nsk ~]# ausearch -i -p 2162
----
type=CRYPTO_SESSION msg=audit(10/05/2018 13:07:54.052:118) : pid=2162 uid=root auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-server cipher=aes256-ctr ksize=256 mac=hmac-sha2-256 pfs=curve25519-sha256@libssh.org spid=2163 suid=sshd rport=50935 laddr=10.0.2.15 lport=22  exe=/usr/sbin/sshd hostname=? addr=10.0.2.2 terminal=? res=success'
----

5. Search the /var/log/audit/audit.log file for successful login attempts

[root@nsk log]# ausearch --message USER_LOGIN --success yes --interpret
----
type=USER_LOGIN msg=audit(11/12/2017 13:04:08.279:110) : pid=662 uid=root auid=root ses=2 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023
msg='op=login id=root exe=/usr/bin/login hostname=nsk addr=? terminal=tty1 res=success'
----
type=USER_LOGIN msg=audit(11/12/2017 13:05:46.297:107) : pid=1242 uid=root auid=root ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c
0.c1023 msg='op=login id=root exe=/usr/sbin/sshd hostname=10.0.2.2 addr=10.0.2.2 terminal=/dev/pts/0 res=success'
----

6. Search for all account and group activities.

[root@nsk log]# ausearch -m ADD_USER -m DEL_USER -m ADD_GROUP -m DEL_GROUP -i
----
type=ADD_GROUP msg=audit(11/15/2017 22:05:29.396:169) : pid=15764 uid=root auid=root ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-group id=dovecot exe=/usr/sbin/groupadd hostname=? addr=? terminal=? res=success'
----
type=ADD_USER msg=audit(11/15/2017 22:05:29.431:171) : pid=15769 uid=root auid=root ses=2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 msg='op=add-user id=dovecot exe=/usr/sbin/useradd hostname=? addr=? terminal=? res=success'

7. Search for all failed PROCTITLE from yesterday up until now

[root@nsk log]# ausearch --start yesterday --end now -m PROCTITLE -sv no -i
----
type=PROCTITLE msg=audit(10/03/2018 18:00:02.623:71) : proctitle=/usr/sbin/httpd -DFOREGROUND
type=SYSCALL msg=audit(10/03/2018 18:00:02.623:71) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x560fb5acc008 a1=O_RDONLY a2=0x1b6 a3=0x24 items=0 ppid=1 pid=1145 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=httpd exe=/usr/sbin/httpd subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(10/03/2018 18:00:02.623:71) : avc:  denied  { read } for  pid=1145 comm=httpd name=php-mapi.cfg dev="dm-0" ino=19498435 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:zarafa_etc_t:s0 tclass=file

Please refer man ausearch for more option.

By default, audit log would be like below.

type=SYSCALL msg=audit(1510471123.129:36): arch=c000003e syscall=175 success=yes exit=0 a0=1901a20 a1=1c5d a2=41a2d8 a3=18fe400 items=0 ppid=7 24 pid=725 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmo d" subj=system_u:system_r:insmod_t:s0 key=(null)
type=PROCTITLE msg=audit(1510471123.129:36): proctitle=2F7362696E2F6D6F6470726F6265002D71002D2D0069707461626C655F6D616E676C65 type=NETFILTER_CFG msg=audit(1510471123.276:37): table=nat family=2 entries=0

Below commands are used to convert the auditlog timestamp to user readable timestamp.

1. AUSEARCH : 

ausearch utility allows us to search Audit log files for specific events. Also it is used to read the audit log epoch timestamp to user readable timestamp.By default, ausearch searches the /var/log/audit/audit.log file. We can specify a different file using the ausearch options -if file_name command

[root@nsk log]# ausearch -i | grep -i CONFIG
type=CONFIG_CHANGE msg=audit(11/12/2017 12:48:40.357:5) : audit_backlog_limit=8192 old=64 auid=unset ses=unset subj=system_u:system_r:unconfined_service_t:s0 res=yes
type=CONFIG_CHANGE msg=audit(11/12/2017 12:48:40.357:6) : audit_failure=1 old=1 auid=unset ses=unset subj=system_u:system_r:unconfined_service_t:s0 res=yes

-i, --interpret : Interpret numeric entities into text

2. DATE :  

A) Date command with %s is used to convert normal time to epoch time.
%s : seconds since 1970-01-01 00:00:00 UTC

Display the current time in the given FORMAT, or set the system date
[root@nsk audit]# date +%s
1538570770
[root@nsk audit]# date +%s
1538570773

B) Date command with -d@ is used to convert epoch time to normal time.
[root@nsk audit]# date -d@1538570776
Wed Oct  3 18:16:16 IST 2018
[root@nsk audit]#

C) If we want to convert the specific system time to epoch time.
[root@nsk audit]# date --date="Wed Oct  3 18:16:16 IST 2018" +%s
1538570776

3. PERL : Perl was originally a language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information

[root@nsk audit]# perl -pe 's/(\d+)/localtime/e' audit.log  | more
type=DAEMON_START msg=audit(Sun Nov Wed Oct  3 20:22:26 2018 12:48:40 2017.186:6974): op=start ver=2.7.6 format=raw kernel=3.10.0-693.el7.x86_64 auid=4294967295 pid=606 uid=0 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=success
type=CONFIG_CHANGE msg=audit(Sun Nov Wed Oct  3 20:22:26 2018 12:48:40 2017.357:5): audit_backlog_limit=8192 old=64 auid=4294967295 ses=429496 7295 subj=system_u:system_r:unconfined_service_t:s0 res=1

Here,
-p : Print out the pattern space
-e : command (This command allows one to pipe input from a shell command into pattern space)
localtime         : it is perl function, it is having 9 elements.
-d : Match 1 or more repetition of digits

Rebuilding the GRUB configuration file in RHEL7

Situation : Whenever there is inconsistency in the GRUB file, empty grub configuration or of the grub file got removed accidentally, follow the below steps to fix the grub issue.







Solution:

STEP 1 : List the partitions by using “ ls “ command
grub > ls
(proc) (hd0) (hd0,msdos3) (hd0,msdos2) (hd0,msdos1) (fd0)

STEP 2: Findout the root partitions by using command ls < partition name >. Run ls command to all above listed available  partitions.