LINUX AND VMWARE STUFF

This Blog is to share our knowledge and expertise on Linux System Administration and VMware Administration

Showing posts with label RHEL Cluster. Show all posts
Showing posts with label RHEL Cluster. Show all posts

Thursday, November 8, 2018

Extend the cluster file system by extending the existing netapp storage lun in RHEL server

Thursday, November 08, 2018 0
If storage team extended the existing LUN instead of creating new LUN, below steps need to follow.

 Run multipath -ll command and search the device info which is mapped to the LUN.

 root@nsk# multipath -ll | grep -A 6 -i 3600a09634224747a367d4b55357c4f87
3600a09634224747a367d4b55357c4f87 dm-6 NETAPP,LUN C-Mode
size=400G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 2:0:0:1 sdg 8:96  active ready running
  `- 1:0:0:1 sdc 8:32  active ready running
3600a09803830436a345d4b51506c4f43 dm-2 NETAPP,LUN C-Mode
size=110G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw

 As per above output, sdg and sdc is the devices.  Now rescan the LUN.

 root@nsk# echo "1" > /sys/block/sdg/device/rescan
root@nsk# echo "1" > /sys/block/sdc/device/rescan 

 Reload multipathd service 

 root@nsk# /etc/init.d/multipathd reload

 Resize the PV (provide complete path like below)

 root@nsk# pvresize /dev/mapper/3600a09634224747a367d4b55357c4f87  

 Extend the LV

 root@nsk# lvextend -L +199.95g /dev/mapper/oracle_vg-oracledata
Tags # RHEL Cluster Continue Reading
By v�o l�c No comments:
Labels:

Saturday, November 4, 2017

How to Manually Deactivate a High Available cluster volume group?

Saturday, November 04, 2017
Follow the below steps 

 unmount all filesystem assocated with the VG:

#umount <mount_point>

deactivate the cluster VG:

#vgchange –a n HAVG_<vgname>

Remove all hostname tags from the VG:

#vgchange –-deltag <server_name>.testdoamin.com HAVG_<vgname>
Tags # RHEL Cluster Continue Reading
By v�o l�c
Labels:

Thursday, November 2, 2017

How to create New Volume Group in RHEL Cluster?

Thursday, November 02, 2017 0

Follow the below steps to create Volume Group in RHEL Cluster

1. Add the disk to the server.

2. Pvcreate the new partitions

3. Create the VG’s


vgcreate -c n -s 32M HAVG_<vg_name> /dev/mapper/mpath#p#

-s : Sets the physical extent size on physical volumes of this volume group
-c : If clustered locking is enabled, this defaults to y indicating that this Volume Group is shared with other nodes in the cluster

Creating a new Logical volume to a HA Volume group

To create a LV to a newly created volume group you have to add the hostname tag to the VG. This will permit one server in the cluster to have access to the volume group at a given time.

1. vgchange --addtag <hostname>.testdomain.com HAVG_<vg_name>
2. lvcreate –L ##M –n lv-<lvname> HAVG_<vg_name>
3. mke2fs –j /dev/HAVG_<vg_name>/lv-<lvname>

Create mountpoints on all nodes in the cluster. Do not add to /etc/fstab

1. mkdir <mount_point>
2. mount /dev/HAVG_<vg_name>/lv-<lvname> <mount_point>
3. Change permissions on mountpoint.
Tags # RHEL Cluster Continue Reading
By v�o l�c No comments:
Labels:

Thursday, October 15, 2015

What are the performance enhancements in GFS2 as compared to GFS?

Thursday, October 15, 2015 0
 GFS2 features  
  • Better performance for heavy usage in a single directory.
  • Faster synchronous I/O operations.
  • Faster cached reads (no locking overhead).
  • Faster direct I/O with preallocated files (provided I/O size is reasonably large, such   as     4M    blocks).
  • Faster I/O operations in general.
  • Faster Execution of the df command, because of faster statfs calls.
  • Improved atime mode to reduce the number of write I/O operations generated by atime.
  When compared with GFS GFS2 supports the following features.
  • Extended file attributes (xattr) the lsattr() and chattr() attribute settings via standard ioctl() calls nanosecond timestamps
  • GFS2 uses less kernel memory.
  • GFS2 requires no metadata generation numbers.
  • Allocating GFS2 metadata does not require reads. Copies of metadata blocks in multiple journals are managed by revoking blocks from the journal before lock release.
  • GFS2 includes a much simpler log manager that knows nothing about unlinked inodes or quota changes.
  • The gfs2_grow and gfs2_jadd commands use locking to prevent multiple instances running at the same time.
  • The ACL code has been simplified for calls like creat() and mkdir().
  • Unlinked inodes, quota changes, and statfs changes are recovered without remounting the journal
Tags # RHEL Cluster Continue Reading
By v�o l�c No comments:
Labels:

What is a Quorum Disk in cluster?

Thursday, October 15, 2015 0
Quorum Disk is a disk-based quorum daemon, qdiskd, that provides supplemental heuristics to determine node fitness.
With heuristics you can determine factors that are important to the operation of the node in the event of a network partition.

    For a 3 node cluster a quorum state is present untill 2 of the 3 nodes are active i.e. more than half. But what if due to some reasons the 2nd node also stops communicating with the the 3rd node? In that case under a normal architecture  the cluster would dissolve and stop working. 
But for mission critical environments and such scenarios we use quorum disk in which an additional disk is configured which is mounted on all the nodes with qdiskd service running and a vote value is assigned to it.

    So suppose in above case I have assigned 1 vote to qdisk so even after 2 nodes stops communicating with 3rd node, the cluster would have 2 votes (1 qdisk + 1 from 3rd node) which is still more than half of vote count for a 3 node cluster. Now both the inactive nodes would be fenced and your 3rd node would be still up and running being a part of the cluster.
Tags # RHEL Cluster Continue Reading
By v�o l�c No comments:
Labels:

What is split-brain condition in Red Hat Cluster?

Thursday, October 15, 2015
 We say a cluster has quorum if a majority of nodes are alive, communicating, and agree on the active cluster members.
    For example, in a thirteen-node cluster, quorum is only reached if seven or more nodes are communicating. If the seventh node dies, the cluster loses quorum and can no longer function.
  • A cluster must maintain quorum to prevent split-brain issues.
  • If quorum was not enforced, quorum, a communication error on that same thirteen-node cluster may cause a situation where six nodes are operating on the shared storage, while another six nodes are also operating on it, independently.
  • Because of the communication error, the two partial-clusters would overwrite areas of the disk and corrupt the file system.
  • With quorum rules enforced, only one of the partial clusters can use the shared storage, thus protecting data integrity.
  • Quorum doesn't prevent split-brain situations, but it does decide who is dominant and allowed to function in the cluster.
  • quorum can be determined by a combination of communicating messages via Ethernet and through a quorum disk.
Tags # RHEL Cluster Continue Reading
By v�o l�c
Labels:

What are Tie-breakers in Red Hat Cluster?

Thursday, October 15, 2015 0
Tie-breakers are additional heuristics that allow a cluster partition to decide whether or not it is quorate in the event of an even-split - prior to fencing.
  • With such a tie-breaker, nodes not only monitor each other, but also an upstream router that is on the same path as cluster communications.
  • If the two nodes lose contact with each other, the one that wins is the one that can still ping the upstream router.
  • That is why, even when using tie-breakers, it is important to ensure that fencing is configured correctly.CMAN has no internal tie-breakers for various reasons. However, tie-breakers can be implemented using the API
Tags # RHEL Cluster Continue Reading
By v�o l�c No comments:
Labels:

Thursday, October 1, 2015

How to Configure Linux Cluster with 2 Nodes on RedHat and CentOS

Thursday, October 01, 2015 0
In an active-standby Linux cluster configuration, all the critical services including IP, filesystem will failover from one node to another node in the cluster.

It explains how to create and configure two node redhat cluster using command line utilities.
The following are the high-level steps involved in configuring Linux cluster on Redhat or CentOS:
• Install and start RICCI cluster service
• Create cluster on active node
• Add a node to cluster
• Add fencing to cluster
• Configure fail over domain
• Add resources to cluster
• Sync cluster configuration across nodes
• Start the cluster
• Verify fail over by shutting down an active node

1. Required Cluster Packages

First make sure the following cluster packages are installed. If you don’t have these packages install them using yum command.
[root@rh1 ~]# rpm -qa | egrep -i "ricci|luci|cluster|ccs|cman"
modcluster-0.16.2-28.el6.x86_64
luci-0.26.0-48.el6.x86_64
ccs-0.16.2-69.el6.x86_64
ricci-0.16.2-69.el6.x86_64
cman-3.0.12.1-59.el6.x86_64
clusterlib-3.0.12.1-59.el6.x86_64

2. Start RICCI service and Assign Password

Next, start ricci service on both the nodes.

[root@rh1 ~]# service ricci start
Starting oddjobd: [ OK ]
generating SSL certificates... done
Generating NSS database... done
Starting ricci: [ OK ]

You also need to assign a password for the RICCI on both the nodes.

[root@rh1 ~]# passwd ricci
Changing password for user ricci.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Also, If you are running iptables firewall, keep in mind that you need to have appropriate firewall rules on both the nodes to be able to talk to each other.

3. Create Cluster on Active Node

From the active node, please run the below command to create a new cluster.
The following command will create the cluster configuration file /etc/cluster/cluster.conf. If the file already exists, it will replace the existing
cluster.conf with the newly created cluster.conf.

[root@rh1 ~]# ccs -h rh1.mydomain.net --createcluster mycluster
rh1.mydomain.net password:

[root@rh1 ~]# ls -l /etc/cluster/cluster.conf
-rw-r-----. 1 root root 188 Sep 26 17:40 /etc/cluster/cluster.conf
Also keep in mind that we are running these commands only from one node on the cluster and we are not yet ready to propagate the changes
to the other node on the cluster.

4. Initial Plain cluster.conf File

After creating the cluster, the cluster.conf file will look like the following:
[root@rh1 ~]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="1" name="mycluster">
<fence_daemon/>
<clusternodes/>
<cman/>
<fencedevices/>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>


5. Add a Node to the Cluster

Once the cluster is created, we need to add the participating nodes to the cluster using the ccs command as shown below.
First, add the first node rh1 to the cluster as shown below.

[root@rh1 ~]# ccs -h rh1.mydomain.net --addnode rh1.mydomain.net
Node rh1.mydomain.net added.

Next, add the second node rh2 to the cluster as shown below.
[root@rh1 ~]# ccs -h rh1.mydomain.net --addnode rh2.mydomain.net
Node rh2.mydomain.net added.
Once the nodes are created, you can use the following command to view all the available nodes in the cluster. This will also display the node
id for the corresponding node.

[root@rh1 ~]# ccs -h rh1 --lsnodes
rh1.mydomain.net: nodeid=1
rh2.mydomain.net: nodeid=2

6. cluster.conf File After Adding Nodes

This above will also add the nodes to the cluster.conf file as shown below.

[root@rh1 ~]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="3" name="mycluster">
<fence_daemon/>
<clusternodes>
<clusternode name="rh1.mydomain.net" nodeid="1"/>
<clusternode name="rh2.mydomain.net" nodeid="2"/>
</clusternodes>
<cman/>
<fencedevices/>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>

7. Add Fencing to Cluster

Fencing is the disconnection of a node from shared storage. Fencing cuts off I/O from shared storage, thus ensuring data integrity.
A fence device is a hardware device that can be used to cut a node off from shared storage.
This can be accomplished in a variety of ways: powering off the node via a remote power switch, disabling a Fiber Channel switch port, or revoking a host’s SCSI 3 reservations.
A fence agent is a software program that connects to a fence device in order to ask the fence device to cut off access to a node’s shared storage (via powering off the node or removing access to the shared storage by other means).
Execute the following command to enable fencing.

[root@rh1 ~]# ccs -h rh1 --setfencedaemon post_fail_delay=0
[root@rh1 ~]# ccs -h rh1 --setfencedaemon post_join_delay=25
Next, add a fence device. There are different types of fencing devices available. If you are using virtual machine to build a cluster, use
fence_virt device as shown below.

[root@rh1 ~]# ccs -h rh1 --addfencedev myfence agent=fence_virt
Next, add fencing method. After creating the fencing device, you need to created the fencing method and add the hosts to the fencing method.
[root@rh1 ~]# ccs -h rh1 --addmethod mthd1 rh1.mydomain.net
Method mthd1 added to rh1.mydomain.net.
[root@rh1 ~]# ccs -h rh1 --addmethod mthd1 rh2.mydomain.net
Method mthd1 added to rh2.mydomain.net.

Finally, associate fence device to the method created above as shown below:
[root@rh1 ~]# ccs -h rh1 --addfenceinst myfence rh1.mydomain.net mthd1
[root@rh1 ~]# ccs -h rh1 --addfenceinst myfence rh2.mydomain.net mthd1

8. cluster.conf File after Fencing

Your cluster.conf will look like below after the fencing devices, methods are added.
[root@rh1 ~]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="10" name="mycluster">
<fence_daemon post_join_delay="25"/>
<clusternodes>
<clusternode name="rh1.mydomain.net" nodeid="1">
<fence>
<method name="mthd1">
<device name="myfence"/>
</method>
</fence>
</clusternode>
<clusternode name="rh2.mydomain.net" nodeid="2">
<fence>
<method name="mthd1">
<device name="myfence"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fencedevices>
<fencedevice agent="fence_virt" name="myfence"/>

</fencedevices>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>

9. Types of Failover Domain

A failover domain is an ordered subset of cluster members to which a resource group or service may be bound.
The following are the different types of failover domains:
• Restricted failover-domain: Resource groups or service bound to the domain may only run on cluster members which are also members
of the failover domain. If no members of failover domain are availables, the resource group or service is placed in stopped state.
• Unrestricted failover-domain: Resource groups bound to this domain may run on all cluster members, but will run on a member of the
domain whenever one is available. This means that if a resource group is running outside of the domain and member of the domain
transitions online, the resource group or
• service will migrate to that cluster member.
• Ordered domain: Nodes in the ordered domain are assigned a priority level from 1-100. Priority 1 being highest and 100 being the
lowest. A node with the highest priority will run the resource group. The resource if it was running on node 2, will migrate to node 1
when it becomes online.
• Unordered domain: Members of the domain have no order of preference. Any member may run in the resource group. Resource group
will always migrate to members of their failover domain whenever possible.

10. Add a Filover Domain

To add a failover domain, execute the following command. In this example, I created domain named as “webserverdomain”,
[root@rh1 ~]# ccs -h rh1 --addfailoverdomain webserverdomain ordered
Once the failover domain is created, add both the nodes to the failover domain as shown below:
[root@rh1 ~]# ccs -h rh1 --addfailoverdomainnode webserverdomain rh1.mydomain.net priority=1
[root@rh1 ~]# ccs -h rh1 --addfailoverdomainnode webserverdomain rh2.mydomain.net priority=2
You can view all the nodes in the failover domain using the following command.
[root@rh1 ~]# ccs -h rh1 --lsfailoverdomain
webserverdomain: restricted=0, ordered=1, nofailback=0
rh1.mydomain.net: 1
rh2.mydomain.net: 2

11. Add Resources to Cluster

Now it is time to add a resources. This indicates the services that also should failover along with ip and filesystem when a node fails. For
example, the Apache webserver can be part of the failover in the Redhat Linux Cluster.
When you are ready to add resources, there are 2 ways you can do this.
You can add as global resources or add a resource directly to resource group or service.
The advantage of adding it as global resource is that if you want to add the resource to more than one service group you can just reference the global resource on your service or resource group. In this example, we added the filesystem on a shared storage as global resource and referenced it on the service.
[root@rh1 ~]# ccs –h rh1 --addresource fs name=web_fs device=/dev/cluster_vg/vol01 mountpoint=/var/www fstype=ext4
To add a service to the cluster, create a service and add the resource to the service.
[root@rh1 ~]# ccs -h rh1 --addservice webservice1 domain=webserverdomain recovery=relocate autostart=1
Now add the following lines in the cluster.conf for adding the resource references to the service. In this example, we also added failover IP to
our service.
<fs ref="web_fs"/>
<ip address="192.168.1.12" monitor_link="yes" sleeptime="10"/>
Tags # RHEL Cluster Continue Reading
By v�o l�c No comments:
Labels: