Simple Gluster Lab

Introduction⌗

In this lab we will build out a small, 2 node Gluster cluster using 2 bricks to back a single volume. This will then be mounted to a client and tested, before a third Gluster node is added bringing the number of bricks in the cluster to 3.

This guide assumes that Libvirt is setup and configured on the host machine.

Initial Cluster

Extended Cluster

Initial Cluster build⌗

Before starting:

Ensure public ssh key is available in ~/.ssh/
Ensure the CentOS 7 image is available here /var/lib/libvirt/images/iso/CentOS-7-x86_64-GenericCloud.qcow2

Setup VMs⌗

To build out the initial lab, the scripts are available on GitHub.

Checkout the setup and tear down scripts from GitHub

bash


git clone https://github.com/greeninja/kvm-gluster-lab.git

Move to the newly checked out folder

bash


cd kvm-gluster-lab

Modify the variable in the script to setup Root Password, Public SSH Key and Base OS location

bash


vim setup.sh

Once the variables have been set, run the script. This will set up all 4 of the VMs to save time.

bash


bash setup.sh

Example output

bash


**********                                                                                                                                                                     
[   5.8] Setting up initial partition table on /var/lib/virt/nvme/client01.gluster.lab.qcow2                                                                                   
[   6.6] Copying /dev/sda1                                                                                                                                                     
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00 
[  22.8] Expanding /dev/sda1 using the ‘xfs_growfs’ method                             
                                                                                                                                                                               
Resize operation completed with no errors.  Before deleting the old disk,                                                                                                      
carefully check that the resized disk boots and works correctly.                                                                                                               
Customising OS for client01                                                                                                                                                    
[   0.0] Examining the guest ...                                                       
[   8.4] Setting a random seed                                                         
[   8.4] Uninstalling packages: cloud-init                                             
[  10.6] Setting the hostname: client01.gluster.lab                       
[  10.6] SSH key inject: root                                                          
[  12.6] Setting passwords                                                             
[  14.4] SELinux relabelling
[  30.7] Finishing off
Defining client01

Starting install...
Domain creation completed.
 Id    Name                    State
----------------------------------------
 123   gluster01.gluster.lab   running
 124   gluster02.gluster.lab   running
 125   gluster03.gluster.lab   running
 126   client01.gluster.lab    running

Now the VMs are up and running, we will shutdown gluster03 before continuing

bash


virsh shutdown gluster03.gluster.lab

Setup Gluster⌗

There will be 2 versions of this setup. The first is manually setting up and configuring Gluster and the second will configure the cluster with Ansible.

With either setup, we will need the VMs IP addresses from DHCP

bash


virsh net-dhcp-leases gluster-lab

Example output

bash


 Expiry Time           MAC address         Protocol   IP address       Hostname    Client ID or DUID
------------------------------------------------------------------------------------------------------
 2021-01-19 09:48:35   52:54:00:00:5e:1e   ipv4       10.44.50.96/24   gluster02   -
 2021-01-19 09:47:41   52:54:00:14:2a:19   ipv4       10.44.50.22/24   gluster01   -
 2021-01-19 09:49:31   52:54:00:93:15:e5   ipv4       10.44.50.86/24   gluster03   -
 2021-01-19 09:50:24   52:54:00:f1:36:77   ipv4       10.44.50.63/24   client01    -

Manual Setup⌗

The following steps need to be taken on each of the Gluster nodes.

(Optional) Install a text editor of choice. vi is available on the nodes

bash


yum install -y vim

Install the Gluster yum repository. Here we will be installing Gluster 6 so that this lab can be used for a future lab.

bash


yum install centos-release-gluster6 -y

Now install glusterfs server

bash


yum install glusterfs-server -y

Enable the Gluster service

bash


systemctl enable glusterd

Start the Gluster service

bash


systemctl start glusterd

Add the first disk to a Physical Volume

bash


pvcreate /dev/vdb

Add the second disk to a Physical Volume

bash


pvcreate /dev/vdc

Confirm both drives are visible as Physical Volumes

bash

pvs

Example output

bash


  PV         VG Fmt  Attr PSize PFree
  /dev/vdb      lvm2 ---  5.00g 5.00g
  /dev/vdc      lvm2 ---  5.00g 5.00g

Create the Volume group

bash


vgcreate brick /dev/vdb /dev/vdc

Check the Volume Group

bash

vgs

Example output

bash


  VG    #PV #LV #SN Attr   VSize VFree
  brick   2   0   0 wz--n- 9.99g 9.99g

Create the Logical Volume for the Gluster brick

bash


lvcreate -l 100%FREE -T brick -n brick01

Check Logical Volume has created

bash

lvs

Example output

bash


  LV      VG    Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  brick01 brick twi-a-tz-- <9.97g             0.00   10.61

Create a filesystem on the Logical Volume

bash


mkfs.xfs -i size=512 /dev/brick/brick01

Create a mount directory for the brick

bash


mkdir -p /bricks/brick01

Add the brick mount to /etc/fstab

bash


/dev/brick/brick01        /bricks/brick01 xfs     rw,noatime,inode64,nouuid   1   2

Mount the volume

bash


mount -a

Check the volume has mounted

bash


df -h

Example output

bash


Filesystem                 Size  Used Avail Use% Mounted on
devtmpfs                   3.8G     0  3.8G   0% /dev
tmpfs                      3.9G     0  3.9G   0% /dev/shm
tmpfs                      3.9G   17M  3.9G   1% /run
tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1                   40G  1.1G   39G   3% /
tmpfs                      783M     0  783M   0% /run/user/0
/dev/mapper/brick-brick01   10G   33M   10G   1% /bricks/brick01

Create a brick directory on the mount point

bash


mkdir -p /bricks/brick01/brick

Now repeat on the second node

Now we will move on to configuring Gluster

These steps need to be done from one Gluster node. Here we’ll be using gluster01

Confirm that the node can resolve the other node’s IP address

bash


ping -c 3 gluster02

Example output

bash


PING gluster02.gluster.lab (10.44.50.96) 56(84) bytes of data.
64 bytes from gluster02.gluster.lab (10.44.50.96): icmp_seq=1 ttl=64 time=0.495 ms
64 bytes from gluster02.gluster.lab (10.44.50.96): icmp_seq=2 ttl=64 time=0.572 ms
64 bytes from gluster02.gluster.lab (10.44.50.96): icmp_seq=3 ttl=64 time=0.578 ms

--- gluster02.gluster.lab ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.495/0.548/0.578/0.042 ms

Probe the other node to add it to the cluster.

bash


gluster peer probe gluster02

Confirm that the node has been added to the cluster

bash


gluster peer status

Example output

bash


Number of Peers: 1

Hostname: gluster02
Uuid: fe542742-d0f4-4e1a-aebd-a745231045f0
State: Peer in Cluster (Connected)

Now create a Gluster volume. Due to it being created over 2 bricks, there is a high risk of split-brain and so the create command will need a force flag

bash


gluster volume create volume01 replica 2 \
  gluster01:/bricks/brick01/brick \
  gluster02:/bricks/brick01/brick \
  force

Check the volume has been created

bash


gluster volume list

Example output
bash
```
volume01
```
Once confirmed, the volume can now be started

bash


gluster volume start volume01

Check the status of the volume

bash


gluster volume status volume01

Example output

bash


Status of volume: volume01
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gluster01:/bricks/brick01/brick       49152     0          Y       17527
Brick gluster02:/bricks/brick01/brick       49152     0          Y       16825
Self-heal Daemon on localhost               N/A       N/A        Y       17548
Self-heal Daemon on gluster02               N/A       N/A        Y       16846
 
Task Status of Volume volume01
------------------------------------------------------------------------------
There are no active volume tasks

The volume is now set up and ready to be used by a client. Mounting the volume and setting up a test is detailed here

Ansible Setup⌗

Within the GitHub repository there is an ansible directory. Here we need to edit the inventory.yaml files to ensure the IP addresses are correct.

Get the current IP addresses from libvirt DHCP

bash


virsh net-dhcp-leases gluster-lab

Example output

bash


 Expiry Time           MAC address         Protocol   IP address        Hostname    Client ID or DUID
-------------------------------------------------------------------------------------------------------
 2021-01-19 12:21:58   52:54:00:37:8d:b7   ipv4       10.44.50.77/24    gluster03   -
 2021-01-19 12:20:10   52:54:00:57:19:d6   ipv4       10.44.50.100/24   gluster01   -
 2021-01-19 12:22:51   52:54:00:92:b1:b7   ipv4       10.44.50.10/24    client01    -
 2021-01-19 12:21:04   52:54:00:af:99:34   ipv4       10.44.50.12/24    gluster02   -

Go to the Ansible directory

bash


cd ~/kvm-gluster-lab/ansible

Edit the invetory.yaml file and update the ansible_host lines to reflect the nodes addresses
bash
```
vim inventory.yaml
```

Example output

yaml


---

gluster:
  hosts:
    gluster01:
      ansible_host: 10.44.50.100
    gluster02:
      ansible_host: 10.44.50.12

gluster_extended:
  hosts:
    gluster03:
      ansible_host: 10.44.50.77

gluster_client:
  hosts:
    client01:
      ansible_host: 10.44.50.10

Once the inventory file is updated, run the initial_setup.yaml playbook

bash


ansible-playbook -i inventory.yaml initial_setup.yaml

Demo of the Ansible Install

Setup Client⌗

Here we’ll look at mounting the volume with Glusterfs

All steps here are run from client01

Install the Gluster client

bash


yum install glusterfs glusterfs-fuse -y

Test mounting the Gluster volume with fuse

bash


mount -t glusterfs gluster01:/volume01 /mnt

Check it has mounted

bash


df -h

Example output

bash


Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             1.9G     0  1.9G   0% /dev
tmpfs                1.9G     0  1.9G   0% /dev/shm
tmpfs                1.9G   17M  1.9G   1% /run
tmpfs                1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vda1             40G  922M   40G   3% /
tmpfs                379M     0  379M   0% /run/user/0
gluster01:/volume01   10G  135M  9.9G   2% /mnt

Unmount Gluster

bash


umount /mnt

Depending on the style of testing needed against the volume will depend on how the volume is mounted. Here is an example for mounting the volume to the /home directory

Example /etc/fstab mount

bash


gluster01:/volume01     /home   glusterfs       defaults        0       0

Mount the volume

bash


mount -a

Check mount

bash


df -h

Example output

bash


Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             1.9G     0  1.9G   0% /dev
tmpfs                1.9G     0  1.9G   0% /dev/shm
tmpfs                1.9G   17M  1.9G   1% /run
tmpfs                1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vda1             40G  922M   40G   3% /
tmpfs                379M     0  379M   0% /run/user/0
gluster01:/volume01   10G  135M  9.9G   2% /home

Checkpoint⌗

At this point there should be a two node Gluster cluster, with one volume exported to a client.

The next section will detail adding the third node and extending the Gluster volume over this node.

If this is as far as the lab is required, see the cleanup section to remove the lab.

Third Replica⌗

In this section we will configure the third node and add it into the Gluster Volume. As with the setup Gluster steps, there will be a manual setup and an Ansible setup

Setup Gluster⌗

First power on gluster03

bash


virsh start gluster03.gluster.lab

Confirm the virtual machine has started

bash


vish list

Example output

bash


 Id   Name                    State
---------------------------------------
 9    gluster01.gluster.lab   running
 10   gluster02.gluster.lab   running
 11   gluster03.gluster.lab   running
 12   client01.gluster.lab    running

Manual Setup⌗

The following steps will need running on gluster03

(Optional) Install a text editor of choice. vi is available on the nodes

bash


yum install -y vim

Install the Gluster yum repository. Here we will be installing Gluster 6 so that this lab can be used for a future lab.

bash


yum install centos-release-gluster6 -y

Now install glusterfs server

bash


yum install glusterfs-server -y

Enable the Gluster service

bash


systemctl enable glusterd

Start the Gluster service

bash


systemctl start glusterd

Add the first disk to a Physical Volume

bash


pvcreate /dev/vdb

Add the second disk to a Physical Volume

bash


pvcreate /dev/vdc

Confirm both drives are visible as Physical Volumes

bash

pvs

Example output

bash


  PV         VG Fmt  Attr PSize PFree
  /dev/vdb      lvm2 ---  5.00g 5.00g
  /dev/vdc      lvm2 ---  5.00g 5.00g

Create the Volume group

bash


vgcreate brick /dev/vdb /dev/vdc

Check the Volume Group

bash

vgs

Example output

bash


  VG    #PV #LV #SN Attr   VSize VFree
  brick   2   0   0 wz--n- 9.99g 9.99g

Create the Logical Volume for the Gluster brick

bash


lvcreate -l 100%FREE -T brick -n brick01

Check Logical Volume has created

bash

lvs

Example output

bash


  LV      VG    Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  brick01 brick twi-a-tz-- <9.97g             0.00   10.61

Create a filesystem on the Logical Volume

bash


mkfs.xfs -i size=512 /dev/brick/brick01

Create a mount directory for the brick

bash


mkdir -p /bricks/brick01

Add the brick mount to /etc/fstab

bash


/dev/brick/brick01        /bricks/brick01 xfs     rw,noatime,inode64,nouuid   1   2

Mount the volume

bash


mount -a

Check the volume has mounted

bash


df -h

Example output

bash


Filesystem                 Size  Used Avail Use% Mounted on
devtmpfs                   3.8G     0  3.8G   0% /dev
tmpfs                      3.9G     0  3.9G   0% /dev/shm
tmpfs                      3.9G   17M  3.9G   1% /run
tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1                   40G  1.1G   39G   3% /
tmpfs                      783M     0  783M   0% /run/user/0
/dev/mapper/brick-brick01   10G   33M   10G   1% /bricks/brick01

Create a brick directory on the mount point

bash


mkdir -p /bricks/brick01/brick

Ansible Setup⌗

Within the GitHub repository there is an ansible directory. Assuming that the inventory.yaml was setup in a previous step we will just run the playbook for gluster03

Move to the Ansible directory

bash


cd ~/kvm-gluster-lab/ansible

Run the playbook

bash


ansible-playbook -i inventory.yaml third_node.yaml

Add Replica⌗

These steps need to be run from an existing Gluster node.

Probe the new node

bash


gluster peer probe gluster03

Confirm the new node has been added

bash


gluster peer status

Example output

bash


Number of Peers: 2

Hostname: gluster02
Uuid: b13fffb6-a127-4895-857c-8cce6f2dd34c
State: Peer in Cluster (Connected)

Hostname: gluster03
Uuid: 9b03a0e9-77dd-4a70-a98b-141842fb92c1
State: Peer in Cluster (Connected)

Check the current volume

bash


gluster volume status volume01

Example output

bash


Status of volume: volume01
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gluster01:/brick/brick01/brick        49153     0          Y       19468
Brick gluster02:/brick/brick01/brick        49153     0          Y       18167
Self-heal Daemon on localhost               N/A       N/A        Y       19489
Self-heal Daemon on gluster02               N/A       N/A        Y       18188
Self-heal Daemon on gluster03               N/A       N/A        Y       17326
 
Task Status of Volume volume01
------------------------------------------------------------------------------
There are no active volume tasks

Add the new brick on gluster03 to the volume

bash


gluster volume add-brick volume01 replica 3 gluster03:/brick/brick01/brick

Check the new brick has been added to the volume

bash


gluster volume status volume01

Example output

bash


Status of volume: volume01
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gluster01:/brick/brick01/brick        49153     0          Y       19468
Brick gluster02:/brick/brick01/brick        49153     0          Y       18167
Brick gluster03:/brick/brick01/brick        49152     0          Y       17384
Self-heal Daemon on localhost               N/A       N/A        Y       20024
Self-heal Daemon on gluster02               N/A       N/A        Y       18596
Self-heal Daemon on gluster03               N/A       N/A        Y       17405
 
Task Status of Volume volume01
------------------------------------------------------------------------------
There are no active volume tasks

Trigger a volume heal once it’s confirmed the new brick is in the volume

bash


gluster volume heal volume01

Check the status of the volume heal

bash


gluster volume heal volume01 info

Example output of when the heal has finished

bash


Brick gluster01:/brick/brick01/brick
Status: Connected
Number of entries: 0

Brick gluster02:/brick/brick01/brick
Status: Connected
Number of entries: 0

Brick gluster03:/brick/brick01/brick
Status: Connected
Number of entries: 0

Summary⌗

By this point there should be a three node Gluster cluster, with a 3x replicated volume presented to a client machine.

The cleanup section below will remove everything create by this lab.

Cleanup⌗

Within the GitHub repository is a teardown.sh script, that will remove all the VMs, the storage and the network.

To clean up the lab

bash


bash ~/kvm-gluster-lab/teardown.sh

Simple Gluster Lab

Contents

Introduction⌗

Initial Cluster build⌗

Setup VMs⌗

Setup Gluster⌗

Manual Setup⌗

Ansible Setup⌗

Setup Client⌗

Checkpoint⌗

Third Replica⌗

Setup Gluster⌗

Manual Setup⌗

Ansible Setup⌗

Add Replica⌗

Summary⌗

Cleanup⌗