Download
Popular Search installationapivitaldebianprovision

Changing One of the Servers in DRBD High Availability

There could be occasions where you need to change one of the servers. This can be due to one of them getting damaged or hardware modification.

Let’s look into changing the Secondary Server. First, we will proceed to destroy the cluster on the Primary Server.

root@debian:~# pcs cluster stop 
root@debian:~# pcs cluster destroy 
root@debian:~# systemctl disable pcsd.service 
root@debian:~# systemctl disable corosync.service 
root@debian:~# systemctl disable pacemaker.service 
root@debian:~# systemctl stop pcsd.service 
root@debian:~# systemctl stop corosync.service 
root@debian:~# systemctl stop pacemaker.service

Next, if we still have access to the Secondary Server, we destroy the cluster as well on the Secondary Server.

root@debian:~# pcs cluster stop --force
root@debian:~# pcs cluster destroy
root@debian:~# systemctl disable pcsd.service 
root@debian:~# systemctl disable corosync.service 
root@debian:~# systemctl disable pacemaker.service
root@debian:~# systemctl stop pcsd.service 
root@debian:~# systemctl stop corosync.service 
root@debian:~# systemctl stop pacemaker.service

Since when destroying the cluster the DRBD volume is unmounted, in the Primary Server we must mount it again manually to avoid interrupting the normal operation of our services. Run the following commands on the Primary Server.

root@debian:~# drbdadm up drbd0
root@debian:~# drbdadm primary drbd0 --force
root@debian:~# mount /dev/drbd0 /vpbx_data

Now, enable the services on the Primary Server to make sure our server continues to work as normal.

root@debian:~# systemctl enable asterisk
root@debian:~# systemctl restart asterisk
root@debian:~# systemctl enable mariadb
root@debian:~# systemctl restart mariadb
root@debian:~# systemctl enable fail2ban
root@debian:~# systemctl restart fail2ban
root@debian:~# systemctl enable vpbx-monitor
root@debian:~# systemctl restart vpbx-monitor

At this moment, the Primary Server is now working as normal. We must now configure the replica with our new server.

First, prepare the new server following the steps for the installation in Section 13, Lesson 1, with the disk partition and the free space. Then enable remote access with the root user.
Change the IP address to a static IP address. Install the dependencies. Change the hostname in the Web UI and CLI using the same hostname as the old Secondary Server. Create the sda3 partition using the fdisk command. And finally, configure the firewall with the ports established in the table from lesson 1. These are the steps for the manual HA setup presented in lesson 1.
This should only be configured in the new Secondary Server.

Now, we will proceed to format the new partition on the new Secondary Server using the following commands.

root@debian:~# mkdir /vpbx_data 
root@debian:~# mke2fs -j /dev/sda3 
root@debian:~# dd if=/dev/zero bs=1M count=500 of=/dev/sda3; sync

Load the DRBD module and enable the service on the new Secondary Server using the following commands.

root@debian:~# modprobe drbd
root@debian:~# systemctl enable drbd.service

Next, we must create new configuration files called drbd0.res in the /etc/drbd.d directory on the new Secondary Server for the new resource named drbd0.

root@debian:~# nano /etc/drbd.d/drbd0.res

Add the following content.

resource drbd0 {
startup {
        wfc-timeout  5;
      outdated-wfc-timeout 3;
        degr-wfc-timeout 3;
      outdated-wfc-timeout 2;
}
syncer {
      rate 10M;
      verify-alg md5;
}
net {
    after-sb-0pri discard-older-primary;
    after-sb-1pri discard-secondary;
    after-sb-2pri call-pri-lost-after-sb;
}
handlers {
    pri-lost-after-sb "/sbin/reboot";
}
on vitalpbx-primary.local {
      device /dev/drbd0;
      disk /dev/sda3;
      address 192.168.10.31:7789;
      meta-disk internal;
}
on vitalpbx-secondary.local {
      device /dev/drbd0;
      disk /dev/sda3;
      address 192.168.10.32:7789;
      meta-disk internal;
      }
}

Note: Although the access interfaces can be used, which in this case is ETH0. It is recommended to use an interface (ETH1) for synchronization, this interface must be directly connected between both servers.

Initialize the metadata storage on each server by executing the following command on the new Secondary Server.

root@debian:~# drbdadm create-md drbd0
Writing meta data...
New drbd meta data block successfully created.

Run the following command on the new Secondary Server to start the drbd0.

You can check the current status of the synchronization using the cat /proc/drbd command.

Once the synchronization is done, you can follow the steps from Section 13, Lesson 1 for configuring the cluster starting with the hacluster user and password up to the creation of the fail2ban service. Then, follow the steps for creating the bascul command and the creation of the role command. All of this is for the new Secondary Server.

After this, you will now have replaced the old Secondary Server with a new one!

Now, let’s look into changing the Primary Server. This method differs from changing the Secondary Server.

First, we proceed to destroy the cluster in the Primary Server if we still can. Run the following commands in the Primary Server.

root@debian:~# pcs cluster stop
root@debian:~# pcs cluster destroy
root@debian:~# systemctl disable pcsd.service 
root@debian:~# systemctl disable corosync.service 
root@debian:~# systemctl disable pacemaker.service
root@debian:~# systemctl stop pcsd.service 
root@debian:~# systemctl stop corosync.service 
root@debian:~# systemctl stop pacemaker.service

Next, we do the same in the Secondary Server with the following commands.

root@debian:~# pcs cluster stop --force
root@debian:~# pcs cluster destroy
root@debian:~# systemctl disable pcsd.service
root@debian:~# systemctl disable corosync.service
root@debian:~# systemctl disable pacemaker.service
root@debian:~# systemctl stop pcsd.service
root@debian:~# systemctl stop corosync.service
root@debian:~# systemctl stop pacemaker.service

Since by destroying the cluster the DRBD unit is unmounted, in the Secondary Server, we must mount it again manually to avoid interrupting the normal operation of our services. Run the following commands in the Secondary Server.

root@debian:~# drbdadm up drbd0
root@debian:~# drbdadm primary drbd0 --force
root@debian:~# mount /dev/drbd0 /vpbx_data

Now, enable the services on the Secondary Server to make sure that our server continues to work as normal.

root@debian:~# systemctl enable asterisk
root@debian:~# systemctl restart asterisk
root@debian:~# systemctl enable asterisk
root@debian:~# systemctl restart asterisk

With this, our Secondary Server is now working as normal. We now proceed to configure the replica with our new server.

First, prepare the new server following the steps for the installation in Section 13, Lesson 1, with the disk partition and the free space. Then enable remote access with the root user. Change the IP address to a static IP address. Install the dependencies. Change the hostname in the Web UI and CLI using the same hostname as the old Primary Server. Create the sda3 partition using the fdisk command. And finally, configure the firewall with the ports established in the table from lesson 1. These are the steps for the manual HA setup presented in lesson 1. This should only be configured in the new Primary Server.

Now, proceed to format the new partition on the new Primary Server using these commands.

root@debian:~# mkdir /vpbx_data
root@debian:~# mke2fs -j /dev/sda3
root@debian:~# dd if=/dev/zero bs=1M count=500 of=/dev/sda3; sync

Then, load the module on the new Primary Server and enable the service with the following commands.

root@debian:~# modprobe drbd
root@debian:~# systemctl enable drbd.service

Next, back up the original global_common.conf file, and create a new one using nano on the Primary Server. Use the following commands on the new Primary Server.

root@debian:~# mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.orig
root@debian:~# nano /etc/drbd.d/global_common.conf

Add the following content.

global {
  usage-count no;
}
  common {
net {
  protocol C;
  }
}

Next, we must create new configuration files called drbd0.res in the /etc/drbd.d directory on the new Primary Server for the new resource named drbd0.

root@debian:~# nano /etc/drbd.d/drbd0.res

Add the following content.

resource drbd0 {
startup {
        wfc-timeout  5;
      outdated-wfc-timeout 3;
        degr-wfc-timeout 3;
      outdated-wfc-timeout 2;
}
syncer {
      rate 10M;
      verify-alg md5;
}
net {
    after-sb-0pri discard-older-primary;
    after-sb-1pri discard-secondary;
    after-sb-2pri call-pri-lost-after-sb;
}
handlers {
    pri-lost-after-sb "/sbin/reboot";
}
on vitalpbx-primary.local {
      device /dev/drbd0;
      disk /dev/sda3;
      address 192.168.10.31:7789;
      meta-disk internal;
}
on vitalpbx-secondary.local {
      device /dev/drbd0;
      disk /dev/sda3;
      address 192.168.10.32:7789;
      meta-disk internal;
      }
}

Note: Although the access interfaces can be used, which in this case is ETH0. It is recommended to use an interface (ETH1) for synchronization, this interface must be directly connected between both servers.

Initialize the metadata storage on each server by executing the following command on the new Primary Server.

root@debian:~# drbdadm create-md drbd0
Writing meta data...
New drbd meta data block successfully created.

On the new Primary Server, run the following command to start the drbd0.

root@debian:~# drbdadm up drbd0

You can check the current status of the synchronization while it’s being performed. The cat /proc/drbd command displays the creation and synchronization progress of the resource.

Once the synchronization is done, you can follow the steps from Section 13, Lesson 1 for configuring the cluster starting with the hacluster user and password up to the creation of the fail2ban service. Then, follow the steps for creating the bascul command and the creation of the role command. All of this is for the new Primary Server.

After this, you will now have replaced the old Primary Server with a new one!

With this, you now have the necessary tools to run VitalPBX in a high-availability environment. Allowing you to rest assured that you have two servers working together for minimal to no downtime.

What are your feelings
Updated on December 16, 2023