3505 - Promoting a Grid Master Candidate (GMC) in the NIOS Grid

Scenario

Your team is performing a Disaster Recovery (DR) exercise, during which you need to switch all Grid management functions to the backup or secondary site, verify everything works, and then switch back to the primary site.

Estimate Completion Time

• 30 to 45 minutes

Credentials

Requirements

• Administrative access to the Grid

• Console access to NIOS appliances

Course References

• 1001: NIOS Grid Overview

• 1002: Using the NIOS Grid Manager UI

• 2002: Managing NIOS Grid Members

• 3001: Setting up the NIOS Grid

Lab Initiation

Access jump-desktop

Once the lab is deployed, you can access the virtual machines required to complete this lab activity. To initiate the lab, click on the jump-desktop tile and login to the Linux UI:

Username: training

Password: infoblox

Initiate lab

To initiate the lab, double-click the Launch Lab icon on the Desktop.

Choose the lab number from the list and click OK.

After clicking OK, you will see a pop-up message with a brief description of the lab task. If the description looks correct, click Yes to continue lab initiation.

Lab initiation will take a couple of minutes to finish.

Once complete, you will see another pop-up message with the login credentials and the URL for the Grid Manager’s User Interface. Note that the credentials may differ from those from prior labs.

Tasks

Task 1: Obtaining information on GMC

Login to the GM web interface and collect the network information of the Grid Master Candidate. If more than one GMC is configured, please write down all their network information. You will need this information for the next task.

Task 2: Documenting service impact

As part of the DR exercise, you need to provide the service impact and plan for an outage (if appropriate). Please answer the following questions:

1. Are DNS/DHCP/IPAM services affected by this activity?

2. Will the service be unreachable for more than 60 minutes?

3. After promoting the backup site, do NIOS administrators need to log in to a different address to access the GM user interface to make changes?

Task 3: Testing the GMC connection to Grid members

Test whether the Grid Master Candidate is connected to the rest of the Grid members using the GMC Promote Test tool

Task 4: Promoting the GMC

Login to the GMC via console access. Execute the command set promote_master.

Task 5: Verifying functionality

Login to the newly promoted GM at its IP address. Please verify that all members have rejoined successfully. The original GM (nios-1) should have rejoined the Grid, and automatically change its role to the new GMC.

Task 6: Promoting original GM

Login to the original GM (nios-1) via console access. Execute the command set promote_master.

Task 7: Verifying functionality

Login to the original GM at its IP address. Please note that all members have rejoined successfully. The GMC (nios-5) should have rejoined the Grid, and automatically change its role to GMC.

Solutions

Task 1 Solution: Obtaining information on GMC

Log in to the GM web interface and examine the services active in the grid and the members on which these services run.

1. Login to the NIOS GM web interface.

2. Navigate to Grid → Grid Manager → Members.

3. Note the services running for ibns2.techblue.net and ibns1.techblue.net. If a service runs correctly, it will be shown with a green icon.

Alternatively, you may find the same information in the web interface this way, one service at a time:

1. Login to NIOS GM web interface

2. Navigate to  Grid → Grid Manager → Services.

3. Click on DHCP to verify if DHCP service is running for ibns2.techblue.net and ibns1.techblue.net .

4. Repeat the same step for DNS and other services.

Task 2 Solution: Documenting service impact

As part of the DR exercise, you need to provide the service impact and plan for an outage (if appropriate). Please answer the following questions:

1. Are DNS/DHCP/IPAM services affected by this activity?
   Answer: Yes, services will restart as a result of GMC promotion. Depending on the configuration and dataset, this can take several minutes. You may choose to restart services one member at a time at an interval (default 30 seconds), instead of restarting services on all members at the same time. This may or may not introduce service disruption, depending on your architecture design. For example, if you have DNS anycast or DHCP failover between members and sites, you may not have a service impact during a GMC promotion.

2. Do HA pairs protect me from service disruptions during a GMC promotion?
   Answer: No. During a GMC promotion, the HA failover does not occur. Services on the active node will restart.

3. Will the service be unreachable for more than 60 minutes?
   Answer: This is unlikely. Services will only be affected during the service restart window.

4. After promoting the backup site, do NIOS administrators need to log in to a different address to access the GM user interface to make changes?
   Answer: Administrators are automatically redirected to the new GM’s login page.

Task 3 Solution: Testing the GMC connection to Grid members

Before promoting a Grid Master Candidate, It is highly recommended to check whether the Grid Master Candidate is connected to the rest of the Grid members using the GMC Promote Test feature, this is to avoid any service disruptions during or after the GMC promotion due to blocked ports needed to communicate with the rest of the Grid members.

1. Login to the NIOS GM web interface.

2. Navigate to  Grid → Grid Manager.

3. From the Toolbar click GMC Promote Test.

   1. From the Select GMC drop-down list, select the Grid Master Candidate that you want to promote to Grid Master.

      

   2. Leave the default timeout timer at 120 seconds.

   3. In the Members table, select all the Grid members.
      NOTE: The Grid Master Candidate must establish a connection with all the selected members.

   4. Click Start to start the test promotion.

      

   5. Click GMC Promotion Test Results to view the status of the test promotion.

Task 4 Solution: Promoting the GMC

Login to the GMC via console access. Execute the command set promote_master.

1. Access the console of nios-5 VM using default credentials.

2. At the prompt, type the command set promote_master.

3. When prompted to add a delay between notification to Grid members, say Y, and accept the default delay of 30 seconds.

4. When prompted to confirm you want to promote this candidate, say Y to accept.

5. You will now see the NIOS appliance restart.

6. Go back to the VM jump-desktop.

7. Wait a couple of minutes and then refresh your web browser.

   1. If the web browser says, “Refused to connect” wait a few more minutes and refresh again.

8. You will be automatically redirected to the new GM’s IP address (10.200.0.100).

9. Login with the default credentials

10. Go to Grid→Grid Manager→Members

11. All members should be online and running

12. You will now see that the icons for ibgm and ibgmc have been swapped around, ibgmc is now the Grid Master, ibgm is a Grid Master Candidate.

Task 5 Solution: Verifying functionality

Login to the newly promoted GM at its IP address (10.200.0.100). Please verify that all members have rejoined successfully. The original GM (nios-1) should have rejoined the Grid, and automatically change its role to the new GMC.

1. Go to Grid→Grid Manager→Members.

2. Check the Status of members and verify that they have all successfully joined the newly promoted ibgmc

Task 6 Solution: Promoting original GM

Login to the original GM (nios-1) via console access. Execute the command set promote_master.

1. Access the console of nios-1 VM using default credentials.

2. At the prompt, type the command set promote_master

3. When prompted to add a delay between notification to Grid members, say Y, and accept the default delay of 30 seconds

4. When prompted to confirm you want to promote this candidate, say Y to accept

5. You will no see the NIOS appliance restart.

6. Go back to jump-desktop.

7. Wait a couple of minutes and then refresh your web browser

   1. If the web browser says, “Refused to connect” wait a few more minutes and refresh again.

8. You will be automatically redirected to the new Grid Master’s IP address (10.100.0.100).

9. Login with the default credentials.

10. Go to Grid→Grid Manager→Members.

11. All members should be online and running.

12. You will now see that the icons for ibgm and ibgmc have once again swapped around, ibgm is once again the Grid Master, ibgmc is back to being the Grid Master Candidate.

Task 7 Solution: Verifying functionality

Login to the newly promoted GM at its IP address (10.100.0.100). Please verify that all members have rejoined successfully.

1. Go to Grid→Grid Manager→Members.

2. Check the Status of members and verify that they have all successfully joined the newly promoted ibgm