3512 - Troubleshoot Grid Master Candidate Promotion in NIOS
Scenario
As part of the Disaster Recovery exercise, you and your fellow grid administrators need to provide a report on NIOS Disaster Recovery readiness. Verify the GM redundancy, by making sure that the GM and GMC will be able to switch over without impacting production too severely. There is a tool in NIOS that will allow you to test GMC promotion, which will provide insight on any potential issues.
Estimate Completion Time
25 to 35 Minutes
Credentials
Description | Username | Password | URL or IP |
|---|---|---|---|
Grid Manager UI | admin | infoblox |
Course References
3001: Setting up the NIOS Grid
3002: Managing NIOS Grid and Grid Members
Lab Initiation
Access jump-desktop
Once the lab is deployed, you can access the virtual machines required to complete this lab activity. To initiate the lab, click on the jump-desktop tile and login to the Linux UI:
Username: training
Password: infoblox
Initiate lab
To initiate the lab, double-click the Launch Lab icon on the Desktop.
Launch Lab
Choose the lab number from the list and click OK.
After clicking OK, you will see a pop-up message with a brief description of the lab task. If the description looks correct, click Yes to continue lab initiation.
Lab initiation will take a couple of minutes to finish.
Once complete, you will see another pop-up message with the login credentials and the URL for the Grid Manager’s User Interface. Note that the credentials may differ from those from prior labs.
Tasks
Task 1: Perform a GMC Promotion Test
Before promoting, use the ‘GMC Promote Test’ tool from Grid Manager and record the results.
Task 2: Perform Grid Master Candidate Promotion
Despite the results, you decide to promote the Grid Master Candidate (ibgmc) anyway. Observe the behavior.
Task 3: Perform Grid Master Candidate Promotion
Once you have identified any issues, fail back the Grid so that member (nios-1) is the Grid Master again.
Solutions
Task 1: Perform a GMC Promotion Test
Before promoting, use the ‘GMC Promote Test’ tool from Grid Manager and record the results.
In the Grid Manager UI, navigate to Grid → Grid Manager, from the Toolbar, select GMC Promote Test
Select the GMC ibgmc.techblue.net, change the timeout to 10 seconds (so we do not wait as long), check the member ibns1.techblue.net, and click Start.
After a while, we see the results, clearly indicating that port 2114 connectivity has failed.
In this exercise, the NIOS admin is only responsible for identifying any potential issues with GMC failover readiness. If any concerns are found during the assessment, the NIOS admin will report them to the appropriate contact for further investigation, involving the network team and/or firewall team if necessary.
Task 2: Perform Grid Master Candidate Promotion
Despite the results, you decide to promote the Grid Master Candidate (ibgmc) anyway. Observe the behavior.
Connect to member (nios-5) using SSH or the console.
Type the command:
set promote_master
Follow the prompts to confirmAfter typing in the command, in the original (nios-1) Grid Manager UI, you will see ibgmc being marked as Offline, and after a while, the web UI becomes unresponsive. Use the web browser refresh button, and you will be redirected to login to https://10.200.0.100/, the newly promoted Grid Master (nios-5)
After logging in to the new Grid Master (nios-5), you will see that HA member (ibns1) is offline, and the original Grid Master (nios-1) is reconnected as Grid Master Candidate
Member (ibns1) will fail to establish VPN to (nios-5) due to the blocked port 2114
Task 3: Perform Grid Master Candidate Promotion
Once you have identified any issues, fail back the Grid so that member (nios-1) is the Grid Master again.
Connect to member (nios-1) using SSH or the console.
Type the command:
set promote_master