3511 - Troubleshooting member joining in NIOS
Scenario
After a hardware failure, your team has acquired a replacement NIOS appliance. However, it is having trouble joining the Grid. Please troubleshoot, locate the root cause(s), and fix.
Estimate Completion Time
25 to 35 Minutes
Credentials
Description | Username | Password | URL or IP |
|---|---|---|---|
Grid Manager UI | admin | infoblox |
Course References
3001: Setting up the NIOS Grid
3002: Managing NIOS Grid and Grid Members
Lab Initiation
Access jump-desktop
Once the lab is deployed, you can access the virtual machines required to complete this lab activity. To initiate the lab, click on the jump-desktop tile and login to the Linux UI:
Username: training
Password: infoblox
Initiate lab
To initiate the lab, double-click the Launch Lab icon on the Desktop.
Launch Lab
Choose the lab number from the list and click OK.
After clicking OK, you will see a pop-up message with a brief description of the lab task. If the description looks correct, click Yes to continue lab initiation.
Lab initiation will take a couple of minutes to finish.
Once complete, you will see another pop-up message with the login credentials and the URL for the Grid Manager’s User Interface. Note that the credentials may differ from those from prior labs.
Tasks
Task 1: Joining new member to the Grid
Use the CLI to access the device nios-4, and configure it with the following information to join the Grid:
IP Address: 10.200.0.105
Subnetmask: 255.255.255.0 (default)
Gateway: 10.200.0.1 (default)
GM VIP: 10.100.0.100
Grid Name: Infoblox (default, case sensitive)
Shared Secret: test (default, case sensitive)
Join the member, observe the error messages. Analyze the log messages on both the GM and the joining member to locate the root cause(s).
Solutions
Task 1: Joining new member to the Grid
There are 2 main causes for the join failure: missing license, and member type mismatch.
Cause 1: Missing License
You will not be able to see these error messages on the GM. Because the member is unable to join the GM, it also cannot send the logs to the GM to be viewed centrally. You have to watch the console screen on the member to see these error messages.
After attempting to join nios-4 to the Grid, you can see the error messages during its bootup screen:
If you have attempted to join once, and see it fail, the second join attempt will show you the reason again. On the CLI, run the set membership command again:
Infoblox > set membership
Join status: No Grid/NIOS/Multi-Grid Management license for this device.
Enter New Grid Master VIP [Default 10.100.0.100]:
To fix this problem, use the set temp license command to assign yourself the following temporary licenses: NIOS (4) and DNSone with Grid (2). You will need to run the command twice to add both licenses.
If prompted for the model, for model, choose IB-V1415 (7):
Cause 2: Member Type Mismatch
The second cause is the member type is mismatched. There are 3 ways you can come to this conclusion: 1) attempt join on nios-4 and see the warning message, 2) download support bundle from Grid Master, or 3) watch live logs on GM as nios-4 joins.
1) Warning on Failed Member
When performing a join from the CLI from the member, using commands such as ‘set network’ or ‘set membership’, it restarts the member, but does not provide detailed message as to why the join failed
The only message we see on the member is during the restart, there is a message the indicates the joining has failed:
Contacting the grid master at 10.100.0.100....
System restart: join failure...
Infoblox system initializing...
If we try to join again, at the start of the joining attempt, a more detailed warning message tells us why the last join attempt failed:
Infoblox > set membership
Join status: Member Type Mismatch on the grid master.
Enter New Grid Master VIP [Default 10.100.0.100]:
2) Support Bundle
We can also download the support bundle from the Grid Master and analyze the logs.
Since this is related to joining the Grid, we should start with locating the clusterd messages in the file log.
Below are entries we should find in the file infoblox.log, indicating the time of join failure and the reason:
[2021/08/20 13:40:30.142] (8470 /infoblox/one/bin/clusterd) handshake.c:820 cd_master_handshake_tunnel_request(): INFO: set VPN channel: 169.254.0.5 for member: 10.201.0.105
[2021/08/20 13:40:30.448] (8470 /infoblox/one/bin/clusterd) master.c:1096 cd_replica_login(): Starting Cluster Join for hardware (IB-V1415) member (VNIOS)
[2021/08/20 13:40:30.449] (8470 /infoblox/one/bin/clusterd) master.c:1265 cd_replica_login(): Member type mismatch. DB: INFOBLOX, MSG: IB-V1415
3) Watch Live Debug Log
If the situation permit, you can also perform the join on the member, and log in to the Grid Master CLI to watch the debug log as messages come in.
You may use the CLI command show log debug follow /clusterd/, this tells syslog to only display debug messages containing the word clusterd.
Infoblox > show log debug follow /clusterd/
[2021/08/20 14:08:54.953] (2232 -serial_console) 1449984 admin> show log debug follow /clusterd/
[2021/08/20 14:08:58.591] (8470 /infoblox/one/bin/clusterd) util.c:5030 cd_signal_revision_change(): signalled revision change 39
[2021/08/20 14:09:01.958] (8470 /infoblox/one/bin/clusterd) handshake.c:820 cd_master_handshake_tunnel_request(): INFO: set VPN channel: 169.254.0.5 for member: 10.201.0.105
[2021/08/20 14:09:02.261] (8470 /infoblox/one/bin/clusterd) master.c:1096 cd_replica_login(): Starting Cluster Join for hardware (IB-V1415) member (VNIOS)
[2021/08/20 14:09:02.261] (8470 /infoblox/one/bin/clusterd) master.c:1265 cd_replica_login(): Member type mismatch. DB: INFOBLOX, MSG: IB-V1415
[2021/08/20 14:09:02.266] (8470 /infoblox/one/bin/clusterd) util.c:5030 cd_signal_revision_change(): signalled revision change 40
To fix this issue, edit the member in Grid Manager UI to change the member. Navigate to Grid → Grid Manager → ibns2 → General, change Member Type from Infoblox to Virtual NIOS.