3522 - Troubleshooting auto-generated DNS records in NIOS
Scenario
Your received a report from internal users that they are experiencing slowness or timeout when attempting to resolve the internal ibns2.techblue.net. After some initial investigation, your team has determined that this name resolves to 2 different IP addresses, 1 of which should not have been listed. However, you cannot delete this A record from the Grid. Please investigate the issues around ibns2.techblue.net and provide a fix.
Estimate Completion Time
20 to 30 minutes
Credentials
Description | Username | Password | URL or IP |
|---|---|---|---|
Grid Manager UI | admin | infoblox |
Requirements
Administrative access to the Grid
Course References
3006: Configuring NAT’ed Glue Records in NIOS
Lab Initiation
Access jump-desktop
Once the lab is deployed, you can access the virtual machines required to complete this lab activity. To initiate the lab, click on the jump-desktop tile and login to the Linux UI:
Username: training
Password: infoblox
Initiate lab
To initiate the lab, double-click the Launch Lab icon on the Desktop.
Launch Lab
Choose the lab number from the list and click OK.
After clicking OK, you will see a pop-up message with a brief description of the lab task. If the description looks correct, click Yes to continue lab initiation.
Lab initiation will take a couple of minutes to finish.
Once complete, you will see another pop-up message with the login credentials and the URL for the Grid Manager’s User Interface. Note that the credentials may differ from those from prior labs.
Tasks
Take 1: Verifying the current behavior
Using lookup tools such as dig, verify what the current configuration and behaviors are for the domain name ibns2.techblue.net.
Hint: Start by finding out what IP addresses the name ibns2.techblue.net is pointing to.
Task 2: Researching and updating configurations on the Grid
Research the configuration on the Grid to locate where the erroneous information originates from and update the configuration.
Solutions
Take 1 Solution: Verifying the current behavior
Using lookup tools such as dig and examining Grid configuration, verify what the current configuration and behaviors are for the domain name ibns2.techblue.net.
Figure 3522-1 shows using the dig command to query the member ibns1 (10.100.0.105) to find out what the name ibns2.techblue.net resolves to. We see 2 IP addresses in the answer.
Figure 3522-1: Looking Up IBNS2 IP Addresses
$ dig @10.100.0.105 ibns2.techblue.net.
; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> @10.100.0.105 ibns2.techblue.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45777
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: cab330ebfc3a19ff4ad7165064c7ce4e5bdfb31f8e67c1c6 (good)
;; QUESTION SECTION:
;ibns2.techblue.net. IN A
;; ANSWER SECTION:
ibns2.techblue.net. 28800 IN A 10.200.0.105
ibns2.techblue.net. 28800 IN A 198.51.100.137
;; Query time: 0 msec
;; SERVER: 10.100.0.105#53(10.100.0.105) (UDP)
;; WHEN: Mon Jul 31 15:07:58 UTC 2023
;; MSG SIZE rcvd: 107Detailed Analysis of Figure 3522-1
Line 1: You may query either ibns1 (10.100.0.105) or ibns2 (10.200.0.105). Both yield the same answer. This emulates what the internal users would see when attempting to resolve this name.
Line 8: We see the aa flag, indicating that the answers below are authoritative from the Grid. In other words, if there are mistakes or errors, they are configured on the Grid.
Line 17 to 18: We see that there are 2 IPv4 addresses, 10.200.0.105 and 198.51.100.137 for this name. It is legal for a name to have 2 (or more) IP addresses. However, if one of them is not reachable, it may cause slowdown or timeout because the default behavior is round-robin.
Verifying reachability
When using the command ping to ping each of the 2 IP addresses, we see that only one of them is reachable from jump-desktop:
Figure 3522-2: Ping by Address
$ ping -c 4 10.200.0.105
PING 10.200.0.105 (10.200.0.105) 56(84) bytes of data.
64 bytes from 10.200.0.105: icmp_seq=1 ttl=63 time=0.462 ms
64 bytes from 10.200.0.105: icmp_seq=2 ttl=63 time=0.501 ms
64 bytes from 10.200.0.105: icmp_seq=3 ttl=63 time=0.523 ms
64 bytes from 10.200.0.105: icmp_seq=4 ttl=63 time=0.530 ms
--- 10.200.0.105 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3062ms
rtt min/avg/max/mdev = 0.462/0.504/0.530/0.026 ms
training@jump-desktop:~ $ ping -c 4 198.51.100.137
PING 198.51.100.137 (198.51.100.137) 56(84) bytes of data.
--- 198.51.100.137 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3071ms
This means, attempts to reach the name ibns2.techblue.net will result in a 50% failure date, due to the round-robin behavior. We can verify this behavior by trying to ping the host by its name:
Figure 3522-3: Ping by Name
$ ping -c 4 ibns2.techblue.net
PING ibns2.techblue.net (10.200.0.105) 56(84) bytes of data.
64 bytes from 10.200.0.105 (10.200.0.105): icmp_seq=1 ttl=63 time=0.422 ms
64 bytes from 10.200.0.105 (10.200.0.105): icmp_seq=2 ttl=63 time=0.417 ms
64 bytes from 10.200.0.105 (10.200.0.105): icmp_seq=3 ttl=63 time=0.344 ms
64 bytes from 10.200.0.105 (10.200.0.105): icmp_seq=4 ttl=63 time=0.393 ms
--- ibns2.techblue.net ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 0.344/0.394/0.422/0.030 ms
training@jump-desktop:~ $ ping -c 4 ibns2.techblue.net
PING ibns2.techblue.net (198.51.100.137) 56(84) bytes of data.
--- ibns2.techblue.net ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3061ms
The name ibns2.techblue.net is likely NAT’ed, its public IP address is 198.51.100.137, and the private IP address is 10.200.0.105. Somehow, both addresses were added to the Grid for this name. Since the public IP address is not reachable from the internal space, this setup is causing issues for internal users. Symptoms may include slow or intermittent name resolution. The fix is to remove the entry for 198.51.100.137 in the Grid, so that the name ibns2.techblue.net will consistently resolve to 10.200.0.105.
Task 2: Researching configuration on the Grid
Now that we have verified the behavior, the next step is to find what part of the Grid configuration is responsible for this unreachable IP address, 198.51.100.137 and remove it.
Searching by IP Address
Login to the GM web interface, click Search (upper right corner) to launch the global search dialog.
Perform a basic search for IP Address 198.51.100.137, click Search.
If you attempted to delete this record, you’ll see an error message informing you that it cannot be modified.
Now we have a problem: This record cannot be deleted. We do have a clue of how this entry as added, the comment of the record says: Auto-created by Add Zone. This means that while adding a DNS zone, this record was added as part of the zone creation. Let’s perform a search on the name and see if we can find more.
Searching by Name
Login to the GM web interface, click Search (upper right corner) to launch the global search dialog.
Perform a basic search for DNS Name ibns2.techblue.net, click Search.
Look under the Type column. We can see there are 3 NS records that match this name. Looking closer, we can see that the NS records are added as a result of adding the zones techblue.net and eng.techblue.net. We should look at those zone’s settings next.
Navigate to Data Management → DNS → Zones, and edit the zone (techblue.net or eng.techblue.net)
Click the Name Servers tab on the left. For the zone techblue.net, it does not show any entries that contain the offending IP address 198.51.100.137; for the zone eng.techblue.net, we can see that it contains an entry for an External Secondary that has the IP address 198.51.100.137.
Delete the External Secondary entry with 198.51.100.137 and save.
Restart services when prompted.
Verifying resolution
Re-run the same dig command from Task 1 to verify the name only resolves to a single IP address.
Figure 3522-4
$ dig @10.100.0.105 ibns2.techblue.net.
; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> @10.100.0.105 ibns2.techblue.net.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12823
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: 6305ccfb76b59202a51bc8cd64c7ee5208ad0354b52a9a22 (good)
;; QUESTION SECTION:
;ibns2.techblue.net. IN A
;; ANSWER SECTION:
ibns2.techblue.net. 28800 IN A 10.200.0.105
;; Query time: 3 msec
;; SERVER: 10.100.0.105#53(10.100.0.105) (UDP)
;; WHEN: Mon Jul 31 17:24:34 UTC 2023
;; MSG SIZE rcvd: 91