Identifying lame delegations in NIOS (3521)
Scenario
Users report they are unable to resolve the Internet domain name lame.training.infoblox.com. A different team manages the external-facing domain training.infoblox.com, and they have come to you seeking assistance. Please troubleshoot for root cause and provide a fix.
You have no administrative access to the other team’s DNS server. You need to rely on lookup tools such as dig to isolate and analyze the root cause(s). You can query against your own DNS servers, however, and compare the results.
Estimate Completion Time
25 to 30 minutes
Credentials
Description | Username | Password | URL or IP |
|---|---|---|---|
Grid Manager UI | admin | infoblox |
Requirements
Administrative access to the Grid.
Course References
3011: DNS Troubleshooting Methodology
Lab Initiation
Access jump-desktop
Once the lab is deployed, you can access the virtual machines required to complete this lab activity. To initiate the lab, click on the jump-desktop tile and login to the Linux UI:
Username: training
Password: infoblox
Initiate lab
To initiate the lab, double-click the Launch Lab icon on the Desktop.
Launch Lab
Choose the lab number from the list and click OK.
After clicking OK, you will see a pop-up message with a brief description of the lab task. If the description looks correct, click Yes to continue lab initiation.
Lab initiation will take a couple of minutes to finish.
Once complete, you will see another pop-up message with the login credentials and the URL for the Grid Manager’s User Interface. Note that the credentials may differ from those from prior labs.
Tasks
Task 1: Querying an open recursive resolver
Confirm the symptom by querying an open recursive resolver for the domain name lame.training.infoblox.com. You may query public resolvers on the Internet such as Google (8.8.8.8).
Task 2: Querying an internal recursive resolver
To get more information on why this error is occurring, send the same query to the internal recursive resolvers that you control, ibns1 (10.100.0.105) or ibns2 (10.200.0.105). The dig results will be the same, but this produces log messages that you can study.
Task 3: Investigating IP address and authoritative name server
Use the information you learned from the first 2 tasks to investigate further for the root cause(s).
You will likely discover additional IP address(es) that may be causing issues. Perform more DNS lookups to investigate if the address(es) belongs to a name server (NS record).
Solutions
Task 1 Solution: Querying an open recursive resolver
Confirm the symptom by querying an open recursive resolver for the domain name lame.training.infoblox.com. You may query public resolvers on the Internet such as Google (8.8.8.8).
Figure 3521-1: External Query
$ dig @8.8.8.8 lame.training.infoblox.com.
; <<>> DiG 9.18.24-0ubuntu0.22.04.1-Ubuntu <<>> @8.8.8.8 lame.training.infoblox.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52905
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; EDE: 23 (Network Error): ([45.120.107.60] rcode=REFUSED for lame.training.infoblox.com/a)
; EDE: 23 (Network Error): ([184.170.237.34] rcode=REFUSED for lame.training.infoblox.com/a)
; EDE: 22 (No Reachable Authority): (At delegation lame.training.infoblox.com for lame.training.infoblox.com/a)
;; QUESTION SECTION:
;lame.training.infoblox.com. IN A
;; Query time: 527 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Tue Sep 03 16:44:34 UTC 2024
;; MSG SIZE rcvd: 271
Detailed Analysis of Figure 3521-1
Line 7: We see the SERVFAIL return code. This means the server 8.8.8.8 encountered an error. Unfortunately, we cannot see the logs on the server 8.8.8.8 to see what that error is.
Line 8: We see the
raflag. This means the server 8.8.8.8 asked other DNS servers to resolve this name, and the resolution failed. Again, we don't know why it failed, there is not enough information here.Line 12-13: This line provides some hint as to why the lookup failed. When 8.8.8.8 attempted to contact the IP addresses 184.170.237.34 and 45.120.107.60, it encountered a REFUSED error. The query to resolve the A record for lame.training.infoblox.com was refused.
Depending on your version of dig and the version of the DNS servers, you may or may not receive the Extended DNS Error (EDE) hint in lines 12-13. If you received this hint, you do not need to perform the next task. Since EDE is not yet universally deployed to every domain, we will describe the steps needed to find the entire error message in logs in Task 2.
Task 2 Solution: Querying an internal recursive resolver
To get more information on why this error is occurring, send the same query to the internal recursive resolvers that you control, ibns1 (10.100.0.105) or ibns2 (10.200.0.105). The dig results is the same, thus we will not show it here. But by executing the query, it produces log messages that you can study on the DNS servers you control.
On the jump-desktop, open a Terminal and execute this command:
dig @10.200.0.105 lame.training.infoblox.com.
Then, check the logs on ibns2 (10.200.0.105).
Login to the GM, navigate to Administration → Logs → Syslog.
Select ibns2.techblue.net from the drop-down menu.
You can search for the domain name and find the log entry.
So, the full error messages are:
REFUSED unexpected RCODE resolving 'lame.training.infoblox.com/A/IN': 45.120.107.60#53
REFUSED unexpected RCODE resolving 'lame.training.infoblox.com/A/IN': 184.170.237.34#53This means the DNS server ibns2 (10.200.0.105) was refused connection when attempting to query the upstream servers 45.120.107.60 and 184.170.237.34 on port 53 to resolve the A record of lame.training.infoblox.com. Although you might be able to make some educated guesses, there is no clear giveaway here. It is best to start from the top and do a delegation trace for this domain name to see how these IP addresses fit into the picture. We will do that in the next task, Task 3.
Task 3: Investigating IP address and authoritative name server
We have learned two new IP addresses. Let’s perform a trace to see if we can find out what these addresses are. We can do this by querying any recursive resolver to perform a trace on the domain name.
Perform a trace:
Figure 3521-2: Delegation Trace
$ dig -4 @8.8.8.8 lame.training.infoblox.com. +trace +nodnssec +nocmd
. 87203 IN NS h.root-servers.net.
. 87203 IN NS d.root-servers.net.
. 87203 IN NS e.root-servers.net.
. 87203 IN NS l.root-servers.net.
. 87203 IN NS b.root-servers.net.
. 87203 IN NS c.root-servers.net.
. 87203 IN NS k.root-servers.net.
. 87203 IN NS f.root-servers.net.
. 87203 IN NS a.root-servers.net.
. 87203 IN NS g.root-servers.net.
. 87203 IN NS i.root-servers.net.
. 87203 IN NS m.root-servers.net.
. 87203 IN NS j.root-servers.net.
;; Received 239 bytes from 8.8.8.8#53(8.8.8.8) in 4 ms
com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
com. 172800 IN NS c.gtld-servers.net.
com. 172800 IN NS d.gtld-servers.net.
com. 172800 IN NS e.gtld-servers.net.
com. 172800 IN NS f.gtld-servers.net.
com. 172800 IN NS g.gtld-servers.net.
com. 172800 IN NS h.gtld-servers.net.
com. 172800 IN NS i.gtld-servers.net.
com. 172800 IN NS j.gtld-servers.net.
com. 172800 IN NS k.gtld-servers.net.
com. 172800 IN NS l.gtld-servers.net.
com. 172800 IN NS m.gtld-servers.net.
;; Received 851 bytes from 199.7.91.13#53(d.root-servers.net) in 4 ms
infoblox.com. 172800 IN NS ns5.infoblox.com.
infoblox.com. 172800 IN NS ns6.infoblox.com.
infoblox.com. 172800 IN NS ns7.infoblox.com.
infoblox.com. 172800 IN NS ns1.infoblox.com.
;; Received 247 bytes from 192.42.93.30#53(g.gtld-servers.net) in 0 ms
lame.training.infoblox.com. 3600 IN NS ginan.techblue.io.
lame.training.infoblox.com. 3600 IN NS imai.techblue.io.
;; Received 133 bytes from 3.211.162.1#53(ns7.infoblox.com) in 0 ms
;; Received 83 bytes from 45.120.107.60#53(ginan.techblue.io) in 244 msDetailed Analysis of Figure 3521-2
Line 1: We use options
+nodnssecand+nocmdto minimize unneeded output fromdig. We turn on+traceto perform a logical or delegation trace, to find all name servers involved.Lines 2 to 15: This lists all the name servers that are authoritative for the root zone. Root is represented by a single dot (.) in the beginning of the line.
Lines 17 to 30: This lists all the name servers that are authoritative for the com zone. We can see the domain name com. in the beginning of the line.
Lines 33 to 38: This lists all the name servers that are authoritative for the infoblox.com zone. There are 5 name servers: ns2, ns3, ns4, ns7, and ns8.
Line 39: This lists the subdomain lame.training.infoblox.com is delegated to ginan.techblue.io and imai.techblue.io
Line 42: In this example, it lists the IP address of ginan.techblue.io: 45.120.107.60.
Verifying the second IP:
As we now know one of the IP addresses belongs to the delegated server for lame.training.infoblox.com, use thedigcommand to verify if the second IP address belongs to the other delegated server.Use the
digcommand to find the IP address of imai.techblue.io:CODE$ dig @8.8.8.8 imai.techblue.io. +short 184.170.237.34Use the
digcommand to find the IP address of ginan.techblue.io:CODE$ dig @8.8.8.8 ginan.techblue.io. +short 45.120.107.60
Querying delegated servers directly:
Now that we know the two IP addresses belong to the delegated servers, we can query them directly. According to the delegation results, name servers imai and ginan should respond with authoritative data for lame.training.infoblox.com.
Figure 3521-3: Querying Delegated Server
$ dig @184.170.237.34 lame.training.infoblox.com. +norecurse
; <<>> DiG 9.18.24-0ubuntu0.22.04.1-Ubuntu <<>> @184.170.237.34 lame.training.infoblox.com. +norecurse
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 56481
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: 548425c61f47c6dc0100000066d8148e5c8c60f104139d32 (good)
;; QUESTION SECTION:
;lame.training.infoblox.com. IN A
;; Query time: 24 msec
;; SERVER: 184.170.237.34#53(184.170.237.34) (UDP)
;; WHEN: Wed Sep 04 08:04:30 UTC 2024
;; MSG SIZE rcvd: 83
dig @45.120.107.60 lame.training.infoblox.com. +norecurse
; <<>> DiG 9.18.24-0ubuntu0.22.04.1-Ubuntu <<>> @45.120.107.60 lame.training.infoblox.com. +norecurse
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 4280
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: 433a2d293243a9480100000066d8178b70d3c28eb098bae5 (good)
;; QUESTION SECTION:
;lame.training.infoblox.com. IN A
;; Query time: 248 msec
;; SERVER: 45.120.107.60#53(45.120.107.60) (UDP)
;; WHEN: Wed Sep 04 08:17:15 UTC 2024
;; MSG SIZE rcvd: 83
Detailed Analysis of Figure 3521-3
Line 1: Since we are querying an authoritative server, we do not need to enable recursion.
Line 7: Our query was refused by the servers
45.120.107.60and184.170.237.34
Why turn off recursion when querying authoritative servers? First of all, most authoritative servers do not accept recursive queries. It is a basic DNS best practice. Secondly, if the target authoritative server cannot resolve the name, we do not want it to ask any other servers to resolve it with a recursive query. For our purpose here, we would rather see the query fail so we can isolate the issue.
Root Cause and Conclusion
The lame.training.infoblox.com domain has been delegated to imai.techblue.io and ginan.techblue.io, but neither server is currently accepting DNS queries.
This is a variation of lame delegation in DNS, where the parent delegates a subdomain to a child, but the child is not set up to handle the subdomain.
The solution is to contact the administrators of imai.techblue.io and ginan.techblue.io and ask them to reconfigure the server to accept DNS queries. While they are at it, check that they actually have the authoritative data for the zone lame.training.infoblox.com.