Skip to main content
Skip table of contents

3521 - Identifying lame delegations in NIOS

Scenario

Users report they are unable to resolve the Internet domain name lame.training.infoblox.com. A different team manages the external-facing domain training.infoblox.com, and they have come to you seeking assistance. Please troubleshoot for root cause and provide a fix.

You have no administrative access to the other team’s DNS server. You need to rely on lookup tools such as dig to isolate and analyze the root cause(s). You can query against your own DNS servers, however, and compare the results.

Estimate Completion Time

  • 20 to 25 minutes

Credentials

Description

Username

Password

URL or IP

Grid Manager UI

admin

infoblox

https://10.100.0.100/

Requirements

  • Administrative access to the Grid.

Course References

  • 3011: DNS Troubleshooting Methodology

Lab Initiation

Access jump-desktop

Once the lab is deployed, you can access the virtual machines required to complete this lab activity. To initiate the lab, click on the jump-desktop tile and login to the Linux UI:

Username: training

Password: infoblox

Initiate lab

To initiate the lab, double-click the Launch Lab icon on the Desktop.

Launch Lab

Launch Lab

Choose the lab number from the list and click OK.

After clicking OK, you will see a pop-up message with a brief description of the lab task. If the description looks correct, click Yes to continue lab initiation.

Lab initiation will take a couple of minutes to finish.

Once complete, you will see another pop-up message with the login credentials and the URL for the Grid Manager’s User Interface. Note that the credentials may differ from those from prior labs.

Screenshot 2024-05-06 at 3.16.57 PM.png

Tasks

Task 1: Querying an open recursive resolver

Confirm the symptom by querying an open recursive resolver for the domain name lame.training.infoblox.com. You may query public resolvers on the Internet such as Google (8.8.8.8).

Task 2: Querying an internal recursive resolver

To get more information on why this error is occurring, send the same query to the internal recursive resolvers that you control, ibns1 (10.100.0.105) or ibns2 (10.200.0.105). The dig results will be the same, but this produces log messages that you can study.

Task 3: Investigating IP address and authoritative name server

Use the information you learned from the first 2 tasks to investigate further for the root cause(s).

You will likely discover additional IP address(es) that may be causing issues. Perform more DNS lookups to investigate if the address(es) belongs to a name server (NS record).


Solutions

Task 1 Solution: Querying an open recursive resolver

Confirm the symptom by querying an open recursive resolver for the domain name lame.training.infoblox.com. You may query public resolvers on the Internet such as Google (8.8.8.8).

Figure 3521-1: External Query
CODE
$ dig @8.8.8.8 lame.training.infoblox.com.

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> @8.8.8.8 lame.training.infoblox.com.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 62045
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; EDE: 23 (Network Error): ([68.183.207.1] rcode=REFUSED for lame.training.infoblox.com/a)
; EDE: 22 (No Reachable Authority): (At delegation lame.training.infoblox.com for lame.training.infoblox.com/a)
;; QUESTION SECTION:
;lame.training.infoblox.com.	IN	A

;; Query time: 180 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Sat Jul 29 05:44:23 UTC 2023
;; MSG SIZE  rcvd: 201

Detailed Analysis of Figure 3521-1

  • Line 7: We see the SERVFAIL return code. This means the server 8.8.8.8 encountered an error. Unfortunately, we cannot see the logs on the server 8.8.8.8 to see what that error is.

  • Line 8: We see the ra flag. This means the server 8.8.8.8 asked other DNS servers to resolve this name, and the resolution failed. Again, we don't know why it failed, there is not enough information here.

  • Line 12: This line provides some hint as to why the lookup failed. When 8.8.8.8 attempted to contact the IP address 68.183.207.1, it encountered a REFUSED error. The query was that was refused was to resolver the A record for the name lame.training.infoblox.com.

Depending on your version of dig and the version of the DNS servers, you may or may not receive the Extended DNS Error (EDE) hint in line 12. If you received this hint, you do not need to perform the next task. Since EDE is is not yet universally deployed to every domain, we will describe the steps needed to find the full error message in logs in Task 2.

Task 2 Solution: Querying an internal recursive resolver

To get more information on why this error is occurring, send the same query to the internal recursive resolvers that you control, ibns1 (10.100.0.105) or ibns2 (10.200.0.105). The dig results is the same, thus we will not show it here. But by executing the query, it produces log messages that you can study on the DNS servers you control.

On the jump-desktop, open a Terminal and execute this command:

dig @10.200.0.105 lame.training.infoblox.com.

Then, check the logs on ibns2 (10.200.0.105).

  1. Login to the GM, navigate to Administration → Logs → Syslog.

  2. Select ibns2.techblue.net from the drop-down menu.

  3. You can search for the domain name and find the log entry.

So, the full error message is:

CODE
REFUSED unexpected RCODE resolving 'lame.training.infoblox.com/A/IN': 68.183.207.1#53

This means the DNS server ibns2 (10.200.0.105) was refused connection, when attempting to query the upstream server 68.183.207.1 on port 53, to resolve the A record of lame.training.infoblox.com. Although you might be able to make some educated guesses, there is no clear giveaway here. It is best to start from the top and do a delegation trace for this domain name to see how this IP address 68.183.207.1 fits into the picture. We will do that in the next task, Task 3.

Task 3: Investigating IP address and authoritative name server

We have learned a new IP address 68.183.207.1. Let’s perform a trace to see if we can find out what this address is. We can do this by querying any recursive resolver to perform a trace on the domain name.

Figure 3521-2: Delegation Trace
CODE
$ dig @8.8.8.8 lame.training.infoblox.com. +trace +nodnssec +nocmd
.			82843	IN	NS	a.root-servers.net.
.			82843	IN	NS	b.root-servers.net.
.			82843	IN	NS	c.root-servers.net.
.			82843	IN	NS	d.root-servers.net.
.			82843	IN	NS	e.root-servers.net.
.			82843	IN	NS	f.root-servers.net.
.			82843	IN	NS	g.root-servers.net.
.			82843	IN	NS	h.root-servers.net.
.			82843	IN	NS	i.root-servers.net.
.			82843	IN	NS	j.root-servers.net.
.			82843	IN	NS	k.root-servers.net.
.			82843	IN	NS	l.root-servers.net.
.			82843	IN	NS	m.root-servers.net.
;; Received 239 bytes from 8.8.8.8#53(8.8.8.8) in 4 ms

com.			172800	IN	NS	e.gtld-servers.net.
com.			172800	IN	NS	a.gtld-servers.net.
com.			172800	IN	NS	i.gtld-servers.net.
com.			172800	IN	NS	d.gtld-servers.net.
com.			172800	IN	NS	j.gtld-servers.net.
com.			172800	IN	NS	f.gtld-servers.net.
com.			172800	IN	NS	b.gtld-servers.net.
com.			172800	IN	NS	k.gtld-servers.net.
com.			172800	IN	NS	c.gtld-servers.net.
com.			172800	IN	NS	g.gtld-servers.net.
com.			172800	IN	NS	h.gtld-servers.net.
com.			172800	IN	NS	l.gtld-servers.net.
com.			172800	IN	NS	m.gtld-servers.net.
;; Received 851 bytes from 192.5.5.241#53(f.root-servers.net) in 0 ms

infoblox.com.		172800	IN	NS	ns2.infoblox.com.
infoblox.com.		172800	IN	NS	ns3.infoblox.com.
infoblox.com.		172800	IN	NS	ns4.infoblox.com.
infoblox.com.		172800	IN	NS	ns7.infoblox.com.
infoblox.com.		172800	IN	NS	ns8.infoblox.com.
;; Received 281 bytes from 192.41.162.30#53(l.gtld-servers.net) in 40 ms

lame.training.infoblox.com. 3600 IN	NS	semita.training.infoblox.com.
;; Received 120 bytes from 104.40.90.56#53(ns8.infoblox.com) in 72 ms

;; Received 83 bytes from 68.183.207.1#53(semita.training.infoblox.com) in 4 ms

Detailed Analysis of Figure 3521-2

  • Line 1: We use options +nodnssec and +nocmd to minimize unneeded output from dig. We turn on +trace to perform a logical or delegation trace, to find all name servers involved.

  • Lines 2 to 15: This lists all the name servers that are authoritative for the root zone. Root is represented by a single dot (.) in the beginning of the line.

  • Lines 17 to 30: This lists all the name servers that are authoritative for the com zone. We can see the domain name com. in the beginning of the line.

  • Lines 33 to 38: This lists all the name servers that are authoritative for the infoblox.com zone. There are 5 name servers: ns2, ns3, ns4, ns7, and ns8.

  • Line 39: This lists the subdomain lame.training.infoblox.com is delegated to the server semita.training.infoblox.com.

  • Line 42: This lists the IP address of semita.training.infoblox.com: 68.183.207.1.

Querying delegated server directly

Now we know the IP address 68.183.207.1 belongs to the delegated server semita.training.infoblox.com, we can query it directly. According to the results of delegation, semita should respond with authoritative data for lame.training.infoblox.com.

Figure 3521-3: Querying Delegated Server
CODE
$ dig @68.183.207.1 lame.training.infoblox.com. +norecurse

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> @68.183.207.1 lame.training.infoblox.com.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 21781
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: d8df99ba179a27540100000064c4aef0bfd806f62d87cf43 (good)
;; QUESTION SECTION:
;lame.training.infoblox.com.	IN	A

;; Query time: 3 msec
;; SERVER: 68.183.207.1#53(68.183.207.1) (UDP)
;; WHEN: Sat Jul 29 06:17:20 UTC 2023
;; MSG SIZE  rcvd: 83

Detailed Analysis of Figure 3521-3

  • Line 1: Since we are querying an authoritative server, we do not need to enable recursion.

  • Line 7: Our query was refused by the server 68.183.207.1

Why turn off recursion when querying authoritative servers? First of all, most authoritative servers do not accept recursive queries. It is a basic DNS best practice. Secondly, if the target authoritative server cannot resolve the name, we do not want it to ask any other servers to resolve it with a recursive query. For our purpose here, we would rather see the query fail so we can isolate the issue.

Root Cause and Conclusion

  1. The lame.training.infoblox.com domain has been delegated to semita.training.infoblox.com (68.183.207.1), which is not accepting DNS queries.

  2. This is a variation of lame delegation in DNS, where the parent delegates a subdomain to a child, but the child is not setup to handle the subdomain.

  3. The solution is to contact the administrators of the server semita (68.183.207.1) and get them to reconfigure the server to accept DNS queries. And while they are at it, check that they actually have the authoritative data for the zone lame.training.infoblox.com.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.