Identifying lame delegations in NIOS (3521)

Scenario

Users report they are unable to resolve the Internet domain name lame.training.infoblox.com. A different team manages the external-facing domain training.infoblox.com, and they have come to you seeking assistance. Please troubleshoot for root cause and provide a fix.

You have no administrative access to the other team’s DNS server. You need to rely on lookup tools such as dig to isolate and analyze the root cause(s). You can query against your own DNS servers, however, and compare the results.

Estimate Completion Time

25 to 30 minutes

Credentials

Description	Username	Password	URL or IP
Grid Manager UI	admin	infoblox	https://10.100.0.100/

Requirements

Administrative access to the Grid.

Learning Content

DNS Troubleshooting Methodology

Lab Initiation

Access jump-desktop

Once the lab is deployed, you can access the virtual machines required to complete this lab activity. To initiate the lab, click on the jump-desktop tile and login to the Linux UI:

Username: training

Password: infoblox

Initiate lab

To initiate the lab, double-click the Launch Lab icon on the Desktop.

Choose the lab number from the list and click OK.

After clicking OK, you will see a pop-up message with a brief description of the lab task. If the description looks correct, click Yes to continue lab initiation.

Lab initiation will take a couple of minutes to finish.

Once complete, you will see another pop-up message with the login credentials and the URL for the Grid Manager’s User Interface. Note that the credentials may differ from those from prior labs.

Tasks

Task 1: Querying an open recursive resolver

Confirm the symptom by querying an open recursive resolver for the domain name lame.training.infoblox.com. You may query public resolvers on the Internet such as Google (8.8.8.8).

Task 2: Querying an internal recursive resolver

To get more information on why this error is occurring, send the same query to the internal recursive resolvers that you control, ibns1 (10.100.0.105) or ibns2 (10.200.0.105). The dig results will be the same, but this produces log messages that you can study.

Task 3: Investigating IP address and authoritative name server

Use the information you learned from the first 2 tasks to investigate further for the root cause(s).

You will likely discover additional IP address(es) that may be causing issues. Perform more DNS lookups to investigate if the address(es) belongs to a name server (NS record).

Solutions

Task 1 Solution: Querying an open recursive resolver

Confirm the symptom by querying an open recursive resolver for the domain name lame.training.infoblox.com. You may query public resolvers on the Internet such as Google (8.8.8.8).

Figure 3521-1: External Query

$ dig @8.8.8.8 lame.training.infoblox.com.

; <<>> DiG 9.18.39-0ubuntu0.22.04.4-Ubuntu <<>> @8.8.8.8 lame.training.infoblox.com.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 11674
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; EDE: 23 (Network Error): ([54.254.111.40] rcode=REFUSED for lame.training.infoblox.com/a)
; EDE: 23 (Network Error): ([16.52.200.183] rcode=REFUSED for lame.training.infoblox.com/a)
; EDE: 22 (No Reachable Authority): (At delegation lame.training.infoblox.com for lame.training.infoblox.com/a)
;; QUESTION SECTION:
;lame.training.infoblox.com.    IN      A

;; Query time: 1358 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Wed Jun 03 18:45:32 IST 2026
;; MSG SIZE  rcvd: 270

Detailed Analysis of Figure 3521-1

Line 7: We see the SERVFAIL return code. This means the server 8.8.8.8 encountered an error. Unfortunately, we cannot see the logs on the server 8.8.8.8 to see what that error is.
Line 8: We see the ra flag. This means the server 8.8.8.8 asked other DNS servers to resolve this name, and the resolution failed. Again, we don't know why it failed, there is not enough information here.
Line 12-13: This line provides some hint as to why the lookup failed. When 8.8.8.8 attempted to contact the IP addresses 54.254.111.40 and 16.52.200.183, it encountered a REFUSED error. The query to resolve the A record for lame.training.infoblox.com was refused.

Depending on your version of dig and the version of the DNS servers, you may or may not receive the Extended DNS Error (EDE) hint in lines 12-13. If you received this hint, you do not need to perform the next task. Since EDE is not yet universally deployed to every domain, we will describe the steps needed to find the entire error message in logs in Task 2.

Task 2 Solution: Querying an internal recursive resolver

To get more information on why this error is occurring, send the same query to the internal recursive resolvers that you control, ibns1 (10.100.0.105) or ibns2 (10.200.0.105). The dig results is the same, thus we will not show it here. But by executing the query, it produces log messages that you can study on the DNS servers you control.

On the jump-desktop, open a Terminal and execute this command:

dig @10.200.0.105 lame.training.infoblox.com.

Then, check the logs on ibns2 (10.200.0.105).

Login to the GM, navigate to Administration → Logs → Syslog.
Select ibns2.techblue.net from the drop-down menu.
You can search for the domain name and find the log entry.

So, the full error messages are:

REFUSED unexpected RCODE resolving 'lame.training.infoblox.com/A/IN': 54.254.111.40#53
REFUSED unexpected RCODE resolving 'lame.training.infoblox.com/A/IN': 16.52.200.183#53

This means the DNS server ibns2 (10.200.0.105) was refused connection when attempting to query the upstream servers 54.254.111.40 and 184.170.237.34 on port 53 to resolve the A record of lame.training.infoblox.com. Although you might be able to make some educated guesses, there is no clear giveaway here. It is best to start from the top and do a delegation trace for this domain name to see how these IP addresses fit into the picture. We will do that in the next task, Task 3.

Task 3: Investigating IP address and authoritative name server

We have learned two new IP addresses. Let’s perform a trace to see if we can find out what these addresses are. We can do this by querying any recursive resolver to perform a trace on the domain name.

Perform a trace:

Figure 3521-2: Delegation Trace

$ dig -4 @8.8.8.8 lame.training.infoblox.com. +trace +nodnssec +nocmd
.                       87203   IN      NS      b.root-servers.net.
.                       87203   IN      NS      a.root-servers.net.
.                       87203   IN      NS      e.root-servers.net.
.                       87203   IN      NS      j.root-servers.net.
.                       87203   IN      NS      i.root-servers.net.
.                       87203   IN      NS      k.root-servers.net.
.                       87203   IN      NS      f.root-servers.net.
.                       87203   IN      NS      c.root-servers.net.
.                       87203   IN      NS      l.root-servers.net.
.                       87203   IN      NS      g.root-servers.net.
.                       87203   IN      NS      m.root-servers.net.
.                       87203   IN      NS      d.root-servers.net.
.                       87203   IN      NS      h.root-servers.net.
;; Received 239 bytes from 8.8.8.8#53(8.8.8.8) in 67 ms

com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
;; Received 851 bytes from 198.97.190.53#53(h.root-servers.net) in 137 ms

infoblox.com.           172800  IN      NS      ns5.infoblox.com.
infoblox.com.           172800  IN      NS      ns6.infoblox.com.
infoblox.com.           172800  IN      NS      ns7.infoblox.com.
infoblox.com.           172800  IN      NS      ns1.infoblox.com.
;; Received 247 bytes from 192.26.92.30#53(c.gtld-servers.net) in 174 ms

lame.training.infoblox.com. 3600 IN     NS      ginan.techblue.io.
lame.training.infoblox.com. 3600 IN     NS      imai.techblue.io.
;; Received 133 bytes from 18.209.84.34#53(ns7.infoblox.com) in 248 ms

;; Received 83 bytes from 54.254.111.40#53(ginan.techblue.io) in 53 ms

Detailed Analysis of Figure 3521-2

Line 1: We use options +nodnssec and +nocmd to minimize unneeded output from dig. We turn on +trace to perform a logical or delegation trace, to find all name servers involved.
Lines 2 to 15: This lists all the name servers that are authoritative for the root zone. Root is represented by a single dot (.) in the beginning of the line.
Lines 17 to 30: This lists all the name servers that are authoritative for the com zone. We can see the domain name com. in the beginning of the line.
Lines 33 to 38: This lists all the name servers that are authoritative for the infoblox.com zone. There are 5 name servers: ns2, ns3, ns4, ns7, and ns8.
Line 39: This lists the subdomain lame.training.infoblox.com is delegated to ginan.techblue.io and imai.techblue.io
Line 42: In this example, it lists the IP address of ginan.techblue.io: 54.254.111.40.

Verifying the second IP:
As we now know one of the IP addresses belongs to the delegated server for lame.training.infoblox.com, use the dig command to verify if the second IP address belongs to the other delegated server.
- Use the dig command to find the IP address of imai.techblue.io:
  $ dig @8.8.8.8 imai.techblue.io. +short 16.52.200.183
- Use the dig command to find the IP address of ginan.techblue.io:
  $ dig @8.8.8.8 ginan.techblue.io. +short 54.254.111.40
Querying delegated servers directly:

Now that we know the two IP addresses belong to the delegated servers, we can query them directly. According to the delegation results, name servers imai and ginan should respond with authoritative data for lame.training.infoblox.com.

Figure 3521-3: Querying Delegated Server

$ dig @16.52.200.183 lame.training.infoblox.com. +norecurse

; <<>> DiG 9.18.39-0ubuntu0.22.04.4-Ubuntu <<>> @16.52.200.183 lame.training.infoblox.com. +norecurse
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 20476
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: e403d1da9b397bb0010000006a202a7567251adb52fbc12f (good)
;; QUESTION SECTION:
;lame.training.infoblox.com.    IN      A

;; Query time: 269 msec
;; SERVER: 16.52.200.183#53(16.52.200.183) (UDP)
;; WHEN: Wed Jun 03 18:51:57 IST 2026
;; MSG SIZE  rcvd: 83

dig @54.254.111.40 lame.training.infoblox.com. +norecurse

; <<>> DiG 9.18.39-0ubuntu0.22.04.4-Ubuntu <<>> @54.254.111.40 lame.training.infoblox.com. +norecurse
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 2832
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: 18b2904c11b48951010000006a202a8f363459339093d19a (good)
;; QUESTION SECTION:
;lame.training.infoblox.com.    IN      A

;; Query time: 59 msec
;; SERVER: 54.254.111.40#53(54.254.111.40) (UDP)
;; WHEN: Wed Jun 03 18:52:23 IST 2026
;; MSG SIZE  rcvd: 83

Detailed Analysis of Figure 3521-3

Line 1: Since we are querying an authoritative server, we do not need to enable recursion.
Line 7: Our query was refused by the servers 16.52.200.183 and 54.254.111.40

Why turn off recursion when querying authoritative servers? First of all, most authoritative servers do not accept recursive queries. It is a basic DNS best practice. Secondly, if the target authoritative server cannot resolve the name, we do not want it to ask any other servers to resolve it with a recursive query. For our purpose here, we would rather see the query fail so we can isolate the issue.

Root Cause and Conclusion

The lame.training.infoblox.com domain has been delegated to imai.techblue.io and ginan.techblue.io, but neither server is currently accepting DNS queries.
This is a variation of lame delegation in DNS, where the parent delegates a subdomain to a child, but the child is not set up to handle the subdomain.
The solution is to contact the administrators of imai.techblue.io and ginan.techblue.io and ask them to reconfigure the server to accept DNS queries. While they are at it, check that they actually have the authoritative data for the zone lame.training.infoblox.com.