3601 - Troubleshooting DNSSEC in NIOS

Scenario

You are a DNS administrator for a company, part of your responsibilities is to resolve user submitted tickets. Users have reported their inability to resolve some domains, In the ticket the user reported www.example.com to be one of the problematic domains. You were informed that some changes were made recently to DNS servers that might have caused the issue.

Please use your DNS knowledge to troubleshoot, locate the root cause(s), and make suggestions to the users.

Estimate Completion Time

• 30 to 45 minutes

Credentials

Course References

• 1204: DNSSEC Fundamentals

• 2204: Explaining DNSSEC

• 2025: Configuring Trust Anchors in NIOS
  

• 3011: DNS Troubleshooting Methodology

Lab Initiation

Access jump-desktop

Once the lab is deployed, you can access the virtual machines required to complete this lab activity. To initiate the lab, click on the jump-desktop tile and login to the Linux UI:

Username: training

Password: infoblox

Initiate lab

To initiate the lab, double-click the Launch Lab icon on the Desktop.

Choose the lab number from the list and click OK.

After clicking OK, you will see a pop-up message with a brief description of the lab task. If the description looks correct, click Yes to continue lab initiation.

Lab initiation will take a couple of minutes to finish.

Once complete, you will see another pop-up message with the login credentials and the URL for the Grid Manager’s User Interface. Note that the credentials may differ from those from prior labs.

Tasks

Task 1: Some names result in SERVFAIL

Users reported trouble resolving the domain www.example.com. when querying the server IBNS1 (10.100.0.105). Please troubleshoot and locate root cause(s). You may use the following commands on the Jump-Desktop (10.35.22.10) to verify the behavior:

^{dig @10.100.0.105 www.example.com. A}

Since your organization has DNSSEC, you may want to first rule out whether this is a DNSSEC issue, or a regular DNS issue.

Solutions

Task 1: Some names result in SERVFAIL

Root Cause

The wrong trust anchor is installed on the Grid.

Detailed Analysis

We want to tell whether or not the SERVFAIL error is caused by DNSSEC. The easiest way is to run the same lookup command twice, first without +cd, the second time with +cd, and compare their outputs.

We can see that this indeed is a DNSSEC validation problem, because when DNSSEC checking is disabled by the +cd switch, the name resolves fine.

Next, we review the log messages on the affected server IBNS1 (10.100.0.105). To do this, we login to the Grid's UI (10.100.0.100) and navigate to Administration → Logs → Syslog, from the Member drop-down list, select ibns1.techblue.net. 

As we execute this dig query, we can see the following syslog messages appear:

Reading the syslog messages carefully, we can see the following message appears more than once:

^{validating ./DNSKEY: unable to find a DNSKEY which verifies the DNSKEY RRset and also matches a trusted key for '.'}

The single dot in DNS represents the root. These messages are basically saying: there is no key that can verify the root.

In the DNSSEC configuration, we install the root key as the trust anchor. These messages indicate that we may have the root key or trust anchor misconfigured. We can access the trust anchor configuration in the Grid by navigating to Data Management → DNS, Toolbar → Grid DNS Properties → DNSSEC, and scroll down to the section TRUST ANCHORS. 

We can verify the correct trust anchor is installed, by querying another DNSSEC validating resolver:

^{dig @8.8.8.8 . DNSKEY +multi}

When carefully compared, what we have installed on the Grid is different from what we obtained from the open validating resolver. Although we have the correct algorithm (8), the key data itself is different. As a responsible DNS(SEC) administrator, we verify this information with multiple sources, including with IANA's root zone (https://www.iana.org/domains/root/files). Once we are confident of the real, correct trust anchor, we replace the incorrect one on the Grid with the correct one.

The trust anchor is a very important file for DNSSEC. We recommend you verify with multiple sources before installing or updating it. The root key or trust anchor changes infrequently. The last time the file changed was in the year 2018/2019 (over a period of several months). Those who did not update their trust anchor experienced the symptoms that DNSSEC-signed zones become bogus (SERVFAIL), until the correct, updated trust anchor is installed.

The DNSSEC root key is well know on the Internet, and is part of the root zone data. You should always verify with multiple trust sources, including the entity that house the original authoritative copy, IANA.