dns resolution not working

Connection sharing, Firewall, Samba..etc
Forum rules
Before you post please read how to get help
Post Reply
iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

dns resolution not working

Post by iamthebull » Tue Oct 15, 2019 5:07 pm

I have a few different computers on my local network running different operating systems. Rather than maintain hosts files on all the machines I rely on the local router to resolve the hostnames to IP addresses. This works for all my local machines except for the computer running Linux Mint.

For example:
Computer 1:
OS: Debian 10
hostname: debian
ip addr: 192.168.1.70

Computer 2:
OS: Linux Mint 19.2 Tina
hostname: mint
ip addr: 192.168.1.79

Local Router:
ip addr: 192.168.1.254

I can ping mint from debian using:

Code: Select all

david@debian:~$ ping mint
PING mint.telus (192.168.1.79) 56(84) bytes of data.
64 bytes from 192.168.1.79 (192.168.1.79): icmp_seq=1 ttl=64 time=15.8 ms
and it works great. See the wireshark trace.
debian1.png
but when I try to ping debian from mint I get:

Code: Select all

david@mint:~$ ping debian
ping: debian: Name or service not known
Here is the wireshark trace: (this forum won't let me upload the pic or post image link)
https://imgur.com/P7USAgJ

It seems as though a correct response is received from the router but the request is repeated several times. And ultimately it fails. Any ideas on what might be the issue here?

User avatar
kato181
Level 4
Level 4
Posts: 422
Joined: Fri Mar 24, 2017 12:33 am
Location: Frederickton NSW

Re: dns resolution not working

Post by kato181 » Thu Oct 17, 2019 7:09 am

Have you enabled share on both machines? Can you ping google from the mint machine? You could also try to ping the mint machine itself and set what you get back.
I would also check the router to see whether the mint's ip outgoing address has been blocked. Check the firewall setting also, it maybe blocking mint outgoing, but I would hardly think that would happen..

iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

Re: dns resolution not working

Post by iamthebull » Thu Oct 17, 2019 3:49 pm

The mint machine is not being blocked. It is getting the DNS reply from the router. So now it has the IP address of debian. But it still complains that name is not known. When in fact it is known or should be from the reply. This, to me, indicates maybe an issue with the DNS resolution service itself rather than a traffic blocking issue.

rene
Level 12
Level 12
Posts: 4307
Joined: Sun Mar 27, 2016 6:58 pm

Re: dns resolution not working

Post by rene » Thu Oct 17, 2019 4:49 pm

Not certain but I am seeing something weird on the wire there. I.e., you query

Code: Select all

Standard query 0x1234 A debian.telus OPT
and get a response

Code: Select all

Standard query response 0x1234 A debian.telus OPT A 192.168.1.70
You're unfortunately not showing queries for non-local names but what I am seeing here on the wire for successfully resolved names is

Code: Select all

Standard query response 0x1234 A debian.telus A 192.168.1.70 OPT
That is, with the OPT and A reversed.

Mint uses systemd-resolved and it seems plausible that your router is "special" or even plain buggy and that systemd-resolved refuses the reply. Clearly we have to compensate for things working for you from the other systems and the easiest test would be to disable systemd-resolved for a bit:

Code: Select all

$ sudo systemctl disable systemd-resolved
$ sudo systemctl stop systemd-resolved
$ sudo rm /etc/resolv.conf
Then edit /etc/NetworkManager/NetworkManager.conf to insert "dns=default" in the [main] section and

Code: Select all

sudo systemctl restart NetworkManager.service
or reboot. If things work for you then we can at least be certain it's (the combination of) systemd-resolved (and your router).

What I myself am at that point not certain of is whether or not this would need to be called a bug in either your router or systemd-resolved (nor, by the way, if when latter it's not in the meantime already fixed; Mint uses a by now fairly old base). If its former you may have some issues convincing systemd-resolved to adapt --- although clearly nothing else seemingly tripping over it would be an argument.

Any case... first just try.

iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

Re: dns resolution not working

Post by iamthebull » Fri Oct 18, 2019 7:08 pm

I disabled systemd-resolved as instructed and now the name resolution seems to be working.

Code: Select all

$ ping debian
PING debian.telus (192.168.1.100) 56(84) bytes of data.
64 bytes from 192.168.1.100 (192.168.1.100): icmp_seq=1 ttl=64 time=15.7 ms
A screenshot of the wireshark trace can be viewed here (wireshark file included in the attached zip - withoutresolved.pcapng):
https://imgur.com/Y1FoGig

I have also included the wireshark file associated with the linked image in the original post (mint2.pcapng - with systemd-resolved enabled).

Here is a screenshot of the wireshark trace for a non-local name (linuxmint.com) after systemd-resolved was disabled. The wireshark file is also included in the attached zip (linuxmintping.pcapng). I meant to do this before disabling systemd-resolved but I forgot. If it would be helpful I can re-enable systemd-resolved and do a capture using a non-local name. Let me know.
https://imgur.com/v2WYuFL

With systemd-resolved removed what is handling dns resolution now?

What can I do now to determine whether my router is causing the issue or systemd-resolved?
Attachments
wireshark.zip
(7.81 KiB) Downloaded 8 times

rene
Level 12
Level 12
Posts: 4307
Joined: Sun Mar 27, 2016 6:58 pm

Re: dns resolution not working

Post by rene » Sat Oct 19, 2019 6:20 pm

iamthebull wrote:
Fri Oct 18, 2019 7:08 pm
What can I do now to determine whether my router is causing the issue or systemd-resolved?
I'm fairly confident it is the combination; in fact fairly confident that your router is running an old version of dnsmasq since they normally do and indeed it has issues with EDNS, i.e., OPT. I am unfortunately in the midst of rearranging a system and can not easily test things but what I expect the case to be is that specifically the options edns0 in the standard /run/systemd/resolve/stub-resolv.conf is what is causing your router to trip.

The option was added per https://github.com/systemd/systemd/comm ... 1c1e566fe8 and for reasons mentioned there. If I'm not mistaken --- I might be; can as said not test things at the moment --- the dnsmasq commit http://thekelleys.org.uk/gitweb/?p=dnsm ... e66128af78 fixes it up dnsmasq-sides. Given that you are unlikely to be able to update dnsmasq on your router that's not too useful for you though.

It's easy to test at least if indeed the added option is the issue. I.e., re-enable systemd-resolved:

Code: Select all

$ sudo systemctl enable systemd-resolved
$ sudo systemctl start systemd-resolved
remove / comment out dns=default in the [main] section of /etc/NetworkManager/NetworkManager.conf and

Code: Select all

$ sudo systemctl restart NetworkManager
This time however, rather than linking /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf, you provide that same content lacking only the options edns0 line:

Code: Select all

$ sudo rm /etc/resolv.conf
$ echo nameserver 127.0.0.53 | sudo tee /etc/resolv.conf
If I'm not mistaken there should be no need to reboot to have the then current situation active but if I'm overlooking something feel free to. It is expected that things will now still work for you, i.e., with the only difference with the original situation the removal of options edns0. If I'm not mistaken you could in fact alternatively also restore the original link:

Code: Select all

$ sudo rm /etc/resolv.conf
$ sudo ln -s ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf 
and set a global environment variable RES_OPTIONS= (i.e., empty) in for example /etc/profile (man resolv.conf) and reboot but that's untested.

While googling I happened upon a few indications that the issue or an issue very much like it is NOT present in upstream systemd-resolved; is specific to Ubuntu's systemd 2.37 which we are using. Does your Debian 10 system use systemd >= 2.40? If so, you could supposedly conversely enable systemd-resolved on it as per the above --- without the NetworkManager edit/restart if Debian or your setup doesn't use it, with the ln -s and without setting RES_OPTIONS= --- and see if that system also has an issue at that point.

As to the more general question: conceptually programs each do their own DNS resolving and something like e.g. dig actually does. Normally though they link to the system resolver, part of the system libc, which contacts the DNS server(s) configured in /etc/resolv.conf; systemd-resolved just inserts itself as a sort of caching proxy/relay between said system resolver and your upstream server(s), which it gets from its own configuration files or in the case of NetworkManager from NetworkManager / dhclient through D-Bus.

On Ubuntu, dnsmasq used to do/be similar and systemd-resolved is, surprisingly, as such not fully evil. What it clearly does not do is make something such as that options edns0 optional/configurable but if this actual issue is indeed Ubuntu or Ubuntu 18.04 specific I'd say we not get to complain about it too loudly.

iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

Re: dns resolution not working

Post by iamthebull » Tue Oct 22, 2019 12:23 am

Ok. So I did the following:

Code: Select all

$ sudo systemctl enable systemd-resolved
$ sudo systemctl start systemd-resolved
Removed dns=default from /etc/NetworkManager/NetworkManager.conf

Code: Select all

$ sudo systemctl restart NetworkManager
$ sudo rm /etc/resolv.conf
$ echo nameserver 127.0.0.53 | sudo tee /etc/resolv.conf
$ sudo reboot
but now the DNS query is being sent to 75.153.171.122 the DNS server of my internet provider and bypassing my local router 192.168.1.254. Shouldn't it use the router since it's listed first?

Code: Select all

$ systemd-resolve --status
Link 3 (wlp3s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.254
                      75.153.171.122
                      2001:568:ff09:10c::53
                      2001:568:ff09:10b::122
          DNS Domain: telus

$ nmcli dev show | grep DNS
IP4.DNS[1]:                             192.168.1.254
IP4.DNS[2]:                             75.153.171.122
IP6.DNS[1]:                             2001:568:ff09:10c::53
IP6.DNS[2]:                             2001:568:ff09:10b::122


rene
Level 12
Level 12
Posts: 4307
Joined: Sun Mar 27, 2016 6:58 pm

Re: dns resolution not working

Post by rene » Thu Oct 24, 2019 7:36 am

The query is being sent to the upstream server(s); the point here would seem to be that you shouldn't have your ISP server as an upstream server on your host in the first place, only on your router. I.e., host -> 192.168.1.254 -> 75.153.171.122.

Normally when you have everything host-side set to automatic DHCP a DNS supplying/forwarding router will indeed only advertise itself as DNS server via DHCP so it appears you have some strange configuration going on? Router should get your ISP DNS-servers via DHCP from the ISP, host should get your router as DNS-server via DHCP from the router.

iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

Re: dns resolution not working

Post by iamthebull » Thu Oct 24, 2019 3:46 pm

I found the setting in the router to fix this configuration. DNS2 was set to pass the DNS from the WAN connection. I changed it to point to the gateway.

My question was more about DNS priority though. Even though the router was setting DNS2 to an upstream DNS server shouldn't my computer use DNS1 (ie. my local router) as the DNS server instead of using the DNS2 server?

iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

Re: dns resolution not working

Post by iamthebull » Thu Oct 24, 2019 4:03 pm

Now that the DNS issue is sorted out I tested the configuration after following your instructions to re-enable systemd-resolved but remove edns0. And I got the same result as in the original post (ie. the same behaviour as shown here https://imgur.com/P7USAgJ). So it would seem that the EDNS is not the issue?

iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

Re: dns resolution not working

Post by iamthebull » Mon Oct 28, 2019 12:56 pm

Is there anything else I can do to help identify the issue?

rene
Level 12
Level 12
Posts: 4307
Joined: Sun Mar 27, 2016 6:58 pm

Re: dns resolution not working

Post by rene » Mon Oct 28, 2019 9:51 pm

iamthebull wrote:
Thu Oct 24, 2019 4:03 pm
And I got the same result as in the original post (ie. the same behaviour as shown here https://imgur.com/P7USAgJ). So it would seem that the EDNS is not the issue?
In some local testing I did I no longer saw OPT records being sent out with "options edns0" removed from /etc/resolv.conf, and it would also seem a bit of coincidence that the A<->OPT reversal I noted originally would NOT be involved, so, no, I do not believe that EDNS will not be the core issue for you.

That said, if you're sure you're testing correctly there's little other I can suggest than leaving systemd-resolved disabled --- which you tested to work, so why aren't you doing that? If it's a matter of you wanting its local caching: I due to unrelated issues had some DNS problems myself and got rid of them by disabling systemd-resolved as well after which it turned out that Ubuntu/Mint is still completely prepared for switching back to dnsmasq; it is installed by default it seems.

To use,

Code: Select all

$ sudo systemctl disable systemd-resolved
$ sudo systemctl stop systemd-resolved
$ sudo rm /etc/resolv.conf
Edit /etc/NetworkManager/NetworkManager.conf to insert "dns=dnsmasq" in the [main] section, optionally create a file e.g. /etc/NetworkManager/dnsmasq.d/cache-size.conf consisting of

Code: Select all

cache-size=1000
to have a 1000-name local DNS cache (150 is the dnsmasq-default but 0 is the Ubuntu/Mint default if you don't intervene like this), and

Code: Select all

$ sudo systemctl restart NetworkManager
I have personally not yet seen any reason for going back to systemd-resolved; the dnsmasq setup is much faster for one.

iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

Re: dns resolution not working

Post by iamthebull » Tue Oct 29, 2019 3:11 pm

I certainly appreciate your help in identifying a workaround to get things working for now. My interest in pursuing this further is to identify the root cause of the problem so that if it is an issue with systemd-resolved it can be fixed for myself and possibly others. And if it is an issue with the router (ie. the OPT<->A reversal) I'm happy to let the blame lie there and leave this to rest. Is this reversal a violation of the DNS protocol? Maybe I have interpreted incorrectly but It seems that you are reluctant to conclude as such. And if you don't know who can I ask?

rene
Level 12
Level 12
Posts: 4307
Joined: Sun Mar 27, 2016 6:58 pm

Re: dns resolution not working

Post by rene » Wed Oct 30, 2019 1:47 am

iamthebull wrote:
Tue Oct 29, 2019 3:11 pm
Is this reversal a violation of the DNS protocol? Maybe I have interpreted incorrectly but It seems that you are reluctant to conclude as such.
Yes, somewhat reluctant. Specifically, as far as I know it's a violation but I when I look at your mint.pcapng wireshark is not in fact barfing on the reply; seems to decode it reasonably --- even if with the actual A reply in the additional RR section rather than the regular answer RR section. This seems at the very least peculiar; is very probably a plain bug in your router, but I wouldn't consider myself a wire-level DNS expert; would need to go digging through RFCs to state such beyond (any) doubt now that wireshark isn't flashing a big red "danger! danger!" sign.

If you care to yourself: the EDNS RFC is https://tools.ietf.org/html/rfc6891 and the main DNS RFC is https://tools.ietf.org/html/rfc1035. On a (very) quick scan I'm not myself the wiser.

In your last post you indicated that something is still sending out OPT queries even without "options edns0" in /etc/resolve.conf. That file is as said a configuration file for the glibc resolver, and if you see OPT queries sent out for for example simply ping when the options statement is removed that doesn't seem to make much sense to me. You'd preferably first be sure what's happening there.

That said... I'd give this issue being not a simple bug specific to your router an approximately 2% chance. Your router will (likely) not be one of a kind, but I would assume from its behaviour regarding EDNS that it's older, and could assume from system-resolve tripping over it that it's not widespread. Well, very widespread. The EDNS option is as was indicated somewhere above fairly new so bug reports might be welcome in any case --- after you'd made sure it's not specific to the Ubuntu 2.37 version which I as mentioned saw some indication of before.

So, 1, verify this happening with upstream systemd >= 2.40 (perhaps Debian 10 will do for that; I don't know), 2, if yes, verify it specific to EDNS by on said version making sure to use an /etc/resolv.conf without "options edns0" and a program using the glibc resolver such as plain ping, 3, optionally go dig through RFCs, 4, report, which I believe for systemd-resolved means the main systemd bugzilla at https://github.com/systemd/systemd/issues.

rene
Level 12
Level 12
Posts: 4307
Joined: Sun Mar 27, 2016 6:58 pm

Re: dns resolution not working

Post by rene » Wed Oct 30, 2019 2:03 am

Oh, by the way, while looking I noticed a .local query being sent upstream now that I'm using dnsmasq rather than systemd-resolved. That is: if anyone uses this thread to also switch to dnsmasq, it's best to additionally dump a file e.g. local.conf in /etc/NetworkManager/dnsmasq.d/ consisting of

Code: Select all

domain-needed
bogus-priv
local=/local/home/lan/
.local is the standard mDNS local suffix; .home and .lan are the (OpenWrt/DD-WRT) default local domains which routers often use. Feel free to add "domain/" for any other domain you don't want to go out on the wire.

iamthebull
Level 1
Level 1
Posts: 19
Joined: Mon Oct 14, 2019 1:50 am

Re: dns resolution not working

Post by iamthebull » Thu Oct 31, 2019 2:32 pm

Immediately after rebooting the Mint machine I ran the following commands.

Code: Select all

$ ll /etc | grep resolv.conf
-rw-r--r--   1 root root       65 Oct 31 11:45 resolv.conf

$ cat /etc/resolv.conf
# Generated by NetworkManager
search telus
nameserver 127.0.0.53

$ cat /etc/NetworkManager/NetworkManager.conf 
[main]
plugins=ifupdown,keyfile

[ifupdown]
managed=false

[device]
wifi.scan-rand-mac-address=no

$ sudo systemctl status systemd-resolved
[sudo] password for david:        
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-10-31 11:45:51 MDT; 9min ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 905 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/systemd-resolved.service
           └─905 /lib/systemd/systemd-resolved

Oct 31 11:45:50 mint systemd[1]: Starting Network Name Resolution...
Oct 31 11:45:51 mint systemd-resolved[905]: Positive Trust Anchors:
Oct 31 11:45:51 mint systemd-resolved[905]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41
Oct 31 11:45:51 mint systemd-resolved[905]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880
Oct 31 11:45:51 mint systemd-resolved[905]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.a
Oct 31 11:45:51 mint systemd-resolved[905]: Using system hostname 'mint'.
Oct 31 11:45:51 mint systemd[1]: Started Network Name Resolution.

This should hopefully confirm that I have followed your instructions correctly.
1) systemd-resolved is enabled and running
2) removed dns=default in /etc/NetworkManager/NetworkManager.conf
3) removed resolv.conf link
4) resolv.conf no longer contains options edns0

Attached is the wireshark capture when I run $ ping debian
** Note: rename file to .pcapng

I see the same type of behaviour. Is EDNS still being used? Is there a way to verify whether or not EDNS is being used even though options edns0 is no longer present? Or is there a way to explicitly turn off EDNS in addition to simply omitting it from the NetworkManager config file?
Attachments
pingdebian191031.txt
(8.28 KiB) Downloaded 4 times

rene
Level 12
Level 12
Posts: 4307
Joined: Sun Mar 27, 2016 6:58 pm

Re: dns resolution not working

Post by rene » Fri Nov 01, 2019 7:59 pm

Seems then that (your) systemd-resolved is still using EDNS; that 127.0.0.53 -> upstream and vice versa is still using EDNS even though you'd likely not see OPT traffic to 127.0.0.53 itself (for which you'd need to select "lo" or, better, "all" as your capture interface); can not currently double-check here. No, as far as I can see there's no (obvious) way to influence that -- which wouldn't surprise me in the slightest. The damn thing being as opaquely or simply unconfigurable as anything else coming out of that corner of the Linux-scape is what spurred my getting rid of it as per above, and to only reconsider if seriously I had to. Am afraid to say that also means I'll myself leave it at the above steps 1 to 4 as advise.

Post Reply

Return to “Other networking topics”