Odd DNS Problem

[mattlach]

Hey all,

I hope this belongs here. I was going to post it in the main networking thread, but there isn't a "new topic" button there for me?

So I recently had a very weird set of problems on my LM 18.3 Desktop, as I have documented here.

The TLDR version is that I dual boot Windows 10 and Linux Mint. (Windows only ever gets used for the odd gaming session) Windows decided to go ahead and corrupt its boot configuration. It was not repairable, so I had to format my EFI partition, and rebuild Windows boot information using Microsofts rescue console. (fun, fun, I was much happier before EFI was a thing)

This - of course - removed grub, and caused my Linux install to become unbootable. While I am great at manually repairing traditional MBR installs, the EFI thing got me confused (couldn't remember how I did it the last time) and eventually I gave up on fixing it manually, and used the boot-repair software discussed here.

It did its thing, purged a bunch of packages, reinstalled grub, and got me up and running again, except, when I booted back up in Linux again, nothing on the system would resolve DNS.

Now I use what probably is a slightly unusual network configuration. I hate the GUI network configuration tool. One of the first things I do on a fresh install is to remove it, and do all my network setup in /etc/network/interfaces.

Typically this is the only place I define my DNS servers using the dns-nameservers string. I always point it straight at my pfSense router and configure which DNS servers to use on the router.

I checked /etc/network/interfaces, and everything there looked normal. Confused I did some googling and came across mentions of /etc/resolv.conf. Apparently having a nameserver entry here is now the preferred way to set DNS? So I looked at mine, and to my surprise it had a nameserver line pointing to a weird IP address of 127.0.0.53. I have no idea how this got there. I don't even know what it may be trying to point to. I know 127.0.0.1 is the local machine, but I have never seen anything else in 127.x.x.x used before.

So, I edited the file replacing 127.0.0.53 with 10.0.1.1 for my router, and my DNS instantly started resolving again.

I am a little concerned though, as the comments in this file suggest that /etc/resolv.conf may not be persistent. I've rebooted a couple of times, and I still have DNS though, so at least it works.

I have a few questions though:

1.) Can anyone explain how the hell this happened? I'm at a loss. I'd rather avoid it happening again, if I can. What on earth is 127.0.0.53, and how did it get there?

2.) Is setting dns-nameservers in /etc/network/interfaces deprecated these days? That's the only way I've ever used to set DNS.

3.) If /etc/resolv.conf is modified by systemd like the comments in the file say, what is the proper way to set a static DNS?


I find it really frustrating how all of these things are changing lately. Systemd, GUI configuration tools, gconf, EFI, etc... These are IMHO all huge leaps backwards. Everything was perfect back on my headless Ubuntu 14.04 LTS servers when there was a text file for everything, you would configure it, and it would stay that way, not changing based on some mysterious system automation.

I feel like Linux is slowly turning into Windows. I hated the registry on Windows and the convoluted way Windows handles services. Why can't everything just be simple text based config files anymore? Ugh.

[greerd]

It looks to me like you're using systemd-resolved which I didn't think Mint 18.3 used, although Mint 19 does.

Form man systemd-resolved

...
/ETC/RESOLV.CONF
Four modes of handling /etc/resolv.conf (see resolv.conf(5)) are supported:

· systemd-resolved maintains the /run/systemd/resolve/stub-resolv.conf file for compatibility with traditional Linux
programs. This file may be symlinked from /etc/resolv.conf. This file lists the 127.0.0.53 DNS stub (see above) as the
only DNS server. It also contains a list of search domains that are in use by systemd-resolved. The list of search
domains is always kept up-to-date. Note that /run/systemd/resolve/stub-resolv.conf should not be used directly by
applications, but only through a symlink from /etc/resolv.conf. This file may be symlinked from /etc/resolv.conf in order
to connect all local clients that bypass local DNS APIs to systemd-resolved with correct search domains settings. This
mode of operation is recommended.

· A static file /usr/lib/systemd/resolv.conf is provided that lists the 127.0.0.53 DNS stub (see above) as only DNS server.
This file may be symlinked from /etc/resolv.conf in order to connect all local clients that bypass local DNS APIs to
systemd-resolved. This file does not contain any search domains.

· systemd-resolved maintains the /run/systemd/resolve/resolv.conf file for compatibility with traditional Linux programs.
This file may be symlinked from /etc/resolv.conf and is always kept up-to-date, containing information about all known
DNS servers. Note the file format's limitations: it does not know a concept of per-interface DNS servers and hence only
contains system-wide DNS server definitions. Note that /run/systemd/resolve/resolv.conf should not be used directly by
applications, but only through a symlink from /etc/resolv.conf. If this mode of operation is used local clients that
bypass any local DNS API will also bypass systemd-resolved and will talk directly to the known DNS servers.

· Alternatively, /etc/resolv.conf may be managed by other packages, in which case systemd-resolved will read it for DNS
configuration data. In this mode of operation systemd-resolved is consumer rather than provider of this configuration
file.

Note that the selected mode of operation for this file is detected fully automatically, depending on whether /etc/resolv.conf
is a symlink to /run/systemd/resolve/resolv.conf or lists 127.0.0.53 as DNS server.
...

What I do is set the dns address in Network Manager which will end up in /etc/resolv.conf, at least it does for me.

[rene]

Yes, 127.0.0.53 is systemd-resolved, a local, caching DNS "server" much like in previous versions of Ubuntu/Mint "dnsmasq" was in a default setup (then, on address 127.0.1.1). This is not generally a problem; certainly if using NetworkManager the configured or via DHCP retrieved upstream server should become an upstream server for systemd-resolved.

Yes, /etc/network/interfaces is basically obsolete: it's a configuration file for the ifupdown network management system, and ifupdown has been replaced by either NetworkManager or, starting with Mint 19, systemd-resolved. Google in fact seems to indicate that Ubuntu 18.04 does not offer ifupdown anymore by default -- but although I normally use NetworkManager, I just checked and it seems to on Mint 19 still work the same as ever.

It would've been helpful if you had posted your /etc/network/interfaces. A working one here is:

Code: Select all

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto enp6s0
#iface enp6s0 inet dhcp
iface enp6s0 inet static
    address 192.168.1.8
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameserver 192.168.1.1

The dns-nameserver bit is configuration for the resolvconf program and makes me end up with:

Code: Select all

rene@t5500:~$ ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 29 aug 28 01:12 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
rene@t5500:~$ cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 192.168.1.1
nameserver 127.0.0.53

If your /etc/resolv.conf is a link to under /run/systemd, it seems the first thing you need to check is whether or not your have the resolvconf package installed at all (I note that when I use the above commented out dhcp configuration in /etc/network/interfaces instead, the 192.168.1.1 as retrieved through DHCP from my router is configured as a systemd-resolved upstream same as when using NetworkManager).

If you do have resolvconf installed; the only thing I can think of that is halfway grub related is you previously having set the "net.ifnames=0" kernel parameter and not now; having old eth0 and alike interface names in /etc/network/interfaces. NetworkManager ignores (by default) any interfaces mentioned in /etc/network/interfaces but would not ignore say enp6s0 if you are now using the new names. If your router then provides address and gateway but not DNS through DHCP -- I actually once had such a router -- the symptom would be explained.

And then again it might be as simple as not / no longer having resolvconf installed...

[EDIT] Too late here to wait for a reply so let me immediately add: consider simply using NetworkManager. It's easily statically configured and while I would myself also count as being of an older configuration-persuasion NetworkManager isn't in fact all bad; it's particularly fairly good at staying out the way once setup. What is more than not bad moreover is simply configuring your systems to use DHCP while configuring your router to always hand out the same IP to the same MAC: makes for zero configuration generally.

[mattlach]

Rene,

Thank you for your very detailed description! I appreciate it.

Yes, 127.0.0.53 is systemd-resolved, a local, caching DNS "server" much like in previous versions of Ubuntu/Mint "dnsmasq" was in a default setup (then, on address 127.0.1.1). This is not generally a problem; certainly if using NetworkManager the configured or via DHCP retrieved upstream server should become an upstream server for systemd-resolved.

Ugh. Is there any way to opt out of this behavior? I'd rather all my systems just pulled their DNS from my router every time rather than adding added complexity to the OS itself.

Yes, /etc/network/interfaces is basically obsolete: it's a configuration file for the ifupdown network management system, and ifupdown has been replaced by either NetworkManager or, starting with Mint 19, systemd-resolved. Google in fact seems to indicate that Ubuntu 18.04 does not offer ifupdown anymore by default -- but although I normally use NetworkManager, I just checked and it seems to on Mint 19 still work the same as ever.

Ugh again. Why do they keep changing stuff like this. The ideal way to configure and set up a linux system is - IMHO - entirely through simple text config files, like /etc/network/interfaces I'm glad I can still use it in mint, but I dread the day it goes away all together I absolutely hate the NetworkManager taskbar app. For every version of Mint that has come with it, I've immediately removed it, and set up the network manually in /etc/network/interfaces as part of my initial setup.

I don't understand why everyone keeps making Linux more like Windows. The reason I switched to Linux 20 years ago was because I didn't like windows.

It would've been helpful if you had posted your /etc/network/interfaces. A working one here is:

Code: Select all

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto enp6s0
#iface enp6s0 inet dhcp
iface enp6s0 inet static
    address 192.168.1.8
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameserver 192.168.1.1

The dns-nameserver bit is configuration for the resolvconf program and makes me end up with:

Code: Select all

rene@t5500:~$ ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 29 aug 28 01:12 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
rene@t5500:~$ cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 192.168.1.1
nameserver 127.0.0.53

Here is mine:

I've basically carried the same file forward from install to install since probably 2009 or so. There's some old commented stuff in there from when I used to use 802.3 link aggregation.

eno1 is one of my two on board Intel 82579V gigabit Ethernet ports I use for my main network traffic.

enp2so is my 10GBaseT 82598EB adapter which is just a direct link to my NAS for maximum speed.

Code: Select all

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
address 10.0.1.116
gateway 10.0.1.1
netmask 255.255.255.0
dns-nameservers 10.0.1.1

auto enp2s0
iface enp2s0 inet static
address 10.0.2.116
netmask 255.255.255.0


#below is for old 802.3ad LACP configuration
#------------------------------------------------------------------------------
##eno1 is manually configured, and slave to the "bond0" bonded NIC
#auto eno1
#iface eno1 inet manual
#bond-master bond0
##
##enp10s0 ditto, thus creating a 2-link bond.
#auto enp10s0
#iface enp10s0 inet manual
#bond-master bond0
##
## bond0 is the bonded NIC and can be used like any other normal NIC.
## bond0 is configured using static network information.
#auto bond0
#iface bond0 inet static
#address 10.0.1.116
#gateway 10.0.1.1
#netmask 255.255.255.0
#dns-nameservers 10.0.1.1
## bond0 uses standard IEEE 802.3ad LACP bonding protocol
#bond-mode 4
#bond-miimon 100
#bond-lacp-rate 1
#bond-slaves eno1 enp10s0[/quote]

One of the reasons I hated network manager was because I couldn't do stuff like the commented bond section in it, at least at the time.   These days I just avoid it out of habit, because I know with /etc/network/interfaces I have complete flexibility to set things up the way I wanted, whereas with the GUI network manager frontend, I'm stuck with whatever fields they decided to add that the typical consumer uses, which is a real shame.   I want enterprise features even for home use.  Again, why I use Linux.


[quote=rene post_id=1524046 time=1536540473 user_id=189205]
If your /etc/resolv.conf is a link to under /run/systemd, it seems the first thing you need to check is whether or not your have the resolvconf package installed at all (I note that when I use the above commented out dhcp configuration in /etc/network/interfaces instead, the 192.168.1.1 as retrieved through DHCP from my router is configured as a systemd-resolved upstream same as when using NetworkManager).[/quote]

Hmm.   Looks like it is NOT a link.
[code]
$ ls -l /etc/resolv.conf
-rw-r--r-- 1 root root 718 Sep  9  2018 /etc/resolv.conf

But the file SAYS it is a link:

Code: Select all

$ cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 10.0.1.1
search localdomain

If I run systemd-resolve --status as the file suggests, I just get an error:

Code: Select all

$ systemd-resolve --status
systemd-resolve: unrecognized option '--status'

It looks like systemd-resolve is not installed:

Code: Select all

$ dpkg -l |grep systemd
ii  libpam-systemd:amd64                         229-4ubuntu21.4                              amd64        system and service manager - PAM module
ii  libsystemd0:amd64                            229-4ubuntu21.4                              amd64        systemd utility library
ii  libsystemd0:i386                             229-4ubuntu21.4                              i386         systemd utility library
ii  systemd                                      229-4ubuntu21.4                              amd64        system and service manager
ii  systemd-shim                                 9-1bzr4ubuntu1                               amd64        shim for systemd
ii  systemd-sysv                                 229-4ubuntu21.4                              amd64        system and service manager - SysV links

but resolvconf is:

Code: Select all

$ dpkg -l |grep resolvconf
ii  resolvconf                                   1.78ubuntu6                                  all          name server information handler

If you do have resolvconf installed; the only thing I can think of that is halfway grub related is you previously having set the "net.ifnames=0" kernel parameter and not now; having old eth0 and alike interface names in /etc/network/interfaces. NetworkManager ignores (by default) any interfaces mentioned in /etc/network/interfaces but would not ignore say enp6s0 if you are now using the new names. If your router then provides address and gateway but not DNS through DHCP -- I actually once had such a router -- the symptom would be explained.

And then again it might be as simple as not / no longer having resolvconf installed...

Hmm. I do remember being annoyed by the new device names once udev went to v197 (or whenever it started). I really did miss my eth0.

I had contemplated undoing this and going back to the old style names, but then I had the big network disaster of 2015 on my Debian Jessie server with 8 network interfaces, where they all decided to change names and none of my VM's VLAN's worked properly anymore, and I quickly realized why the new predictable names were useful.

I had to spend way too much time figuring out which mac address belonged to which adapter, and editing everything in /etc/udev/rules.d/70-persistent-net.rules. It was particularly annoying since it is a headless server, I had no ethernet, so I had to do it from the console with no copy and paste...

The etc/network/interfaces on that server - btw - is one of the prime reasons why I like manually configured networks in text files. It now only has 6 network interfaces, but they are a combination of one single interface, one bond of two interfaces, and one link aggregation of three interfaces, each assigned a virtual interface name so my VM's and containers can access them correctly. I don't know if network manager has more capability today than the last time i played with it, but when I last did, it certainly couldn't.

Anyway, no, I never passed this kernel parameter to my kernel.

Again, appreciate your time, and effort in explaining all this. It is rare to come across someone who really knows their stuff and is willing to educate.

Personally, I tend to prefer manual control of my system above all else, and the more GUI tools (which I can't easily see how they work) or automated tools are used, I feel less in control.

Maybe at some point there will be a distribution just for us old school text file configuration types, that shuns all of this modern stuff. I want Linux to stay Linux, not to morph into a Windows GUI wannabe.

[rene]

Ugh again. Why do they keep changing stuff like this.

In your linked thread I saw you mention 19 but on rereading I see you're actually still on 18.3? systemd-resolved is not yet relevant for 18 so you in fact should get to ignore it. No idea however how you then get that 127.0.0.53 nameserver so I must say the problem description is confusing me.

On 18.x and with NetworkManager removed (how and in what sense did you remove it?) the only thing you should need is a regular old, static /etc/resolv.conf. That is, with the first two just for good measure,

Code: Select all

$ sudo systemctl stop systemd-resolved
$ sudo systemctl disable systemd-resolved
$ sudo systemctl stop resolvconf
$ sudo systemctl disable resolvconf
$ sudo rm /etc/resolv.conf
$ echo "nameserver 10.0.1.1" | sudo tee /etc/resolv.conf

and, optionally, remove the dns-nameserver(s) line from /etc/network/interfaces altogether: its only purpose is to have resolvconf in fact create the now manually created, static /etc/resolv.conf.

N.B. The fact that it said anything about systemd-resolved is no doubt a simple matter of static such text in /etc/resolvconf/resolv.conf.d/head; does not in fact say anything about systemd-resolved in fact running on your system -- and on 18, it shouldn't be.

If you for some reason like to keep the dns-nameserver in /etc/network/interfaces then you should skip and/or systemctl enable/start resolvconf again, and make /etc/resolv.conf be a link to /run/resolvconf/resolv.conf -- but really, no need.

[mattlach]

In your linked thread I saw you mention 19 but on rereading I see you're actually still on 18.3? systemd-resolved is not yet relevant for 18 so you in fact should get to ignore it. No idea however how you then get that 127.0.0.53 nameserver so I must say the problem description is confusing me.

Yes, I am on 18.3. I just had a LM19v2 USB stick laying around from recently having installed it on another machine. I used that stick for my attempted boot repair, to chroot into my system.

On 18.x and with NetworkManager removed (how and in what sense did you remove it?) the only thing you should need is a regular old, static /etc/resolv.conf. That is, with the first two just for good measure,

Code: Select all

$ sudo systemctl stop systemd-resolved
$ sudo systemctl disable systemd-resolved
$ sudo systemctl stop resolvconf
$ sudo systemctl disable resolvconf
$ sudo rm /etc/resolv.conf
$ echo "nameserver 10.0.1.1" | sudo tee /etc/resolv.conf

and, optionally, remove the dns-nameserver(s) line from /etc/network/interfaces altogether: its only purpose is to have resolvconf in fact create the now manually created, static /etc/resolv.conf.

Thank you, I will look at this.

I disabled network manager as follows (from this post)

Code: Select all

systemctl disable NetworkManager.service

N.B. The fact that it said anything about systemd-resolved is no doubt a simple matter of static such text in /etc/resolvconf/resolv.conf.d/head; does not in fact say anything about systemd-resolved in fact running on your system -- and on 18, it shouldn't be.

If you for some reason like to keep the dns-nameserver in /etc/network/interfaces then you should skip and/or systemctl enable/start resolvconf again, and make /etc/resolv.conf be a link to /run/resolvconf/resolv.conf -- but really, no need.

Good to know! Thank you!

I will ahve to look into this further when I get home from work.

[rene]

I disabled network manager as follows (from this post)

Code: Select all

systemctl disable NetworkManager.service

On Mint 19 that is not in fact enough to disable NetworkManager, as a simple ps ax | grep NetworkManager after reboot shows. Here I need "mask" instead of "disable".

I asked since when I was testing things earlier, NetworkManager not in fact being stopped meant it took over /etc/resolv.conf management -- but when I just now retried things, for some to me unknown reason the above still works fine even with NetworkManager running. So YMMV: if you find NetworkManager f... rollicing with your as per above manually created /etc/resolv.conf upon reboot, "systemctl mask NetworkManager.service" it.