[solved] Is there a way to tame systemd-resolved in Split DNS Setup?

[sebbo]

Update: it seems I found the root cause - maybe don't bother reading all this and have a look at the solution first to understand the actual problem faster.

Hi,

I'm having a kind of a problem with my Split DNS Setup for quite some time now and now it got so bad, I can't just accept it any longer.
It used to work without any problems some time ago, but in the past months this started to make problems and it got worse.

The following is my setup:

Laptop, Linux Mint 20.2
Ethernet and WiFi
IPv6 disabled for some time now (yes, slap me in the face, I even work with IPv6 in my day job, so I know that disabling IPv6 shall never be used to solve Problems )

Router: 192.168.2.1
I run a DNSmasq on a BananaPi which serves as a DHCP Server and a DNS Server: 192.168.2.15

I have several services running on different machines, lets call them
service1.mydomain.tld 192.168.2.43
service2.mydomain.tld 192.168.2.44
service3.mydomain.tld 192.168.2.60


The Dnsmasq on the BananaPi/192.168.2.15 will resolve these domain names in my local network.

In the public DNS, all these three DNS records point to a CNAME
service1.mydomain.tld ---> CNAME dynamic.mydomain.tld
service2.mydomain.tld ---> CNAME dynamic.mydomain.tld
service3.mydomain.tld ---> CNAME dynamic.mydomain.tld
---> dynamic.mydomain.tld ---> A record: my temporary/dynamic Public WAN IP from the router (192.168.2.1)

Locally, dynamic.mydomain.tld ---> 192.168.2.30 (for some legacy reasons)

Now the problem:
When i am at home, after boot, the serviceX names will happily resolve to the local 192.168.2.x Addresses and Nextcloud, XMPP and whatelse i'm selfhosting at home will run just fine.

However, certain applications seem to trigger some mechanism, that will make resolved (or whatever else component there is in all the abstractions and cachings) resolve ServiceX to the public DNS informatoin, i.e. dynamic.mydomain.tld.
Funnily, once this is cached somehow, All serviceX domains will resolve to 192.168.2.30.
As there is no XMPP, no NextCloud and all the other services running, all the applications start failing.

I have written some script that will flush the DNS cache and then it might work.

Here it is more concrete

1. Reboot
2. Open terminal
3. watch -n 0.1 dig service1.mydomain.tld
----> A 192.168.2.XX

4. Start Thunderbird
5. Thunderbird will update Caldav calendar on my nextcloud
6. I can see immediately that the watch -n 0.1 dig service1.mydomain.tld changes to
CNAME dynamic.mydomain.tld
A 192.168.2.30

Reminder: 192.168.2.30 is a server that does not know about the service running on 192.168.2.XX

I have the same problem with dino (a modern XMPP client - it is actually nice software! just saying)
Here the scenario is

1. Reboot
2. Open terminal
3. watch -n 0.1 dig service1.mydomain.tld
----> A 192.168.2.XX

4. Start dino
5. Dino will happily connect XMPP to the XMPP server -

6. send an image via xmpp http_upload (which runs on xmpp.mydomain.com - 192.168.2.43) to the account connected in dino
7. Dino will resolve the http link

8. I can see immediately that the watch -n 0.1 dig service1.mydomain.tld changes to
CNAME dynamic.mydomain.tld
A 192.168.2.30


I can reproduce both scenarios perfectly.

I can't explain what is going on and i can't really find someone with the same situation.
The problem is: I can't really hardcode the DNS server somewhere as i use my laptop somewhere else and there DNS might be on 192.168.2.1 or so and not .2.15 ...

TLDR: how can i make systemd-resolved just do its f*cking(sorry) job to query the DNS server it was supposed to query because it is configured via DHCP.

Thanks for your help, it is really frustrating

br
sebastian

[djph]

looks like something's running DoH (DNS over HTTPS) queries instead of what you want. Just ran into that myself with a firefox update. Fix was to tell FF to not try resolving things with DoH

[sebbo]

looks like something's running DoH (DNS over HTTPS) queries instead of what you want. Just ran into that myself with a firefox update. Fix was to tell FF to not try resolving things with DoH

Thanks for your reply.

Your comment made me very curious and sent me down a 2 hour debugging rabbit hole.

1. Thunderbird is not configured to use DOH (for firefox i had disabled this explicitly anyway).

2. Using wireshark, i found out the following:

Everytime an IPv6 aware application tries to resolve serviceXY.mydomain.tld, it will ask for A and AAAA records.
Because my dnsmasq does not know any AAAA records, it will forward this to public dns.
Public dns will say "Hey, that is CNAME -> dynamic.mydomain.tld"
dnsmasq then says "Oh, i know that guy! it is 192.168.2.30" and will answer this together with the CNAME stuff to systemd-resolved or whatever is querying it.
(It might be that there is one more intermediate step, but that is more or less the result).

Funnily, applications will query for AAAA records, even though no usable interface is ipv6 enabled...

So... as IPv6 is not working anyway, my solution is, to just define the "undefined/unspecified" IPv6 address, which is "::" by the way, to all my services in the dnsmasq.

So far it is working.

So... not sure to really blame systemd-resolved in the end, but this weird cname caching, I still don't like it.

Thanks for reading so far

[donten]

I disable ipv6 entirely so far...Unless you have a definite need for it, of course.

Detailed info on how:

https://itsfoss.com/disable-ipv6-ubuntu-linux/

[sebbo]

I disable ipv6 entirely so far...Unless you have a definite need for it, of course.

Detailed info on how:

https://itsfoss.com/disable-ipv6-ubuntu-linux/

Disabling IPv6 is not the right way.

The number of IPv6 only hosts is rising due to IPv4 shortage.
The number of IPv6 only clients is also rising. Transition layers only add complexity and ideally everyone would just embrace and use IPv6!

I would only recommend to disable it if there are good reasons and leave it enabled by default.

[djph]

I disable ipv6 entirely so far...Unless you have a definite need for it, of course.

I would only recommend to disable it if there are good reasons and leave it enabled by default.

Generally the underlying problem is a garbage ISP implementation (e.g mine gives me a single /64... well, thanks but you're doing it wrong, and now v6 won't work in my LAN)


(edit - yes, I know how to quote... )

[donten]

So, you have a =definite= need for ipv6, is that right? If not, just do what I suggested. As far as I know, we have a ways to go before ipv6 is -required-.