I hit the systemd wall (probably)

Questions about cabled networking
Forum rules
Before you post please read how to get help
DAMIEN1307
Level 8
Level 8
Posts: 2093
Joined: Tue Feb 21, 2017 8:13 pm
Location: Alamogordo, New Mexico, USA

Re: I hit the systemd wall (probably)

Post by DAMIEN1307 » Wed Dec 19, 2018 3:34 pm

please copy and paste the code below and lets see what we got.

Code: Select all

ip route show 
ORDO AB CHAO

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Wed Dec 19, 2018 3:41 pm

Code: Select all

default via 192.168.5.99 dev enp2s0 proto static metric 100 
169.254.0.0/16 dev enp2s0 scope link metric 1000 
192.168.5.0/24 dev enp2s0 proto kernel scope link src 192.168.5.10 metric 100 
99 is my GATEWAY 10 is my machine
67 but still kicking

User avatar
trytip
Level 11
Level 11
Posts: 3711
Joined: Tue Jul 05, 2016 1:20 pm

Re: I hit the systemd wall (probably)

Post by trytip » Wed Dec 19, 2018 3:49 pm

is this running active:
systemctl status systemd-resolved
Image

gm10
Level 19
Level 19
Posts: 9248
Joined: Thu Jun 21, 2018 5:11 pm

Re: I hit the systemd wall (probably)

Post by gm10 » Wed Dec 19, 2018 3:50 pm

yossarian_yon wrote:
Wed Dec 19, 2018 3:18 pm
When I flip the switch in network manager left and right internet is working till the time that
some process do it again. Is that systemd-resolver or Network manager I don't know.
Ma actual network hardware is enp2s0
Should be easy enough to check the logs when that happens to see what triggers the change. Are you sure it's not that you've got multiple connections configured but only one correctly and then other one gets activated due to an interruption or something? Bit of a silly scenario but I haven't really heard of a statically configured gateway to get "lost" randomly. Something must trigger it. I doubt the resolver is involved though, I still don't think this has anything to do with DNS.

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Wed Dec 19, 2018 3:55 pm

trytip wrote:
Wed Dec 19, 2018 3:49 pm
is this running active:
systemctl status systemd-resolved

Code: Select all

~$ ip route show
default via 192.168.5.99 dev enp2s0 proto static metric 100 
169.254.0.0/16 dev enp2s0 scope link metric 1000 
192.168.5.0/24 dev enp2s0 proto kernel scope link src 192.168.5.10 metric 100 
yoss@gray:~$ systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-12-18 13:29:56 CET; 1 day 7h ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 804 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/systemd-resolved.service
           └─804 /lib/systemd/systemd-resolved

Dec 18 13:29:56 gray systemd[1]: Starting Network Name Resolution...
Dec 18 13:29:56 gray systemd-resolved[804]: Positive Trust Anchors:
Dec 18 13:29:56 gray systemd-resolved[804]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cd
Dec 18 13:29:56 gray systemd-resolved[804]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc68345710
Dec 18 13:29:56 gray systemd-resolved[804]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-
Dec 18 13:29:56 gray systemd-resolved[804]: Using system hostname 'gray'.
Dec 18 13:29:56 gray systemd[1]: Started Network Name Resolution.
lines 1-20/20 (END)
67 but still kicking

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Wed Dec 19, 2018 4:03 pm

Should be easy enough to check the logs when that happens to see what triggers the change. Are you sure it's not that you've got multiple connections configured but only one correctly and then other one gets activated due to an interruption or something? Bit of a silly scenario but I haven't really heard of a statically configured gateway to get "lost" randomly. Something must trigger it. I doubt the resolver is involved though, I still don't think this has anything to do with DNS.
What logs should I check? Yes there was in the past multiple connections and something went wrong.
I said multiple times that there is nothing wrong with DNS.
67 but still kicking

gm10
Level 19
Level 19
Posts: 9248
Joined: Thu Jun 21, 2018 5:11 pm

Re: I hit the systemd wall (probably)

Post by gm10 » Wed Dec 19, 2018 4:07 pm

yossarian_yon wrote:
Wed Dec 19, 2018 4:03 pm
What logs should I check? Yes there was in the past multiple connections and something went wrong.
I said multiple times that there is nothing wrong with DNS.
I know but you keep bringing up the resolver (i.e. DNS). ;)

/var/log/syslog records gateway changes but since you're running journald you can check the journal via journalctl, that aggregates basically all logs and has more information in particular for systemd-related actions.

DAMIEN1307
Level 8
Level 8
Posts: 2093
Joined: Tue Feb 21, 2017 8:13 pm
Location: Alamogordo, New Mexico, USA

Re: I hit the systemd wall (probably)

Post by DAMIEN1307 » Wed Dec 19, 2018 4:10 pm

DO NOT RUN THIS UNTIL GM10 CAN LOOK AT THIS FIRST AND SAY IF ITS GOOD OR NO GOOD!!!..he knows better than me if this might be of use...please wait for him to authenticate...DAMIEN

Code: Select all

ip route add default via 192.168.5.99 dev enp2
ip route add will add provided default gateway.

default means target network is all which is default.

via 192.168.5.99 is your default gateway network address.

dev enp2 is your network interface for default gateway.
ORDO AB CHAO

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Wed Dec 19, 2018 4:17 pm

I know but you keep bringing up the resolver (i.e. DNS). ;)
Isn't resolver duty to find GATEWAY address too? :?: I was sure that is.
Last edited by yossarian_yon on Wed Dec 19, 2018 4:26 pm, edited 1 time in total.
67 but still kicking

gm10
Level 19
Level 19
Posts: 9248
Joined: Thu Jun 21, 2018 5:11 pm

Re: I hit the systemd wall (probably)

Post by gm10 » Wed Dec 19, 2018 4:25 pm

DAMIEN1307 wrote:
Wed Dec 19, 2018 4:10 pm
DO NOT RUN THIS UNTIL GM10 CAN LOOK AT THIS FIRST AND SAY IF ITS GOOD OR NO GOOD!!!..he knows better than me if this might be of use...please wait for him to authenticate...DAMIEN

Code: Select all

ip route add default via 192.168.5.99 dev enp2
lol, nothing wrong with that (except it should need a sudo I imagine) and in any case nothing that cannot be undone. Preferably we discover what's causing the problem but it's not a bad idea to manually add the gateway after it did get lost and see what it says (e.g. unreachable).
yossarian_yon wrote:
Wed Dec 19, 2018 4:17 pm
I know but you keep bringing up the resolver (i.e. DNS). ;)
Isn't resolver duty to find GATEWAY address? :?: I was sure that is.
https://www.freedesktop.org/wiki/Softwa ... /resolved/

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Thu Dec 20, 2018 5:24 am

Now I found something in "syslog" which doesn't mind much to me. There is change of "dhcpcd" which I don't understend. And this "avahi-daemon" is whitening that "102" address which is unnecessary, but doesn't write GATEWAY address which is necessary. I cut what I mind is interesting part. Some light please :)
12:12 am
I'm reading about avahi-daemon, is that necessary for Mint? Can I disable it?
13:00 am
Uf it's not that simple :)
just reminder
192.168.5.102 # wild address from beginning of problem
192.168.5.99 # gateway
192.168.5.10 # my machine

Code: Select all

Dec 20 09:35:40 gray NetworkManager[834]: <info>  [1545294940.1874] connectivity: (enp2s0) timed out
Dec 20 09:35:40 gray dbus-daemon[820]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=834 comm="/usr/sbin/NetworkManager --no-daemon " label="unconfined")
Dec 20 09:35:40 gray NetworkManager[834]: <info>  [1545294940.1874] manager: NetworkManager state is now CONNECTED_SITE
Dec 20 09:35:40 gray dhcpcd[949]: enp2s0: deleted IP address 192.168.5.102/24
Dec 20 09:35:40 gray systemd[1]: Starting Network Manager Script Dispatcher Service...
Dec 20 09:35:40 gray dhcpcd[949]: enp2s0: deleting default route via 192.168.5.99
Dec 20 09:35:40 gray dhcpcd[949]: enp2s0: deleting route to 192.168.5.0/24
Dec 20 09:35:40 gray avahi-daemon[869]: Withdrawing address record for 192.168.5.10 on enp2s0.
Dec 20 09:35:40 gray avahi-daemon[869]: Leaving mDNS multicast group on interface enp2s0.IPv4 with address 192.168.5.10.
Dec 20 09:35:40 gray avahi-daemon[869]: Joining mDNS multicast group on interface enp2s0.IPv4 with address 192.168.5.102.
Dec 20 09:35:40 gray avahi-daemon[869]: Withdrawing address record for 192.168.5.102 on enp2s0.
Dec 20 09:35:40 gray avahi-daemon[869]: Leaving mDNS multicast group on interface enp2s0.IPv4 with address 192.168.5.102.
Dec 20 09:35:40 gray avahi-daemon[869]: Interface enp2s0.IPv4 no longer relevant for mDNS.
Dec 20 09:35:40 gray avahi-daemon[869]: Joining mDNS multicast group on interface enp2s0.IPv4 with address 192.168.5.102.
Dec 20 09:35:40 gray avahi-daemon[869]: New relevant interface enp2s0.IPv4 for mDNS.
Dec 20 09:35:40 gray avahi-daemon[869]: Registering new address record for 192.168.5.102 on enp2s0.IPv4.
Dec 20 09:35:40 gray avahi-daemon[869]: Registering new address record for 192.168.5.10 on enp2s0.IPv4.
Dec 20 09:35:40 gray dbus-daemon[820]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Dec 20 09:35:40 gray nm-dispatcher: req:1 'connectivity-change': new request (1 scripts)
Dec 20 09:35:40 gray systemd[1]: Started Network Manager Script Dispatcher Service.
Dec 20 09:35:40 gray nm-dispatcher: req:1 'connectivity-change': start running ordered scripts...
Dec 20 09:35:40 gray nm-dispatcher: req:2 'connectivity-change': new request (1 scripts)
Dec 20 09:35:40 gray nm-dispatcher: req:2 'connectivity-change': start running ordered scripts...
whole log on this address
https://jumboiskon.tportal.hr/share/994 ... e85fbdda71
67 but still kicking

gm10
Level 19
Level 19
Posts: 9248
Joined: Thu Jun 21, 2018 5:11 pm

Re: I hit the systemd wall (probably)

Post by gm10 » Thu Dec 20, 2018 8:52 am

Ok, so it's a DHCP issue as expected. I slimmed your log further down to highlight what happened:
yossarian_yon wrote:
Thu Dec 20, 2018 5:24 am

Code: Select all

Dec 20 09:35:40 gray NetworkManager[834]: <info>  [1545294940.1874] connectivity: (enp2s0) timed out
Dec 20 09:35:40 gray NetworkManager[834]: <info>  [1545294940.1874] manager: NetworkManager state is now CONNECTED_SITE
Dec 20 09:35:40 gray dhcpcd[949]: enp2s0: deleted IP address 192.168.5.102/24
Dec 20 09:35:40 gray dhcpcd[949]: enp2s0: deleting default route via 192.168.5.99
Looks like the DHCP client couldn't retrieve an updated DHCP lease and thus reset your connection. Should be an issue with either the DHCP server or make sure you're not using MAC randomization on reconnect, it might de-auth you in that case. Or, as I mentioned earlier, just disable DHCP for that connection entirely and go with a static configuration. Make sure to disable DHCP for both IPv4 and IPv6 btw (or disable IPv6 entirely for the connection since you do not appear to be using it - set it to ignore).

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Thu Dec 20, 2018 9:30 am

Looks like the DHCP client couldn't retrieve an updated DHCP lease and thus reset your connection. Should be an issue with either the DHCP server or make sure you're not using MAC randomization on reconnect, it might de-auth you in that case. Or, as I mentioned earlier, just disable DHCP for that connection entirely and go with a static configuration. Make sure to disable DHCP for both IPv4 and IPv6 btw (or disable IPv6 entirely for the connection since you do not appear to be using it - set it to ignore).
I don't have MAC randomize, How I can disable DHCP? My router have disabled DHCP. My IPv4 addresses are on manual, my DNS is Automatic off.
What else I can change. You mention that I change that in network manager, but they are already there.
I'm sorry but i'm confused :?
67 but still kicking

gm10
Level 19
Level 19
Posts: 9248
Joined: Thu Jun 21, 2018 5:11 pm

Re: I hit the systemd wall (probably)

Post by gm10 » Thu Dec 20, 2018 9:41 am

yossarian_yon wrote:
Thu Dec 20, 2018 9:30 am
I don't have MAC randomize, How I can disable DHCP? My router have disabled DHCP. My IPv4 addresses are on manual, my DNS is Automatic off.
What else I can change. You mention that I change that in network manager, but they are already there.
I'm sorry but i'm confused :?
I'm confused as well because what the logs say should not be possible in that case for all I know. Maybe post all your connection settings:

Code: Select all

sudo cat /etc/NetworkManager/system-connections/*
Warning: If you've got any password-based authentication configured then the password will be in that output, check before posting.

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Thu Dec 20, 2018 10:10 am

I'm confused as well because what the logs say should not be possible in that case for all I know. Maybe post all your connection settings:
That is now when eweryting is working. In min time I upgraded to "M19.1C" in hope that will help. Probably not

Code: Select all

yoss@gray:~$ sudo cat /etc/NetworkManager/system-connections/*
[sudo] password for yoss:     
[connection]
id=Iskon
uuid=3cc17dd4-a251-4f9b-881b-878c06f3f8e0
type=ethernet
permissions=
timestamp=1545257672

[ethernet]
cloned-mac-address=D0:50:99:58:C4:FA
mac-address=D0:50:99:58:C4:FA
mac-address-blacklist=

[ipv4]
address1=192.168.5.10/24,192.168.5.99
dns=192.168.5.55;
dns-search=
ignore-auto-dns=true
ignore-auto-routes=true
method=manual
67 but still kicking

gm10
Level 19
Level 19
Posts: 9248
Joined: Thu Jun 21, 2018 5:11 pm

Re: I hit the systemd wall (probably)

Post by gm10 » Thu Dec 20, 2018 10:44 am

Let's try this. Edit the connection file in your favourite text editor in root mode. In case of doubt:

Code: Select all

sudo -e /etc/NetworkManager/system-connections/*
Replace the contents with this:

Code: Select all

[connection]
id=Iskon
uuid=3cc17dd4-a251-4f9b-881b-878c06f3f8e0
type=ethernet
permissions=

[ethernet]
mac-address-blacklist=

[ipv4]
address1=192.168.5.10/24,192.168.5.99
dns=192.168.5.55;
dns-search=
may-fail=false
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
ip6-privacy=0
method=ignore
Then restart Network Manager:

Code: Select all

sudo systemctl restart NetworkManager.service

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Thu Dec 20, 2018 12:01 pm

Done everything you sad, and completely lost internet so I'm back on "M18.3C". I reverted your settings to backup I made,
restarted network manager but no honey. In "nmcli" everything looks good. Now I'm competently lost :( :?
67 but still kicking

gm10
Level 19
Level 19
Posts: 9248
Joined: Thu Jun 21, 2018 5:11 pm

Re: I hit the systemd wall (probably)

Post by gm10 » Thu Dec 20, 2018 12:06 pm

yossarian_yon wrote:
Thu Dec 20, 2018 12:01 pm
Done everything you sad, and completely lost internet so I'm back on "M18.3C". I reverted your settings to backup I made,
restarted network manager but no honey. In "nmcli" everything looks good. Now I'm competently lost :( :?
Well yes, the backup was in this thread so I wasn't worried about that. Losing connectivity even after restoring settings is weird though. Did you remember to restart NetworkManager.service? Or reboot? Also did the connection get established at all with my settings or what happened?

Among the changes I did was the removal of your cloned MAC address and the disabling of IPv6 (frankly I didn't understand why your connections file didn't have an [ipv6] section but that's another story. Oh and I removed the ignore-auto-routes=true since you hadn't defined any manual routes. Frankly those settings should have worked.

User avatar
yossarian_yon
Level 1
Level 1
Posts: 23
Joined: Sun Jun 03, 2018 3:21 am

Re: I hit the systemd wall (probably)

Post by yossarian_yon » Thu Dec 20, 2018 12:35 pm

gm10 wrote:
Thu Dec 20, 2018 12:06 pm

Well yes, the backup was in this thread so I wasn't worried about that. Losing connectivity even after restoring settings is weird though. Did you remember to restart NetworkManager.service? Or reboot? Also did the connection get established at all with my settings or what happened?
Yes I restarted Net. manager.service (no hony) and after that I rebooted. No connections, I ping google that passed so mast be something with DNS. I changed DNS to 1.1.1.1 but nothing. I'm very, very close to reinstall complete system. Probably tomorrow.
Among the changes I did was the removal of your cloned MAC address and the disabling of IPv6 (frankly I didn't understand why your connections file didn't have an [ipv6] section but that's another story. Oh and I removed the ignore-auto-routes=true since you hadn't defined any manual routes. Frankly those settings should have worked.
Probably because my ISP doesn't use "ipv6". I don't know, didn't think about that.
67 but still kicking

gm10
Level 19
Level 19
Posts: 9248
Joined: Thu Jun 21, 2018 5:11 pm

Re: I hit the systemd wall (probably)

Post by gm10 » Thu Dec 20, 2018 12:45 pm

yossarian_yon wrote:
Thu Dec 20, 2018 12:35 pm
I ping google that passed so mast be something with DNS.
Yay, finally a DNS issue. :lol:

Check

Code: Select all

systemctl status systemd-resolved
status message and log below, as well as

Code: Select all

systemd-resolve --status
Also try this:

Code: Select all

systemd-resolve --flush-caches
systemd-resolve google.com

Post Reply

Return to “Ethernet”