Cannot reach local webservers through public IP

[dissar]

Hi,

I am running several web services on a server at my home. I have configured NAT on my dd-wrt router.
They all work fine on my devices except on my desktop mint computer.

I am using duckdns which is automatically updated if my public IP changes.
I can ping that duckdns domain name, and ping will print my public IP address, which is the same as I will get from whatsmyip.com.
However, when I try either the IP or the duckdns-address in a browser, nothing happens until it times out. I also have a desktop application that is connecting to one of the web services which also does not work. curl returns '(28) Failed to connect to ... : Connection timed out'.

I have listed two DNS servers in my router. They are atm local adguard servers but I get the same result with OpenDNS, Google's DNS and others.

Code: Select all

systemd-resolve --status

lists one of them under 'Global' and the other two under my network card/'DNS Servers'. Here is also a third address which I do not recognize (starting with 148.). DNS Domain is '~.' and also one of the DNS server I have listed in my router.

Code: Select all

cat /etc/resolv.conf

The file is generated by resolvconf(8). nameserver: 127.0.0.53, search is one of the DNS servers I have listed in my router.

But I have a temporary solution. When I connect to VPN (Express VPN), it works.
However I cannot ping the address anymore. Perhaps if I wait some time ping will work. When I tried this before, it replied on ping, but ping listed a different IP than my public IP. So I think there is some magic happening through my VPN service.

uname -v returns: #144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021

Any help greatly appreciated. Let me know if I need to post more details.

[rene]

Given that ping displays the correct IP (and, supposedly, pings) and that using the IP rather than DNS-name in the browser does not change things, DNS would not be the issue.

Just to be certain: you have installed a webserver on some local system, presumedly not the Mint system itself, and can reach it by public IP from e.g. a Windows system, an Android system, ...., just not the Mint system, unless you on the Mint system go through a VPN.

This would seem to suggest that, again, a system-local firewall on the Mint system is interfering, with the VPN-route working due it not triggering on the then different interface/address.

We're having more of these lately, supposedly due to I believe Mint 20 preinstalling a firewall. As fas as I'm aware it's still not enabled by default though so did you experiment in that sense? I'm still on Mint 19 myself (and if it turns out that in fact Mint 20 itself not only preinstalls but even enables a firewall I'm going to be switching distributions). What does sudo iptables -L say and if more than basically nothing, was that your or Mint's doing?

[rene]

Note; thought I might as well upgrade a system and check myself and am (semi...) happy to report that no firewall is enabled by default on a new install of Mint 20.1.

Code: Select all

rene@t5500:~$ sudo ufw status
Status: inactive
rene@t5500:~$ systemctl status iptables-persistent
Unit iptables-persistent.service could not be found.
rene@t5500:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

If something else for you: please undo what you did.

[dissar]

Hi, Rene

Thank you for your reply. I disabled ufw and rebooted. It is now inactive but that did not do remedy this.
Here is some more info:

Code: Select all

$ cat /etc/issue
Linux Mint 19.1 Tessa \n \l

Code: Select all

$ sudo ufw status
Status: inactive

Code: Select all

$ systemctl status iptables-persistent
Unit iptables-persistent.service could not be found.

I can see that the output of iptables -L is different from yours on a new install. Can you help decode this for me?

Code: Select all

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere 

I am not sure if this is my or mints doing. I have had this install for several years and I cannot remember all the changes I have made.
Since this problem only affects my own public IP, and that I don't notice it while on VPN I cannot pin point at what time this problem started.

I tried wireshark on my interface and filtering my public ip. I don't want to post the results and reveal my address, but in a nutshell this is what happens:

• I go to a web browser and try to access web service A
• Wireshark, mint -> public IP: A few grey lines marked [SYN]
• Wireshark, mint -> public IP: A few grey lines marked both [TCP Retransmission] and [SYN]
• Wireshark, public ip -> mint: One red line marked [RST, ACK]
• Wireshark, mint -> public IP: A few more [TCP Retransmission]
• I close the web browser tab and open a new one. I type the address of web service B (different server on the same network)
• Wireshark, mint -> public IP: A lot of grey lines marked [SYN]
• Wireshark, mint -> public IP: A lot of grey lines marked [TCP Retransmission] and [SYN]
• I close the second tab

So one of the web services (or the router?) actively denies one of the requests ([RST, ACK]), but the other one never replies.

[rene]

Aah, it's docker in your case, seemingly one of the few good reasons to actually have a firewall enabled. However, seeing as how you did not remark on that it might be the case that you're not in fact using docker containers? I've no personal experience with it, but judging from this:

https://www.digitalocean.com/community/ ... untu-20-04

you must have at one point explicitly installed it. If you don't use it, it'll be best to get rid of it again or at least disable it. That is, I expect that

Code: Select all

systemctl status docker

says that it's active? If so,

Code: Select all

systemctl stop docker && systemctl disable docker

and if that given link is how you once installed it, sudo apt-get purge --autoremove docker-ce.

Now, of course, if you do want to use docker for different purposes it should be perfectly possible to (re-)configure iptables to have that happen, but frankly, having no experience with docker I wouldn't be able to tell you if you could do so just in the regular iptables-persistent manner or if docker would now override that. Before I try and find out how and what as to needed rules, please confirm/deny that simply disabling/removing all of docker isn't an option or doesn't (yet) help.

[EDIT] as to that docker link; I just see you are using Mint 19 i.e. Ubuntu 18.04, not 20. As such this may be the better link: https://docs.docker.com/engine/install/ubuntu/. Seems to say for complete removal:

Code: Select all

sudo apt-get purge --autoremove docker docker-engine docker.io containerd runc docker-ce docker-ce-cli containerd.io

followed by sudo rm -rf /var/lib/{docker,containerd} if you have old images under there that you want gone.

[dissar]

Hi,

Yes, docker was active. I'm not actively using it on this system though. I'm tinkering with it from time to time, and I also use certain desktop application through docker.

I tried to stop and disable it, and rebooted. It removed all the docker stuff from iptables, but no cigar.
Firewall still disabled.

Removed docker (all packages as per your edit) and rebooted again. Still nothing.
Tried wireshark again, still the same pattern.

Code: Select all

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

[rene]

Urmpf. Nothing from your iptables output jumped out at me indeed but it's been years since I actually dealt with iptables personally so though I'd cheat and ask you to get rid of docker entirely so as to keep me from having to dig through rules...

Can you, just to test, if not already installed instal an SSH server on the system that you can't reach the webserver on (sudo apt-get install openssh-server if it is/were an Ubuntu/Mint system) and forward port 22 in your modem/router and see if you can ssh to its public IP from Mint? That would rule out anything specific to the webserver (and I'd be left with some real trouble trying to explain the VPN difference).

[dissar]

I have two servers, a debian system and an ubuntu server. They both have ssh installed.
Made a NAT rule to forward to port 22. Tried this on both servers.

I cannot reach any of them from my mint desktop (connection times out). From the termius app on an iPhone I can connect just fine.

[rene]

I was starting to suggest that the difference between the normal and VPN-situation might be respectively IPv6 vs. IPv4 until I reread your OP and noticed again you were also connecting directly to the IP; the IPv4 I assume. To humour me, I don't suppose that ssh -4 ... from the Mint system works either?

Grasping at straws but do you have peculiar routing set up on the Mint system? Nothing odd in route -46n?

Thinking, but I might need to give up: seems to not make sense. Yes, maybe the dd-wrt router rather than the Mint system itself?

[dissar]

Trying to force ipv4 ssh -4 ... also did nothing.
I don't want to post the results of route -46n not to reveal any sensitive information. I am not familiar with route and have never set anything up there.

I tried to connect my phone via USB and use that network interface. That worked.
I also booted a Windows 10 system. Same problem there, and it also works if I connect through a VPN.
So I think you are right, there might be something in the dd-wrt router and it seems like this thread is not a linux mint networking issue.
I am the sole administrator of the router, and yet I don't recall doing anything regarding this machine specifically except giving it a static IP outside the scope of the DHCP server. Might that have something to do with it?

[rene]

Might that have something to do with it?

Supposedly, yes. Your phone assumedly got a DHCP-retrieved dynamic IP from the DD-WRT router and if that works there seems little other opportunity. Given that behaviour's the same on a W10 box it'll be a DD-WRT-sides issue --- but what I always advise anyway is to not give individual machines a "static IP" in the old-fashioned actually-static sense, but a what I dub "steady IP" where you keep the individual machine on DHCP but configure the router to hand out the same IP to the same machine via DHCP MAC-based IP-reservations.

I'd quite advise that, if you don't feel like digging through DD-WRT's setup, and frankly always: you supposedly reinstall/reconfigure individual machines more often than the router.

If it turns out to be that, that's interesting to know...

[dissar]

Actually, the "static" IP was issued by the router by assigning the IP to the mac address.
On dd-wrt Services/DHCP Server/Static leases.
I tried now, to remove my machine from the list but still nothing. I now have a dynamic IP from the DHCP servers range.
The phone, I believe, is connected through it's mobile network/APN and not WiFi when it 'shares' internet through USB.

I might try to reset the router somehow, or reinstall it. But first I'll try to get a USB network card, just to try a different interface/MAC.
I'll post results.

So far, your help is greatly appreciated. Thank you.

[rene]

Only thing is that it isn't helping. Yes, phone not on Wi-Fi makes more sense than what I assumed: I'm not a mobile-tech person. For now no other ideas left than indeed slapping that router about...

[m00tpoint]

In my (enterprise) experience, NAT's are applied from a specific interface to one or more other interfaces, e.g. "outside" to "inside." Your attempts to create a tcp session to an "inside" server from an "inside" host through a NAT on the outside interface of the router strike me as quite unlikely to work.

Your pings to the public IP work because the public IP is on one of the router's interfaces. Your packets are being received by the system that has that IP address, so it replies. If your webserver were running on your router, things might possibly work, because there's no NAT involved, just packets arriving on an interface on a system with multiple interfaces. But traversing the NAT is another matter entirely.

When you connect through your VPN service, things work because your traffic leaves your router, gets re-addressed by the VPN, and then hits your server from the outside, properly traversing the NAT.

The usual solution to resolving www.example.com for inside users and outside users to have split DNS, where a dns server inside the network responds with the private address of the server, whereas the outside world sees the same server through the public IP it's natted to. You might consider having the router serve the private IP of the server when it receives a DNS query for your URL.

m00tpoint

[rene]

Yes, the ping thing is/was useful basically only as to confirm DNS; many modem/routers are configured to drop pings anyway so not too useful any more as a diagnostic tool generally.

It was otherwise specified though that things work fine from his other "devices" --- although admittedly if those "devices" were as his phone above not in fact on his LAN that doesn't say much either. But it in any case works fine for me on Mint 19 and 20 to e.g. ssh to my public IP when the relevant public port is forwarded in my modem/router to 22 on a local machine, both from said local machine itself and a different machine on my LAN. Not an issue...

Have also thought about e.g. router-behind-modem in the sense of double NAT and/or DMZ and all --- but if there's a possibility that that can be involved after being given that things work from a VPN-connection then I'm not seeing it. Really the only thing that made and still makes sense to me is a firewall dropping the connect, if not on either Mint- or server-system itself then on (modem or) router.

[dissar]

Hi, just to clarify a few things;

The ddwrt-router that I configured NAT on has obtained the public IP. Its WAN port is connected directly to a fiber to copper (RJ45) converter so there should not be a double NAT.

When I try to ping my public IP, I don't use the IP-address, but my duckdns domain name; ping {my domain}.duckdns.org
The ping program then returns xx bytes from {my-isp.something.xx} ({ip address}): icmp_seq=1 ttl=64 time=0.192 ms
I can confirm that the {ip address} is the same as displayed on the ddwrt web interface (WAN IP).

The other devices I have tried it on are
- Another mint computer (20 Ulyana) only connected through WiFi on the same LAN. Works.
- A MacBook only connected through WiFi on the same LAN. Works.
- An iPhone. I disabled mobile data to make sure that it could only connects through WiFi on the same LAN. Works.
- Windows 10 (dual boot, same computer as my primary mint system). Does not work.

I guess I could look at setting up a split dns as proposed, but why does it work for all my other devices and just not this one computer?
I can connect to my web services just fine by using the local IP as well. But because I'm using https I get an error because the certificate is for my duckdns domain name. Not really a big deal, I can make an exception in my browser but it just bothers me that one of my computers needs to do it that way.

[rene]

Had you not already tried with a real dynamic IP the conclusion would be that it's the in the router configured IP for that system that is outside the subnet that e.g. that Mint 20 system is on. And in that sense I do advise to look at e.g. ip addr in detail to see if the working and not working Mint system get e.g. the same netmask (both show a 192.168.<same>.<different>/24 address, say) --- but given that it also didn't work with full-dynamic it does not make a lot of sense, and you may need to post less-censored results for anyone to have a further chance. Also, still that previously suggested route -46n output.

[dissar]

Yep, can confirm. ip addr on both computers show that their IP is 10.10.10.different/24.

Here is the output from route -46n. As far as I can see, only sensitive information is the ipv6 addresses which I removed.

The non working mint 19 Tessa computer

Code: Select all

$ route -46n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.10.1      0.0.0.0         UG    100    0        0 enp3s0
10.10.10.0      0.0.0.0         255.255.255.0   U     100    0        0 enp3s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 enp3s0
Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
fe80::/64                      ::                         U    100 1     0 enp3s0
fe80::/64                      ::                         U    256 1     0 enp3s0
::/0                           ::                         !n   -1  1     1 lo
::1/128                        ::                         Un   0   6     9 lo
fe80::I:hid:this:part/128      ::                         Un   0   2     0 enp3s0
ff00::/8                       ::                         U    256 5   669 enp3s0
::/0                           ::                         !n   -1  1     1 lo

Working mint 20 Ulyana computer

Code: Select all

$ route -46n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.10.1      0.0.0.0         UG    600    0        0 wlp112s0
10.10.10.0      0.0.0.0         255.255.255.0   U     600    0        0 wlp112s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 wlp112s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
::/0                           ::                         !n   -1  1     0 lo
::1/128                        ::                         U    256 1     0 lo
fe80::/64                      ::                         U    256 1     0 vethf37fa6b
fe80::/64                      ::                         U    256 1     0 docker0
fe80::/64                      ::                         U    600 1     0 wlp112s0
::/0                           ::                         !n   -1  1     0 lo
::1/128                        ::                         Un   0   14     0 lo
fe80::I:hid:this:part/128      ::                         Un   0   2     0 docker0
fe80::I:hid:this:part/128      ::                         Un   0   4     0 wlp112s0
fe80::I:hid:this:part/128      ::                         Un   0   2     0 vethf37fa6b
ff00::/8                       ::                         U    256 14     0 wlp112s0
ff00::/8                       ::                         U    256 13     0 vethf37fa6b
ff00::/8                       ::                         U    256 13     0 docker0
::/0                           ::                         !n   -1  1     0 lo

[rene]

Although the difference between Mint 19 and 20 would not be explained by it 10.10.10.0/24 is sort of odd. 10.x.y.z/8 is a class A network. You've subnetted that yourself which isn't generally speaking wrong per se --- but I'd personally still much advise reconfiguring the router to 192.168.x.0/24 even if only to get the possibility that your router is getting something wrong there out of the way. Seemingly not a big chance either, but well, what remains that has...?

[dissar]

Hi,

Yes, I reconfigured the IP range myself. It's a class A ip but I don't get why that would be a problem.
However, I tried what you suggested.
- change routers ip to a class C network, 192.168.10.1
- change the static leases on server to be in the 192.168.10.0 network
- change NAT rules to new address
- reach the server from my phone through duckdns name - ok
- reach the server from my mint pc through its ip address - ok
- reach the server from my mint pc through duckdns name - failed

There must be something in my routers configuration. Haven't gotten a USB network interface yet, but I want to try with a different MAC address.