Will get the logs if you think it shed some light on the situation, however I am not exactly see how would they helpful, since it is obvious that the process eating up my memory is the xfdesktop process. Since I wrote here it is at 3.5GB now so still growing.
I did searched first and I also saw a couple of 3+ years old threads on the internet about xfdesktop memory leaks but there were never an exact solution also I thought that memory leaks that old should be already fixed. Maybe I'm just too naive.
I'm actually coming from Mate since one day it just randomly started glitching out till the point it never loaded again. Did not managed to fix it. Again I did not used that Mint version as an active desktop as well, just running a few services on it like torrenting, samba, nfs. Everything else was "factory default" like in the current case. After ~2 years Mate just gave up. I was pretty pissed that a window manager just dying like that when I did not even use the machine regularly (I log in, if I have to reconfigure something like adding new shares or changing the raid config, that's all).
So I thought that xfce is a bit more stable window manager being a more mature software. Maybe I was just too naive again.
Also since I installed this system a few years back there was no problem with xfdesktop. It just started behaving like this a couple of months ago.