Possible hacking attack -Expert Help Needed

[ajayX1]

My System detail:
Kernel: 4.15.0-161-generic x86_64 bits: 64 compiler: gcc
v: 7.5.0 Desktop: Cinnamon 4.2.4 wm: muffin dm: LightDM Distro: Linux Mint 19.2 Tina
base: Ubuntu 18.04 bionic
My root and home directory are on same disk partition. Free disk space of disk is about 3GB . It is an 120GB SSD. fstrim is set to off. I have also installed wine.

So what is happening: About 3-4 times in a day out of sudden a notification appears showing disk space is low only ~750 mb space left. Fan start running too fast. Some app start writing data to disk partition. Within 2 minute the disk space resume back to what was earlier i.e. about 3GB. Well I am not able to trace what app or malware is doing this.

I ran ext4magic to recover files that were written during last that few minutes. off-course it shows something has happened as shown below:

Code: Select all

sudo ext4magic /dev/sda1 -H -a 1634964430 -b 1634973502

|-----------c_time  Histogram-----------------  after  --------------------  Sat Oct 23 10:17:10 2021
1634965337 :      505 |******************************************        |   Sat Oct 23 10:32:17 2021
1634966244 :       54 |*****                                             |   Sat Oct 23 10:47:24 2021
1634972593 :      201 |*****************                                 |   Sat Oct 23 12:33:13 2021
1634973500 :      601 |**************************************************|   Sat Oct 23 12:48:20 2021

|-----------d_time  Histogram-----------------  after  --------------------  Sat Oct 23 10:17:10 2021
1634965337 :     4614 |**************************************************|   Sat Oct 23 10:32:17 2021
1634966244 :       53 |*                                                 |   Sat Oct 23 10:47:24 2021
1634972593 :       53 |*                                                 |   Sat Oct 23 12:33:13 2021
1634973500 :       71 |*                                                 |   Sat Oct 23 12:48:20 2021
  
|-----------cr_time Histogram-----------------  after  --------------------  Sat Oct 23 10:17:10 2021
1634965337 :      474 |**********************************************    |   Sat Oct 23 10:32:17 2021
1634966244 :       47 |*****                                             |   Sat Oct 23 10:47:24 2021
1634972593 :      144 |**************                                    |   Sat Oct 23 12:33:13 2021
1634973500 :      510 |**************************************************|   Sat Oct 23 12:48:20 2021

But the files it recover it becomes impossible to track what files were modified/ tempered with and what process doing it. So I want help in tracing the culprit process/ app/ malware. Thanks

[MikeNavy]

Hi,

- The space left on your disk is too small: 10 to 15% free space lefts is necessary (installation of programs, temp files, logs, swap...).

- What you see might be the result of memory swaps on your SSD: if the memory size is small, the system is obliged to save one part on the memory on disk before to open a program (for example).

You could do some cleaning:

Code: Select all

sudo apt-get autoclean
sudo apt-get clean
sudo apt-get autoremove --purge

and, with your file manager, delete the content of the '.cache' directory in your '/home/user', where user is your username.

More generally: https://easylinuxtipsproject.blogspot.c ... -mint.html

You could also decrease swappiness: https://easylinuxtipsproject.blogspot.c ... html#ID1.4

Regards,

MN

[ajayX1]

Thanks, The more that I can add is that the this "low space" was already there fore more than 3 months. But I am facing the problem only from last approx 1 month.
my swapfile is 1.8GB.
Ram is 4GB.
Further I do not use any such heavy software like video editing.

I shall try to get as much detail as possible next time it happens.

[AndyMH]

A 120GB SDD is on the small side. Are you using timeshift and are you using the defaults? It will save snapshots in /timeshift and is a common cause of running out of drive space.

[ajayX1]

No timeshift all graphic effects OFF

[pdc_2]

so to what extent do you accept the advice that the drive is too small; so the inference is: that ...... that is where your problem lies;

.... to what extent do you feel you afford a new drive, to increase your storage capacity?

[SMG]

Thanks, The more that I can add is that the this "low space" was already there fore more than 3 months. But I am facing the problem only from last approx 1 month.

When was the last time you cleaned out all the old unused kernels? Your amount of free space is such that you are lucky you are not being blocked from logging in to Mint. You really need to clean the drive or off-load less-used files to other storage mediums.

[ajayX1]

I deleted old kernel and ran bleachbit almost 5 days back.

[gittiest personITW]

Might seem like a stupid question, but its worth asking - have you emptied recycle bin and open Nemo, right-click Open as Root and look in:
/root/.local/share/Trash
See how many gigs are in there and delete.
Close that red window and BE VERY CAREFUL what you do in that window.

[ajayX1]

/root/.local/share/Trash only ~100 kb

[gittiest personITW]

It was worth a go. Had a look in mine just now and there was about 20GB.

[gittiest personITW]

Could you try Baobab or Filelight in Software Manager to see if there is 'something' that sticks out.
It could just be, as others have said, you need a bigger drive.
Is there anything obvious we are overlooking, like you edit movie files, or work with raw files etc etc? Maybe ripping DVD's, or downloading Youtube vids or iso images of operating systems?

[ajayX1]

I uninstalled several apps that I think not required. It cleaned up lot of space. Now I come to the point.

After waiting for few days. As I suspect that attacker may be alert. Now I installed

Code: Select all

arpon

,

Code: Select all

fail2ban

and found that 2 unknown MAC addresses are showing up in log. I ran wireshark and searched for those unknown MAC address on web nothing found. They were suspicious. So I tried to find more detail using nmap using following commands:

Code: Select all

sudo nmap -sP  <IP address>
sudo nmap -sT <IP address>
sudo nmap -sS <IP address>
sudo nmap -O <IP address>
sudo nmap -A <IP address>

result was 80/tcp filtered http
443/tcp filtered https

MAC Address: AE:36:1D:XX:XX:XX (Unknown)
Too many fingerprints match this host to give specific OS details UNKNOWN OS

Today I also find one another MAC address trying to connect to my LAN that was earlier only visible 3 months back when I was using Wifi. Now for last few hours no such suspicious MAC address reflected in log.

something was really wrong and some one trying or already hacked in to my system.

[chiefjim]

One more thing to look at. Check your /media files, root and user. If you ever have attempted to save something to an external media but it wasn't yet mounted the system will still attempt to save it. Pretty easy to spot if you do so before mounting any other external media.

Been there, done that.

[ajayX1]

chiefjim I am not getting it. Can you please explain me how.

[sleeper12]

I deleted old kernel and ran bleachbit almost 5 days back.

Be careful with Bleachbit: https://easylinuxtipsproject.blogspot.c ... s.html#ID4

[Aztaroth]

About cleaning to get space, you can :
- clean the flatpak cache :

Code: Select all

sudo rm -rf /var/tmp/flatpak-cache-*

- keep only logs for one week :

Code: Select all

sudo journalctl --vacuum-time=1w

(corrected with help of t42's feedback)

Also, take a look at your biggest files and see if they're really needed. Here a possible how-to :
Enter this command (corrected with help of t42's feedback too) :

Code: Select all

cd /

You'll be at the top of the File System.
Now :

Code: Select all

sudo find -type f -exec du -Sh {} + | sort -rh | head -n 10

will display the 10 biggest files on your disk (if you want more, change the last number). Take a close look to them (see if Junior hasn't downloaded a pile of X-rated movies in 4K quality ) . Get rid of the unnecessary ones.

Warning : unplug any other external disk or USB key before using the sudo find command. They will be scanned too because linked in /mnt or /media.
However, for users wanting to participate to the longest find command contest :

Code: Select all

sudo find -type f ! -path "*/mnt/*" ! -path "*/media/*" -exec du -Sh {} + | sort -rh | head -n 10

will do it without unplugging.

[t42]

Though OP has been absent for a week, still...

- keep only logs for one week :

Code: Select all

sudo journalctl --vacuum-time=1s

reading the manual, journalctl(1):

Code: Select all

--vacuum-time=
           Removes the oldest archived journal files until ... ... all archived
           journal files contain no data older than the specified
           timespan (specified with the usual "s", "m", "h", "days",
           "months", "weeks" and "years" suffixes)

Enter this command twice :

Code: Select all

cd ..

First time, it will replace the ~ of your prompt with /home, second time with /
You'll be at the top of the File System.

why not cd / as working directory can change since opening a terminal session

[Aztaroth]

Though OP has been absent for a week, still...

- keep only logs for one week :

Code: Select all

sudo journalctl --vacuum-time=1s

reading the manual, journalctl(1):

Code: Select all

--vacuum-time=
           Removes the oldest archived journal files until ... ... all archived
           journal files contain no data older than the specified
           timespan (specified with the usual "s", "m", "h", "days",
           "months", "weeks" and "years" suffixes)

Enter this command twice :

Code: Select all

cd ..

(corrected with help of t42's feedback)

why not cd / as working directory can change since opening a terminal session

Did i write 1s ?
In my native language (French), s is week (semaine), but of course the idea was :

Code: Select all

sudo journalctl --vacuum-time=1w

However, on the principle, the idea wasn't so bad : with 1 sec vacuum-time, the journal logs would have almost been as clean as a whistle, but to realize it you'll have to --rotate too.

Code: Select all

sudo journalctl --vacuum-time=1s --rotate

About cd /, you're right too. But before "launching" the sudo command, I tested it on my system first, with each step in file hierarchy : ~, /home and / and instead of typing paths, I went to the upper level with cd .. as recalling the last or forelast command is easy with the up arrow. But it's obviously better to enter cd / when you're sure you want to go to the root directory.

Thanks for reading my post and feedback. Corrections have been made.