using rsync w/ external /media eats root storage

Questions about applications and software
Forum rules
Before you post please read how to get help
Post Reply
kurtking
Level 1
Level 1
Posts: 39
Joined: Thu Aug 02, 2012 9:51 am

using rsync w/ external /media eats root storage

Post by kurtking » Thu Sep 05, 2019 8:58 pm

While creating new /home partition backup (by folder groups) to an external USB HDD w/ an RSYNCs script, Cindy ran out of / space. I boosted root partition from 30 to 140 GB in stages. After adding 30 GB and quickly running out of root space, at 2nd restart w/ 80 GB more the / usage was 49%. While running the Documents folder backup it climbed past 94%. Before I could quit the backup process I had gotten a low storage warning. Trying to restart, entering a password fails because of low storage, After having come to a low storage warning (before a no storage message) you must add storage. Restarting without more storage, after the password is entered I got an Xsession message: “warning: unable to write to /tmp;X session may exit with an error” w/ an OK button.

The BIG problem here is that the /media/home/device folder is never cleared from RSYNC usage even after removing the device & restarting. It retains the full text of every file written. Thus it expands the root partition without limit.

From the time I started allocating a separate home w/ Betsy, I found that the root partition could be small, so I’ve been living with a 30 GB root for 5 years or so, occasionally running my backup, and never ran out of storage. But Cindy doesn’t forget media storage with rsync.

Since I’ve already lost a whole day with this and am running out of storage I didn’t test the similar backup function to an internal HDD.

Sample RSYNC folder group script:

Code: Select all

# kkbkupext
... 
echo "Documents files and sub-dirs"
rsync -avu \
--include=/home/kurt/Documents/* \
--include=*/ --include=*.* --exclude=/* \
/home/kurt/Documents/** /media/kurt/659186ea-07f8-4dea-94c2-eca5cba35de8/Documents/ >> /tmp/rsync_home_log
TIA for any help.
Kurt

rene
Level 12
Level 12
Posts: 4312
Joined: Sun Mar 27, 2016 6:58 pm

Re: using rsync w/ external /media eats root storage

Post by rene » Fri Sep 06, 2019 2:47 am

Although your --include's look weird, rsync is nothing but a fancy (s)cp: if your / filled up while copying to /media/kurt/what/ever then the only possible conclusion is that /media/kurt/what/ever lived on the / partition; that you neglected to insert/mount the external drive, that is. To fix, be sure to physically remove the external drive and delete everything under /media/kurt that should not be there (assuming you still have the original content, of course). Then reinsert external drive, be sure it mounts, and redo.

User avatar
WharfRat
Level 21
Level 21
Posts: 13360
Joined: Thu Apr 07, 2011 8:15 pm

Re: using rsync w/ external /media eats root storage

Post by WharfRat » Fri Sep 06, 2019 3:23 am

Just out of curiosity can you paste back the results of sudo du -hxd1 /media with the external unplugged.
ImageImage

User avatar
catweazel
Level 19
Level 19
Posts: 9749
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: using rsync w/ external /media eats root storage

Post by catweazel » Fri Sep 06, 2019 4:29 am

lexqbit wrote:
Fri Sep 06, 2019 4:25 am
You can use the timeshift app (it also uses srync) it might help you make it simple and saves you time by scheduling the backups. https://teejeetech.in/timeshift/ .I am not assuming you are a new linux user, I use Linux for many years and I jumped on the chance to use timeshift since it saves me time by doing backups.
Timeshift is not a backup program.
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.

User avatar
catweazel
Level 19
Level 19
Posts: 9749
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: using rsync w/ external /media eats root storage

Post by catweazel » Fri Sep 06, 2019 4:38 am

lexqbit wrote:
Fri Sep 06, 2019 4:35 am
I am 100% sure timeshift is a backup program ...
https://github.com/teejee2008/timeshift
It is NOT a backup tool and is not meant to protect user data.
The emphasis is by the author of Timeshift.
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.

User avatar
catweazel
Level 19
Level 19
Posts: 9749
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: using rsync w/ external /media eats root storage

Post by catweazel » Fri Sep 06, 2019 4:49 am

lexqbit wrote:
Fri Sep 06, 2019 4:45 am
catweazel wrote:
Fri Sep 06, 2019 4:38 am
lexqbit wrote:
Fri Sep 06, 2019 4:35 am
I am 100% sure timeshift is a backup program ...
https://github.com/teejee2008/timeshift
It is NOT a backup tool and is not meant to protect user data.
The emphasis is by the author of Timeshift.
Well when you copy paste something copy paste the entire sentense:
It is irrelevant. Timeshift is not a backup program. EOD.
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.

deepakdeshp
Level 16
Level 16
Posts: 6090
Joined: Sun Aug 09, 2015 10:00 am

Re: using rsync w/ external /media eats root storage

Post by deepakdeshp » Fri Sep 06, 2019 3:00 pm

IMO time shift is a system snapshot utility, which will restore the system to an earlier state. This wouldn't be possible without backing up system in the first place , then restoring it. This makes it a backup program for your os excluding your home folder, which can be added to the snapshot too.
If I have helped you solve a problem, please add [SOLVED] to your first post title, it helps other users looking for help, and keeps the forum clean.
Regards,
Deepak

I am using Mint 19.2 Cinnamon 64 bit with AMD A8/7410 processor . Memory 8GB

kurtking
Level 1
Level 1
Posts: 39
Joined: Thu Aug 02, 2012 9:51 am

Re: using rsync w/ external /media eats root storage

Post by kurtking » Sun Sep 08, 2019 12:12 pm

WharfRat wrote:
Fri Sep 06, 2019 3:23 am
Just out of curiosity can you paste back the results of sudo du -hxd1 /media with the external unplugged.

Code: Select all

kurt@cm600-e51-lmde3:~$ sudo du -hxd1 /media
[sudo] password for kurt:         
123G	/media/kurt
4.0K	/media/root
123G	/media
kurt@cm600-e51-lmde3:~$ 

rene
Level 12
Level 12
Posts: 4312
Joined: Sun Mar 27, 2016 6:58 pm

Re: using rsync w/ external /media eats root storage

Post by rene » Sun Sep 08, 2019 12:20 pm

That confirms the initial suspicion then. A mount point is just a directory; when you copied to under said directory when nothing was in fact mounted on it it just did exactly that: copy to under said directory, hence, onto the filesystem on which said directory lived, hence, the root filesystem. As said, with the external drive disconnected, just manually delete everything that should not in fact be there.

User avatar
WharfRat
Level 21
Level 21
Posts: 13360
Joined: Thu Apr 07, 2011 8:15 pm

Re: using rsync w/ external /media eats root storage

Post by WharfRat » Sun Sep 08, 2019 12:21 pm

If the external was not mounted when you ran du then 123G /media/kurt indicates that a backup occurred when the external was not mounted.

So essentially the files are residing in the / (root) folder of /media/kurt

Make sure you do have a backup on the external and if so just delete the content of /media/kurt with the external disconnected.
ImageImage

kurtking
Level 1
Level 1
Posts: 39
Joined: Thu Aug 02, 2012 9:51 am

Re: using rsync w/ external /media eats root storage

Post by kurtking » Sun Sep 08, 2019 3:59 pm

rene wrote:
Fri Sep 06, 2019 2:47 am
Although your --include's look weird, rsync is nothing but a fancy (s)cp: if your / filled up while copying to /media/kurt/what/ever then the only possible conclusion is that /media/kurt/what/ever lived on the / partition; that you neglected to insert/mount the external drive, that is. To fix, be sure to physically remove the external drive and delete everything under /media/kurt that should not be there (assuming you still have the original content, of course). Then reinsert external drive, be sure it mounts, and redo.
About two years ago I developed the scripts to backup and restore my /home folder / partition / drive. to either an internal drive or external, and after each use I’ve unmounted it when not doing the backup (or restore). If the process is to or from an external device, I turned off its power switch when not using it (the OS [now Nemo and Disks] does not see it in that state). Then either I leave it plugged in but unmounted, or unplug to use it elsewhere. (When plugged in and I turn it on, it automounts.) This problem of the 30 GB root being grossly insufficient started with Cindy. But maybe that's not the issue. See my questions below. See also my answer to WharfRat.

Each of the four scripts (int/ext, bkup/rest) that I run has 11 rsync commands – one for just the top /home folder (no recursion) and its files, and one for for each top level sub-folder (with recursion). I broke them into short lines so I could more easily compare them clause by clause to each other with xed – necessary since I was learning rsync and discovering which files/sub-folders and types I really wanted to copy or not. Many changes (from misunderstanding rsync) I needed to propagate with contextual changes.

My objectives were to facilitate both data recovery (using the internal backup) from system failure, and some operational continuity (using the external backup) during repair.

Questions:
1. What is supposed to happen to a /media/user/device folder when the rsync command terminates normally?
2. What is supposed to happen to a /media/user/device folder when the script terminates normally?
3. What is supposed to happen to a /media/user/device folder when the device is unmounted?
4. Do those actions get prevented when the root goes out of space? Low storage warning? Or Out?
5. At what fs% is the warning issued?
6. With the device unmounted is it safe (as su) to delete that device from /media?

Granted that my /home folder grows over time (I collect interesting data) (and that’s slow enough to manage by eye). And I have been lax in backing up, so maybe one of the rsyncs had to write more changed files than I had root space for with the script process. Does that mean I need to monitor root freespace with a grep / df command after each rsync command and terminate the script (how?) at that point?

TIA,
Kurt

User avatar
Pierre
Level 19
Level 19
Posts: 9173
Joined: Fri Sep 05, 2008 5:33 am
Location: Perth, AU.

Re: using rsync w/ external /media eats root storage

Post by Pierre » Sun Sep 08, 2019 9:26 pm

the program TimeShift is an backup program for your System Files - only
- it does Not back-up Your Data

thus you should use the program Back-up Tool for Your Data back-up.


MODERATOR NOTE:
do refrain from attacking other Forum Users, as you will attract the attention of the Forum.Administration.Team
& that could be Fatal for your Forum Membership Account.
Image
Please edit your original post title to include [SOLVED] - when your problem is solved!
and DO LOOK at those Unanswered Topics - - you may be able to answer some!.

rene
Level 12
Level 12
Posts: 4312
Joined: Sun Mar 27, 2016 6:58 pm

Re: using rsync w/ external /media eats root storage

Post by rene » Mon Sep 09, 2019 12:37 am

kurtking wrote:
Sun Sep 08, 2019 3:59 pm
Questions:
The questions imply you not being clear on this, and I am afraid my earlier reply is still the most directly useful manner in which I can explain things. Allow me to quote myself:
A mount point is just a directory; when you copied to under said directory when nothing was in fact mounted on it it just did exactly that: copy to under said directory, hence, onto the filesystem on which said directory lived, hence, the root filesystem.
That is, other than some minimal space to keep the system up in the first place, not any free space on your root fs is required for a backup to an external drive, using rsync or whatever else you may care for. When the external drive's plugged in its fs is (should be, rather) mounted on the directory /media/kurt/659186ea-07f8-4dea-94c2-eca5cba35de8 and any location under that directory refers to a location on said external drive's fs. Your rsync command copies only to the external drive's fs, nowhere else.

If the external drive's fs had in fact been mounted, that is. The mountpoint is as said a mere directory, and when you ran the copying script when it was not it just copied to under that directory which at that point is a mere directory on your root fs. All this is independent of rsync, cp, or whatever else you may use to do the copying. I.e.,

1/2. Nothing specific.
3. The "/media/user/device" folder should show as empty with the device unmounted. Should even disappear if the drive is not connected or switched on: that mountpoint is dynamically created and destroyed by the system so as to mount the external drive's fs on.
4. Not in a fundamental sense; as long as the system is not all the way down on free space to what is considered minimal to keep it chugging along in the first place the fact that no space on the root fs is required to copy to the external drive's fs means there's no connection.
5. 5% but don't bother even knowing that; it's as explained fully secondary to anything this issue is about.

Only at 6 are we at the the issue itself. Yes, if you with the external drive disconnected make sure that anything that is then still under /media/kurt/659186ea-07f8-4dea-94c2-eca5cba35de8 is something that is either already backed up to the external drive or something still available from wherever you backed it up from, you can simply delete the entire folder "659186ea-07f8-4dea-94c2-eca5cba35de8" from under /media/kurt. It should as explained not be there with the external drive unconnected or switched of.

It's very likely that a backup just ran when the drive wasn't plugged in or, by the way, when "659186ea-07f8-4dea-94c2-eca5cba35de8" wasn't in fact its automatic mountpoint under /media/kurt any longer due to you reformatting the external drive (giving its fs a different UUID) or through giving it a label. That is, verify that mountpount IS still the correct mountpoint by simply looking in /media/kurt after connecting the drive and after having cleaned it up first as per above, but otherwise, just make sure that the external drive is in fact mounted on the expected location when you run the backup script.

Post Reply

Return to “Software & Applications”