Large difference in size of a Foxclone disk image and output of df[CLOSED]

[Daisuke]

I am using Foxclone to periodically save disk images of /dev/sda (4 partitions created during the LM install process). Foxclone has glitched twice when I accept the default compression option so I have just been backing up uncompressed for now. Disk images can be fairly large, so I pruned both / and /home recently by moving files to other drives to be backed up by another means. I am trying to figure out why Foxclone .img files are much larger than what would be predicted by the output of df.

Code: Select all

~$ df -ht ext4
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        46G   17G   28G  38% /
/dev/sdf1       3.6T  100G  3.4T   3% /data/backup
/dev/sda4       1.8T  5.2G  1.7T   1% /home

Code: Select all

/data/backup/DiskImages/LinuxMint/LM-22-05-18$ ls -lh
total 52G
-rw-r--r-- 1 root root  490 May 18 19:53 20220518-uncompressed.backup
-rw-r--r-- 1 root root 1.0M May 18 19:48 20220518-uncompressed.grub
-rw-r--r-- 1 root root   89 May 18 19:53 20220518-uncompressed.note.txt
-rw------- 1 root root 2.2M May 18 19:48 20220518-uncompressed.sda1.img
-rw-r--r-- 1 root root  615 May 18 20:04 20220518-uncompressed.sda1.img.chk
-rw-r--r-- 1 root root  671 May 18 19:48 20220518-uncompressed.sda1-log.txt
-rw------- 1 root root  17G May 18 19:49 20220518-uncompressed.sda2.img
-rw-r--r-- 1 root root  621 May 18 19:59 20220518-uncompressed.sda2.img.chk
-rw-r--r-- 1 root root  678 May 18 19:49 20220518-uncompressed.sda2-log.txt
-rw------- 1 root root  35G May 18 19:52 20220518-uncompressed.sda4.img
-rw-r--r-- 1 root root  626 May 18 20:03 20220518-uncompressed.sda4.img.chk
-rw-r--r-- 1 root root  684 May 18 19:53 20220518-uncompressed.sda4-log.txt
-rw-r--r-- 1 root root  310 May 18 19:48 20220518-uncompressed.sfdisk

The size of /dev/sda2 reported by df agrees exactly with the Foxclone file 20220518-uncompressed.sda2.img (both are 17GB),
But the size of /dev/sda4 reported by df is 5.2GB while the corresponding Foxclone file 20220518-uncompressed.sda4.img is 35GB.

This is not to cast any shade on Foxclone since it must be using a different view of the used blocks than df is using. So my question is this. Can I do anything to release the blocks that no longer contain data as viewed by Foxclone and come away with smaller disk images?

[xenopeek]

Noting that I don't use disk imaging (I use automated incremental backups for personal files, like Deja Dup or Pika Backup, and don't bother with system backups because reinstalling takes 5 minutes max) but looking at the Foxclone manual https://www.foxclone.com/download/foxcloneV50.pdf it says it is simply a front-end for partclone https://partclone.org/. It also says it uses df to determine the number of blocks in use. I don't know but maybe try the partclone.info command on that 20220518-uncompressed.sda4.img file to see what it says is in it https://partclone.org/usage/partclone.info.php.

[Daisuke]

looking at the Foxclone manual https://www.foxclone.com/download/foxcloneV50.pdf it says it is simply a front-end for partclone https://partclone.org/. It also says it uses df to determine the number of blocks in use. I don't know but maybe try the partclone.info command on that 20220518-uncompressed.sda4.img file to see what it says is in it https://partclone.org/usage/partclone.info.php.

Yes, the manual is excellent and even though I have read it through, I was mainly focused on the procedural sections. Thanks for pointing this out. I tried your suggestion and installed partclone. Here is what I got, but it simply confirms that this is the info df provided when run from the Foxclone Live USB.

Code: Select all

$ sudo partclone.info ./20220518-uncompressed.sda4.img
Partclone v0.3.11 http://partclone.org
Unknown mode
File system:  EXTFS
Device size:    1.9 TB = 466161664 Blocks
Space in use:  36.8 GB = 8978494 Blocks
Free Space:     1.9 TB = 457183170 Blocks
Block size:   4096 Byte

image format:    0002
created on a:    64 bits platform
with partclone:  v0.3.11
bitmap mode:     BIT
checksum algo:   CRC32
checksum size:   4
blocks/checksum: 256
reseed checksum: yes

I tried running df as sudo but it made no difference. Unfortunately, AndyMH must be busy right now. So this will have to wait until he can shed some light on this puzzle.

Code: Select all

:~$ sudo df -ht ext4
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        46G   17G   28G  38% /
/dev/sda4       1.8T  5.3G  1.7T   1% /home
/dev/sdf1       3.6T  102G  3.3T   3% /data/backup

In any case, I am grateful for your help. Thanks.

[Daisuke]

Since this post is not getting much interest, my next idea will be to try Rescuezilla to see the size of the image which it creates for the home partition.

[motoryzen]

Since this post is not getting much interest, my next idea will be to try Rescuezilla to see the size of the image which it creates for the home partition.

I don't blame you if you're frustrated and time is of necessity in solving your situation. AndyMH is the actual software dev of Foxclone. He might be fairly busy lately, but usually he is fairly fast to chime in in Foxclone related or other backup situations. Thus I don't see why he'd be irritated if you sent him a private message asking for him and pointing him to this url aka your post. *shrugs*

[ricardogroetaers]

Code: Select all

~$ df -ht ext4
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        46G   17G   28G  38% /
/dev/sdf1       3.6T  100G  3.4T   3% /data/backup
/dev/sda4       1.8T  5.2G  1.7T   1% /home

Code: Select all

/data/backup/DiskImages/LinuxMint/LM-22-05-18$ ls -lh
total 52G
-rw------- 1 root root  35G May 18 19:52 20220518-uncompressed.sda4.img

But the size of /dev/sda4 reported by df is 5.2GB while the corresponding Foxclone file 20220518-uncompressed.sda4.img is 35GB.

Take a good look at the output of the df and ls commands and the observation will answer you.

Observation:
I do not know and do not use Foxclone.
I don't know how Foxclone works and what options are available to create the image file.
The author of the program can clarify.
In my humble opinion, I suppose the copy options used for sda2 were different from the options used for sda4.

[Daisuke]

Take a good look at the output of the df and ls commands and the observation will answer you.

I did take a good look. I see a difference of 5.2G to 35G (i.e. GB) so a factor of 7 larger for the /home partition. /dev/sda4 is my ext4 /home partition.

You are too kind, but I think you have overestimated my limited abilities. The settings for Foxclone apply to all partitions being backed up. And according to the online documentation, the process is the same.

So I would be very grateful if you will tell me what you see, since you see more than I do!

[Daisuke]

I had another idea for trying to understand the large difference in the size of the image of my /home partition created by Foxclone and the size reported by df.
First, I booted into a Linux Mint Live USB, edited /etc/fstab to add the mount point, and ran mount -a which mounted /dev/sda4 to /mnt/home1. Then I tried df here. These are the results.

Code: Select all

mint@mint:/etc$ sudo mount -a
mint@mint:/etc$ cd /mnt/home1
mint@mint:$sudo df -ht ext4 /mnt/home1
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       1.8T  5.5G  1.7T   1% /mnt/home1

Then I booted into the Foxclone USB and took a few more screen shots which are below.
In both cases df reports a smaller used space than what Foxclone is backing up. So I am wondering if the problem is that the disk needs to be defragmented since the used space may be spread over a very large section of the disk and Foxclone detects this. When I have time, I will start a new thread related to that possibility.

[ricardogroetaers]

Take a good look at the output of the df and ls commands and the observation will answer you.

I did take a good look. I see a difference of 5.2G to 35G (i.e. GB)....

As I said, I don't know Foxclone, so I don't know how it works.
I still use the good old Image for DOS.

Please note:

- according to the df program, sda2 is 46 GB in size with 17 GB used. Its copy by Foxclone is 17 GB (same size as the space used by the data). Probably the information about the partition does not take up much space.

- sda4 is 1.8 TB (1,800 GB) in size (39 times the size of sda2) and only 5.2 GB used.
Your copy by Foxclone is 35 GB. The information about the partition probably takes up a lot of space.

I don't know if Foxclone contemplates the options:
- copy unused sectors (optional)
- copy data from hibernation and paging (optional)
- specify the maximum size of the copy file. If 1 file is not enough, the program will create several "chain" files.

Translated with www.DeepL.com/Translator (free version)

[Daisuke]

I don't know if Foxclone contemplates the options:
- copy unused sectors (optional)
- copy data from hibernation and paging (optional)
- specify the maximum size of the copy file. If 1 file is not enough, the program will create several "chain" files.

These are good points. I can answer two of these according to the Foxclone v50 documentation found here.
It only copies used sectors (or blocks).
It does not specify a maximum size of the copy file.
I am not sure about data from hibernation and paging.
I do not know and could not find online a way to check if hibernation is enabled, but I read that the default mode it to not enable it and enabling it seems to be a bit complicated. I have not enabled it on my system.

[Daisuke]

In absence of support for Foxclone, I tried Rescuezilla. It does not glitch on compression so I will just switch to that. I will mark this closed rather than solved.

[Termy]

It's possible that partclone(8) is for some reason unnecessarily cloning free space, which is then being compressed, but I'm not familiar with that tool. I've used CloneZilla for over a decade in Windows and Linux with absolutely perfect success with several restores and backups over different filesystems, partitions, and drives. That being said, CloneZilla does use partclone(8) as well, but it also does seem to do a bunch of other magic. YMMV.

[Daisuke]

I've used CloneZilla for over a decade in Windows and Linux with absolutely perfect success with several restores and backups over different filesystems, partitions, and drives. That being said, CloneZilla does use partclone(8) as well, but it also does seem to do a bunch of other magic. YMMV.

Rescuezilla is an easy-to-use GUI front-end on Clonezilla whose interface is a rather daunting, although I have used it myself without problems a few times when attempting a P2V conversion from a Windows 7 system. (That did not actually work out!) For fast and easy, Rescuezilla does the backup job nicely for my plain vanilla LM system. It also comes packaged with a friendly set of other useful tools including a web browser. The compression I got on that /home partition was impressive too.

I had to look up YMMV! Thanks for the good advice!

[Termy]

[...]

YMMV is an oldie. Didn't know Rescuezilla was based on CloneZilla, but I guess the name gave it away. It sounds good. I remember how daunting CloneZilla was when I first used it. I was so worried about doing something wrong and blasting away previous data.

[Daisuke]

I remember how daunting CloneZilla was when I first used it. I was so worried about doing something wrong and blasting away previous data.

Yes, I had the same feeling. Rescuezilla is much easier, but I still take a lot of care in choosing the drive to which the restore will be applied. I have used it and Foxclone both. While Foxclone is more polished, it has crashed when creating a compressed image. So far, Rescuezilla has not. And the compressed backup of 60GB is only 12GB, so a significant space saver. I finally had time to test the compressed Rescuezilla backup, and it worked perfectly. So now I can just maintain things with BackInTime and TimeShift. I will feel much more confidant doing updates/upgrades with this comprehensive backup system in place.

I also have been trying to figure out why partclone, which is the command used to make partition images in both Rescue/Clonezilla and also in Foxclone, is copying 7 times as much data as df shows in my /home partition. My current theory is that the SSD device driver must be trying to balance the writes by spreading out files over many additional blocks. If a file is small, it can take an entire 4K block (which is what I think the block size is on an ext4 partition) even though it may only hold a few hundred bytes. There is also the possibility that there is some amount of space reserved for root. To find how much is reserved I need to run

Code: Select all

tune2fs -l

and I am leery of running this command since I am not sure if it is OK to do on a mounted file system. So I will let it go for now because compression solves the problem. I am not really sure if my speculation/analysis is correct And I would like to have a conclusive answer. But one never knows if someone who knows a lot about partclone or filesystems will come along one day and clear up the matter. I live in hope

[antikythera]

7x as much data? Possibly then partclone is doing sector by sector backup of that partition rather than just the files.

[Daisuke]

Possibly then partclone is doing sector by sector backup of that partition rather than just the files.

You have the right of it. partclone.org describes its function this way: "Partclone provides utilities to save and restore used blocks on a partition..." This is in keeping with the idea of a clone, or identical copy of a partition, which also includes the partition metadata in addition to the file contents.

I am not completely sure of the difference between sector and block though. For ext4 partitions the block size can vary but on my system, it is 4K bytes.

Code: Select all

$ sudo blockdev --getbsz /dev/sda4
4096

BTW, your name, antikythera, is very cool!