mount: [image.img]: failed to setup loop device: Operation not permitted (SOLVED)

[linx255]

Mint 18 Mate 64
4.15.0-34-generic

Hello, for years I've been using 'dd' command to create .img files of partitions, and restoring them by mounting them and rsyncing them to the target partition.

I've never encountered any problems mounting my .img files until today when I noticed my newest .img file does not mount but my second oldest .img file does (yesterday I recall having no problem mounting as either ro or rw):

sudo mount -o loop,ro newest.img /temp/ produces the error:

mount: cannot mount /dev/loop0 read-only


sudo mount -o loop,rw newest.img /temp/ produces the error:

mount: /data/newest.img: failed to setup loop device: Operation not permitted


sudo mount newest.img /temp/ produces the error:

mount: /data/newest.img: failed to setup loop device: Operation not permitted


Now when I try mounting the 2nd newest file (and all older):

sudo mount -o loop,ro 2nd_newest.img /temp/ WORKS!


sudo mount -o loop,rw 2nd_newest.img /temp/ produces the error:

mount: /data/2nd_newest.img: failed to setup loop device: Operation not permitted


sudo mount 2nd_newest.img /temp/ produces the error:

mount: /data/2nd_newest.img: failed to setup loop device: Operation not permitted


The 'dd' command I've always used to create the .img files is:

sudo dd if=/dev/sda$partition of=/data/newest.img


The 'mount' commands I've always used to mount the .img file (for rsyncing to another mounted partition) is:

Code: Select all

sudo mkdir /sda$partition
sudo mount /dev/sda$partition /sda$partition
sudo mount /data/newest.img /temp

The 'rsync' command I've always used to restore the mounted .img files is:

sudo rsync -advAX --progress --delete /temp/ /sda$partition/


It seems somehow the .img file I'm creating is unmountable ( old ones work fine ). 'e2fsck' returned no errors on any partitions and loop devices appear to be all present and working. It just doesn't like my new .img files.

Without the ability to create and restore partition images I'm having an IT nightmare and my whole system is at risk of destabilizing.
It makes no sense unless some kind of system update changed the way my image capturing works.

Clues, please? Thanks. I've been troubleshooting and researching these errors for hours and failed to dig up anything useful.


UPDATE: I just discovered this mount command works for my newest.img:

sudo mount -o ro,norecovery,loop /data/newest.img /temp

The first time I mounted it this way it would not unmount (was busy) and I had to use 'umount -l'. Now it mounts and unmounts with the norecovery option and does not get 'busy'.

However, this does not explain why the newest.img requires 'norecovery' and the older ones do not.

[rene]

sudo mount -o loop,ro newest.img /temp/ produces the error: mount: cannot mount /dev/loop0 read-only

That part as well as your own update is consistent with a filesystem needing journal recovery ...

sudo mount -o loop,rw newest.img /temp/ produces the error: mount: /data/newest.img: failed to setup loop device: Operation not permitted

... and even though that part isn't, I believe this second part to be not directly related to the first.

As to the first part, consistency with a filesystem needing journal recovery:

Code: Select all

rene@t5500:~$ dd if=/dev/zero of=clean.img bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 0,945915 s, 1,1 GB/s
rene@t5500:~$ mkfs -t ext4 clean.img
mke2fs 1.44.1 (24-Mar-2018)
Discarding device blocks: done                            
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 5019e901-de26-404e-8a85-6fe16ba5f86f
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
rene@t5500:~$ sudo mount -o loop,rw clean.img tmp/
rene@t5500:~$ echo foo | sudo tee tmp/foo
foo
rene@t5500:~$ cp clean.img unclean.img
rene@t5500:~$ sudo umount tmp/
rene@t5500:~$ sudo mount -o loop,ro unclean.img tmp/
mount: /home/rene/tmp: cannot mount /dev/loop0 read-only.

That is, same message as in your case. That second one I'm not reproducing:

Code: Select all

rene@t5500:~$ sudo mount -o loop,rw unclean.img tmp/
rene@t5500:~$ sudo umount tmp/
rene@t5500:~$ sudo mount -o loop,ro unclean.img tmp/
rene@t5500:~$ sudo umount tmp/

That is, after mounting rw one time, ro is fine again (because the journal has been recovered).

However, I (very) vaguely recall some issue with loop devices at around Mint 18 times, which I see you use, and would as such advise an fsck on newest.img; this fixes the ro mount here:

Code: Select all

rene@t5500:~$ sudo mount -o loop,rw clean.img tmp/
rene@t5500:~$ echo foo | sudo tee tmp/foo
foo
rene@t5500:~$ cp clean.img unclean.img
rene@t5500:~$ sudo umount tmp/
rene@t5500:~$ sudo mount -o loop,ro unclean.img tmp/
mount: /home/rene/tmp: cannot mount /dev/loop0 read-only.
rene@t5500:~$ e2fsck -f unclean.img 
e2fsck 1.44.1 (24-Mar-2018)
unclean.img: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
unclean.img: 12/65536 files (0.0% non-contiguous), 12956/262144 blocks
rene@t5500:~$ sudo mount -o loop,ro unclean.img tmp/
rene@t5500:~$ 

That is, what happens after e2fsck -f newest.img?

I will say by the way I'm puzzled as to why you not simply dd the images back into place if you dd'ed them from the device in the first place. I.e., why the mount and rsync stuff if you could also just sudo dd if=newest.img of=/dev/sd<C><N> for the correct C and N?

[linx255]

That part as well as your own update

Is there a specific update, i.e. Update Manager, or are we just talking Mint version?

filesystem needing journal recovery

Ok, I have never had this happen. Good to know! I have no idea what caused this, but now I need to incorporate commands into my image maintenance scripts to automatically detect and fix journal problems.

However, I (very) vaguely recall some issue with loop devices at around Mint 18 times

Oh my, I didn't realize that such staple commands as mount changed so radically from 18 to 19. I haven't had time to upgrade to 19... Last time I tried it totally screwed up my theme/colors and there was apparently no way to maintain the same appearance I had with 18, and it was really unusable. So 19 got put on the back burner, and I got busy working on other things. ( If there's a method to maintaining the exact appearance through the upgrade I'd be interested to know! I think I posted about that. )

When I got time later today I will try to repair journals, or do fsck/e2fsck on newest.img. So echo foo | sudo tee tmp/foo will damage the journal of clean.img?

I'm deploying the .img files to different partitions so they contain the same data, for redundancy.

The problem with dd is it also grabs all the inodes and what ends up happening is the inodes on each partition deployed to are identical and control the same files, so it effectively breaks the isolation between the partitions, and creates a massive, confusing Siamese twin file system/ total wreck.

Rsync, on the other hand, creates new inodes every time any file is created/or at least uses the existing one which isn't normally identical to any other.

Also, I haven't verified it but logic dictates that dd grabs along deleted data in the partition and rsync only grabs non-deleted files. If someone can verify this let me know; I haven't tested forensic software on it yet, but it occured to me that syncing the image back with rsync probably fails to capture the deleted data, making file recovery either impossible or impaired due to the deleted data potentially being overwritten, but that's better than having an inode catastrophe ( I posted about this too )! Or you might want to make deleted data unrecoverable. I'm trying to figure out different ways to manage partition data. For now I'm alright with rsync because I have a fairly robust backup system so forensic recovery is not really needed, and dd is only useful for a partition for which there is no duplicate.

[rene]

Is there a specific update, i.e. Update Manager, or are we just talking Mint version?

Ah, no, I was just talking about the "UPDATE:" you added to your own message.

What I expect happened is simply that you forgot to cleanly unmount the source partition before grabbing "newest.img" from it; i.e., basically the same manner in which I simulated things. because, no, the only thing that "echo" did was write something to the filesystem. It's creating the image (cp clean.img unclean.img, in the simulation) before cleanly unmounting when some data IS still in flight that creates a filesstem with a journal needing recovery.

Also, no, there is/was no fundamental change wrt. mount between 18 and 19; it's just that I recall some issue/buglet with loop devices around that time. As mentioned, very vaguely recall and am not finding it again now, but I'd until the fsck was attempted still consider the also failing rw mount a separate issue.

The "identical inodes" thing doesn't seem to make overly much sense to me (identical FS UUID does but you can change that after the fact) but I'll assume there's a reason. Certainly it IS indeed the case that the images of the source partition include deleted data and that simply dd'ing them back would as such have the destination partitions include it as well. Yes, certainly rsync doesn't.

Anyways, I expect the issue will be solved after e2fsck -f newest.img and will then consist of the advise to not forget to unmount before grabbing an image.

[linx255]

e2fsck fixed newest.img and it mounts fine with:

Code: Select all

sudo mount newest.img /temp
sudo mount -o loop,ro newest.img /temp
sudo mount -o loop,rw newest.img /temp

What I was trying to explain about dd is that if you restore an image to two partitions, not only with the UUIDs will be identical, but the inodes of every file in each partition will be identical which means if you change a file on one partition those changes will be reflected on the other because the inodes in each of the partitions actually points to the same place.

When I first started experimenting with dd for managing file systems I started experiencing this and got frustrated when the two partitions seemed fused even though I had set unique UUIDs and fstabs after writing the images. Someone on the forum identified the problem and sure enough when I switched to rsync, problem solved. Of course, if the partitions are on two physically separate drives then dd could be used without this problem.

[rene]

Know what you were saying but that doesn't in fact make sense: inodes are a per fs resource. Perhaps you on both systems had the for example same fs mounted on /home, or was accessing through inter-device symlinks, or...

Anyways; good to know the problem's solved.

[linx255]

inodes are a per fs resource

I'm not sure I understand. Of course, two OS' on different partitions will generate different inodes as new files are created, but as I understand it, when using dd, the inode numbers of existing files are copied and still point to the same location on the hard drive. Then the OS has no idea it was cloned from dd, shrugs its shoulders, and carries on thinking all its existing inodes are unique.

This is consistent with the effects I experienced. I would login to one OS, make changes, and then see those same changes by logging into the other OS. Or I could make a change to a file in the file system in /dev/sda5, mount /dev/sda6 and see the same change there. I made sure each partition was set to a different UUID and that fstab pointed to those UUIDs ( with one /home residing in /dev/sda5 and another in /dev/sda6 ) to make sure each partition was actually separate from the others, and this had no effect.

It seems like another senior forum member actually tipped me off to this phenomenon; I'd have to look back to confirm. It seems this happens because the neither the OS nor dd were really intended to manage identical OS' on the same hard drive--probably an uncommon setup. If they were, then dd would have an option to regenerate inodes either when restoring an image.

If there is an expert (or if you are one) in this area I'd like to confirm my understanding.

And thank you for confirming the dd / rsync difference regarding deleted data persistence.

[rene]

[ ... ] as I understand it, when using dd, the inode numbers of existing files are copied and still point to the same location on the hard drive.

No, this is not the case. What inodes hold (directly or indirectly) are the numbers of the actual datablocks for a file, but per-fs: block N from the filesystem on /dev/sda1 is completely independent from block N from the filesystem on /dev/sda2. If you clone a filesystem onto a different partition then changes to file "foo" in block 1234 on the original filesystem have no effect whatsoever on file "foo" in block 1234 on the cloned filesystem and vice versa.

You explicitly claim to have experienced otherwise but you'd have to provide me with a reproducible example for me have a chance of telling you what the issue in fact is/was; it's not anything inherently to do with dd; duplicating inodes is fine; they're per-fs. As said, one thing I can think of is that you were in fact from both systems mounting one and the same filesystem on e.g. /home in which case clearly changes by both are changes to said same fs but imagination otherwise fails me. If you were told this was to be expected, "because of inodes" or otherwise, you were told wrong.

[linx255]

I stand corrected. I was not able to reproduce what I thought was my understanding of inodes. You're right, it doesn't change even if the inode #s are shared. In that case, I don't understand or recall what the real issue was. I'll dig up the original post. It just remember finding out that I could not use dd to maintain identical OS images because it caused problems, or so I thought. Somehow it was getting all mixed up and it did not have to do with grub or fstab as far as I recall. My bad, I don't know I formed this view, then!