[SOLVED] Failure to clone with dd

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
Post Reply
Thermoman
Level 2
Level 2
Posts: 53
Joined: Sat Dec 07, 2013 6:04 pm

[SOLVED] Failure to clone with dd

Post by Thermoman »

My most recent hard disk failure (I think the third in a little more than fifty years) persuaded me that it was time to retire my ageing Mint Maya computer and to move up to the latest supported Mint release. The availability of modestly-priced recycled computers, with the latest Mint version pre-installed, makes changing hardware and software together an attractive option for me. Trying to install Linux myself has proved to be a lot more difficult, in the past, than I was led to believe, so I acquired a new (recycled) computer with Mint Sylvia pre-installed.

Experience has shown the importance of regular back-ups, for which I sequentially clone the system disk onto three external hard disks of the same size – 80Gb in the case of the Maya computer. This is more than adequate for my needs. I use no more than 20Gb. The Linux dd command has allowed me to clone cyclically, and apparently faultlessly, for a number of years past, using the familiar grandfather-father-son régime.

A potential problem with the replacement computer was that it was originally offered with a 160Gb disk. In order to avoid compatibility problems, and to continue to use my existing cloning disks, I asked for this to be replaced by an 80Gb disk, which was readily done for no extra charge.

Commissioning the new computer at first appeared to be going well. I connected the most recent Maya clone as an external USB device, which was recognized and mounted immediately, allowing all my application files to be transferred. It all worked perfectly until I tried to reverse the data flow – ie implement the cloning back-up procedure on the replacement computer. I mounted one of the older Maya back-ups with the intention of overwriting it. The dd command appeared to be working normally for the usual 45 minutes or so but then failed to terminate. After allowing significantly more time, to see if the then apparently inactive cloning procedure would end, it had to be manually aborted, leaving the disk corrupted. fdisk then showed that, whilst the sdb1 and sdb2 partitions were apparently accurate facsimiles of the sda1 and sda2 partitions, sdb5 appeared to contain an enormous amount of data and bore no resemblance to sda5. The result was the same when I attempted to clone to an initially blank, unpartitioned, disk. A suggestion that partitioning may be the cause of the problem therefore seems unlikely. The Linux ‘dd’ command does not appear to be affected by the initial status of the target disks, which, as far as I can judge, are apparently partitioned automatically, if necessary, or not if the appropriate structure is already present.

Valid clones can continue to be made from the original Maya computer. After the most recent back-up was employed to install the application files and data onto the Sylvia machine, the associated disk was used to replace the failed system disk on the the Maya computer. Being bootable, virtually by definition, this immediately restored the Maya machine to normal operation from more or less where it had left off. When a Maya clone is connected to the Sylvia computer, however, whilst it appears to be fully accessible that is effectively restricted to read-only mode.

What I need to determine, and hopefully fix, is why cloning operations on the Maya computer, using the dd command, are invariably successful, whilst the same procedure fails on the Sylvia computer, even though carried out in an almost identical environment using virtually the same hardware. As a ‘newbie’, however, I now find myself well out of my depth.

Perhaps there is a clue in the word ‘virtually’. Whilst the Maya system disk is of the same size and manufacture as the three back-up disks, that is not strictly true of the Sylvia system disk, nor of the replacement disk for the clone that has taken the place of the failed Maya system disk. Whilst all the disks are notionally 80Gb in size they are not now all of the same manufacture. I noticed that, when interrogated by fdisk or referenced in the dd report, very slight size differences are apparent – so slight that I assume them to be negligible but is this so? Another, possibly also irrelevant, detail is that the Sylvia computer originally featured a 160Gb system disk. The extensive error message resulting from the manual abortion of the cloning operation made mention of a possible code page error. Could there perhaps be a setting somewhere which is indicating to dd that it is performing with differently-sized disks? On the other hand, is there a switch that can be added to the dd command to persuade it to stop once the valid data has been cloned?
Last edited by Thermoman on Sun Aug 12, 2018 9:22 am, edited 1 time in total.

gm10
Level 20
Level 20
Posts: 10873
Joined: Thu Jun 21, 2018 5:11 pm

Re: Failure to clone with dd

Post by gm10 »

dd the way you seem to be running it creates an exact copy of one block device onto another. If your target is smaller, then yes, that would matter (I never tried to write an image to a smaller target so I don't actually know if that's the error you are seeing). Also you cannot make an exact copy of a partition while it is in use, your description left me wonder whether that's not what you are doing though.

You could consider other backup solutions like Timeshift (rsync) which backup only your files, not every sector on your drive.

Mute Ant
Level 14
Level 14
Posts: 5134
Joined: Tue Sep 03, 2013 7:45 pm
Location: Norfolk UK

Re: Failure to clone with dd

Post by Mute Ant »

All the processes of a running Mint are in RAM because the CPU simply can't access data that isn't connected to the address-bus.

In Mint each mounted file-system is allowed a maximum of 20% of the machine's RAM to temporarily cache permanent changes. If a cached change persists for 30 seconds, the data is physically sent to the storage device. So any attempt to command dd from or to a drive with mounted file-systems is not neccessarily a clone, the destination can differ by the amount of unwritten activity cached in RAM.

For a large amount of data, 30 seconds is a trivial fraction of the total backup time, during which any write to the source-drive might not get to the destination-drive.

To make your dd backup work every time, even from a running Mint, you should do the REISUB sequence and insert your backup as an extra activity...
  • REISU(backup)B
    o Enable all the SYSRQ codes with command sudo sysctl kernel.sysrq=1
    o Get your terminal ready to do your backup command.
    o Prepare for a safe software reboot with the keypress sequence Alt+SysRq+(R+E+I+S+U)
    o Do your backup command.
    o Reboot with the final keypress Alt+SysRq+(B)
...or if a reboot is undesirable, remount the file systems with command sudo mount -a.

Command dd was originally for much smaller drives but the technique is still valid in 2018, making a 'forensic' copy of a unique drive, especially drives with encrypted data, where you can't wait for the owner to 'remember' the password. It's low-level and idiotically obedient though, it tries to do exactly what you tell it to do, without helping and without caring about the consequences.

I prefer the shell built-in command cp (dd is an optional extra) but it's just as hazardous if you get it wrong...
Alt+SysRq+(R+E+I+S+U)
sudo cp /dev/sdX /dev/sdY
Alt+SysRq+(B)
While you're waiting, read the free novel we sent you. It's a Spanish story about a guy named "manual".

Thermoman
Level 2
Level 2
Posts: 53
Joined: Sat Dec 07, 2013 6:04 pm

Re: Failure to clone with dd

Post by Thermoman »

I am very grateful for the replies but I must admit that, from level 1, a level 13 response looks a bit daunting. Nevertheless it raises a couple of interesting issues that I would like to explore before getting into too much complexity.

A nagging difficulty that remains, is to try to understand why dd worked faultlessly for years on a computer with 2Gb of RAM, cloning the 80Gb hard disk onto a series of external back-up hard disks of the same size, when it will not do so on a replacement computer, with 3Gb of RAM, cloning to the self-same 80Gb drives, particularly when the replacement computer system hard-drive is of the same size as its predecessor and contains virtually the same software/data. I might have understood it better if things had been the other way around. My level 1 brain suggests that there must be some identifiable difference between the two systems to cause this. It would be nice to find it. In the mean time there appear to be two options worth exploring before what was a relatively trivial operation is turned into a more complex one.

Reference to the rôle of RAM/cache leads me to wonder whether there is any mileage in adding more memory. The replacement computer is rather modestly supplied with 3, of the possible 8, Gbs but, as already mentioned, the previous computer managed happily with 2Gb. Nevertheless, another problem elsewhere had already led me to wonder whether more RAM was needed on the new machine. Small spreadsheets, under LibreOffice 5, are saved in roughly the same time as those on the previous computer running an earlier version of LibreOffice, whereas larger spreadsheets – typically up to a couple of thousand rows of perhaps 20 columns – now take an inordinately extended time to save. Literally minutes elapse before the speed-bar even appears. There seems to be a buffering bottleneck somewhere. This cannot, surely, be a new feature? If so, even moderately large spreadsheets could render LibreOffice unusable. Is the bottleneck located in memory?

My second thought relates to the block size setting in dd. Could this have any relevance? Admittedly this is a matter than I can test rather more easily than RAM size. The previous setting of bs=512K was selected by trial and error. I don’t remember seeing much benefit from a larger block size but cloning became progressively, and significantly, slower as it was reduced. I am not sure what the default is but, if bs was not set, the associated cloning operation became very slow. Maybe, however, there might be some benefit in slowing it down a bit.

User avatar
catweazel
Level 19
Level 19
Posts: 9910
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Failure to clone with dd

Post by catweazel »

Thermoman wrote:
Fri Aug 10, 2018 7:43 am
I am very grateful for the replies but I must admit that, from level 1, a level 13 response looks a bit daunting. Nevertheless it raises a couple of interesting issues that I would like to explore before getting into too much complexity.
Translated it means, don't try to clone an active disk.
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.

gm10
Level 20
Level 20
Posts: 10873
Joined: Thu Jun 21, 2018 5:11 pm

Re: Failure to clone with dd

Post by gm10 »

OP I thought you had said the new disks weren't exactly the same size?

The ideal block size ultimately depends on how the drive operates internally. Modern drives read more than 512 bytes in one go no matter how many you request. Your drives must be quite ancient to be that small so I won't guess at their capabilities, but 512 bytes is the slowest option no matter what. It won't affect the success of your cloning operation though.
catweazel wrote:
Fri Aug 10, 2018 7:48 am
Translated it means, don't try to clone an active disk.
was I too subtle? ;)
gm10 wrote:
Thu Aug 09, 2018 6:39 am
you cannot make an exact copy of a partition while it is in use

User avatar
catweazel
Level 19
Level 19
Posts: 9910
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Failure to clone with dd

Post by catweazel »

gm10 wrote:
Fri Aug 10, 2018 8:02 am
catweazel wrote:
Fri Aug 10, 2018 7:48 am
Translated it means, don't try to clone an active disk.
was I too subtle? ;)
Not at all. The OP and Mute Ant's reply were tl;dr. Your post got lost in the glare :)
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.

Dyfi
Level 4
Level 4
Posts: 242
Joined: Fri Nov 27, 2009 7:25 am

Re: Failure to clone with dd

Post by Dyfi »

A running system can be successfully dd'd by using the "fsfreeze -f /" command prior to making the backup. This temporarily freezes the root file system which prevents any disk writing whist dd is running. When the backup has completed use "fsfreeze -u /" to unfreeze your file system.

Post edited - Typo corrected
Last edited by Dyfi on Fri Aug 10, 2018 9:48 am, edited 1 time in total.

User avatar
catweazel
Level 19
Level 19
Posts: 9910
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Failure to clone with dd

Post by catweazel »

Dyfi wrote:
Fri Aug 10, 2018 8:50 am
"fsreeeze -f /"
Don't try this at home, folks.
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.

Dyfi
Level 4
Level 4
Posts: 242
Joined: Fri Nov 27, 2009 7:25 am

Re: Failure to clone with dd

Post by Dyfi »

Typo error - the correct commands are "fsfreeze -f /" and "fsfreeze -u /"

# fsfreeze --help

Usage:
fsfreeze [options] <mountpoint>

Suspend access to a filesystem.

Options:
-f, --freeze freeze the filesystem
-u, --unfreeze unfreeze the filesystem

-h, --help display this help
-V, --version display version

For more details see fsfreeze(8).
#

User avatar
thx-1138
Level 7
Level 7
Posts: 1926
Joined: Fri Mar 10, 2017 12:15 pm
Location: Athens, Greece

Re: Failure to clone with dd

Post by thx-1138 »

A manual freeze of the root filesystem is dangerous and is almost never the right tool for the job.
^...Also sprach Zarathustra Eric Sandeen, 2017-07-25...(where Eric Sandeen stands for)...
Fedora project member and filesystem developer at Red Hat. Occupation: Senior Software Engineer, Red Hat Inc.
So, to re-iterate what catweazel said...
catweazel wrote:
Fri Aug 10, 2018 8:59 am
Don't try this at home, folks.
...maybe it's not of much concern trying it on... /home, but i'd certainly wouldn't recommend trying such at / (or back at home).... :wink:

Dyfi
Level 4
Level 4
Posts: 242
Joined: Fri Nov 27, 2009 7:25 am

Re: Failure to clone with dd

Post by Dyfi »

Each to their own method of backing up - the most important factor with linux is to learn to backup correctly. I use the fsfreeze command regularly as illustrated and my backups have never failed.

User avatar
catweazel
Level 19
Level 19
Posts: 9910
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Failure to clone with dd

Post by catweazel »

Dyfi wrote:
Fri Aug 10, 2018 10:02 am
Each to their own method of backing up - the most important factor with linux is to learn to backup correctly. I use the fsfreeze command regularly as illustrated and my backups have never failed.
That completely misses the point. Freezing / is dangerous at any time of the day.
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.

gm10
Level 20
Level 20
Posts: 10873
Joined: Thu Jun 21, 2018 5:11 pm

Re: Failure to clone with dd

Post by gm10 »

Agreed, it's just not worth taking the risk. If you need to clone your / root then boot into another partition first. Personally I've got a Tiny Core Linux image in my EFI partition, mainly as a comfortable recovery option in case I mess something up badly but you can certainly run your dd from there as well.

Thermoman
Level 2
Level 2
Posts: 53
Joined: Sat Dec 07, 2013 6:04 pm

Re: Failure to clone with dd

Post by Thermoman »

Now I'm totally confused! l think I had better retire from the fray and leave the big guns to battle it out.
I am bemused to learn that an active disk cannot be cloned with dd. I did it for about four years without a single failure. When my system hard disk eventually quit on the Maya computer I mounted the most recent Maya clone externally on the replacement Sylvia machine and transferred all my application files and structures - system restored! Then I took the back-up clone out of its caddy and used it to replace the failed system disk in the old Maya computer, which then booted up and carried on from where it had left off. Hey presto - bootable clone. I suppose it is a bit like the bumble bee. It fails to understand that it is theoretically unable to fly, so it carries on doing it. As a newbie, I had no idea that I could not clone an active disk so I too carried on doing it!

Finally, to clarify my remarks about disk sizes. All the hard disk involved in this discussion are notionally of the same size - ie 80GiB. But the word 'notionally' may hide a multitude of sins. Whilst some of the disks are of the same type and manufacture that is not true of all of them, the differences being expressed in small variations in some parameters, which might ultimately be crucial. Gparted shows failed clones to have valid sdb1 & sdb2 partitions but not sdb5 - the linux-swap partitions, which are absent - that is all except the 'bumble bee' clones. They are just that - clones.

It looks as though I will have to find another way of doing this job.

gm10
Level 20
Level 20
Posts: 10873
Joined: Thu Jun 21, 2018 5:11 pm

Re: Failure to clone with dd

Post by gm10 »

The size thing matters because you are not just cloning the data on your disks but every single sector on them, whether in use or not. So if your source disks has theoretical 1000 sectors and your target disk has theoretical 999 sectors then your image can't be fully written. As I said above, I don't know if that's the reason you having trouble with the process, I don't know what dd says in that situation since I never tried it.

Regarding cloning active partitions - the problem isn't that you cannot do it, the problem is that you don't get a guarantee that the content on the partition won't change while you're doing it. So you might write partially old and partially new data to you backup destination, resulting in a partially corrupt backup. To avoid that you can lock the disk down to prevent any changes to it, but that carries the risk of crashing your system or otherwise making it unstable, that's our little side-discussion above.

Thermoman
Level 2
Level 2
Posts: 53
Joined: Sat Dec 07, 2013 6:04 pm

Re: Failure to clone with dd

Post by Thermoman »

Having reread and digested all the comments, I have decided not to pursue this topic any further. Exactly why dd unfailingly appeared to create successful clones on my old computer, but staunchly refuses to do so on the new one, remains a mystery but the solution has become irrelevant. The killer observation was that I should not have been attempting it from an active system disk anyway. The reason now appears obvious but I would not have worked it out unaided. The trouble is that, if you try something and it appears to work, you give yourself a pat on the back and carry on, not considering whether you should be doing it. You only really start to examine the situation when things go wrong, as in this case.

Maybe dd was happy on the old machine because the target cloning disks and the computer system source disk were identical in terms of manufacturer, size and model number. This is no longer true on the new computer. All the disks are still of the same nominal size – 80Gb – and Gparted shows them to have exactly the same number of heads, sectors per track, cylinders and total sectors but they come from three different manufacturers. One even displays a slightly different size. Whether that is the cause of the problem will probably never be known but it no longer really matters. If, as was pointed out, what was produced from the old computer were not true clones anyway, then, even though conveniently bootable, could, or should, they be regarded as suitable replacements for a failed system hard disk? The intervals between hard disk, or computer, failures tend to be long and technology meanwhile moves on. In reality it has to be admitted that there is rarely a rôle for an old clone in a new software/hardware environment. They might be useful, to some extent, as backup file repositories but there are other ways of doing that and these I am currently pursuing, with as yet mixed success.

In the mean time, thank you everyone for your help. I have learned the error of my ways and will now mark this thread as [SOLVED].

Post Reply

Return to “Newbie Questions”