UUIDs not unique? <SOLVED>

Forum rules
Before you post please read this

UUIDs not unique? <SOLVED>

Postby SkinnySeal on Tue Aug 27, 2013 1:24 pm

When I ran bkid I was shocked to get this duplicate UUID for my eSata connected external hard drives:

/dev/sde1: LABEL="Dionysus" UUID="803CC3A53CC39516" TYPE="ntfs"
/dev/sdf1: LABEL="Aphrodite" UUID="803CC3A53CC39516" TYPE="ntfs"

I installed MINT 15 yesterday, so I wanted to add these eSata connected drives to fstab. But as the UUIDs are identical for both of these hard drives, I don't see how that would be possible. My home PC has 2 eSata ports, so I purchases a 2nd eSata cable last week in lieu of a hard drive's previous USB2 connection.

It's no issue if I add these hard drives after I boot; they are treated as separate external hard drives. But I would prefer to have them load as part of the boot process.

Any ideas?
Last edited by SkinnySeal on Fri Aug 30, 2013 12:18 am, edited 2 times in total.
SkinnySeal
Level 1
Level 1
 
Posts: 34
Joined: Mon Jun 20, 2011 11:00 pm

Linux Mint is funded by ads and donations.
 

Re: UUIDs not unique?

Postby nomko on Tue Aug 27, 2013 1:36 pm

What's the result when you do:
Code: Select all
sudo blkid
?
Still the duplicate UUID?
ASUS P5KPL-AM SE*Intel Core2 Quad CPU Q8200*4 GiB ram*nVidia GeForce 9500GT*Realtek RTL8101/8102E*Optiarc DVD-RW AD-7200A*HDD:Samsung HD501LJ/HD103SI*Sitecom X5-N600 Router (WLR-5100)

My personal website
User avatar
nomko
Level 8
Level 8
 
Posts: 2246
Joined: Sat Feb 25, 2012 7:28 pm
Location: The Hague, The Netherlands

Re: UUIDs not unique?

Postby viking777 on Tue Aug 27, 2013 1:42 pm

Mount them with the label not the uuid it is a hugely superior method anyway.

Example from my fstab:

Code: Select all
LABEL=backup /media/backup ext4 user,noauto,rw 0 0


Obviously don't use the noauto command if you want them to mount automatically.

I don't know why your two disks have the same UUID though. The only time I have seen that happen is when I have cloned a partition.

If you can confirm the duplication using other methods, you may well be better off manually changing the ID value of one of them with tune2fs. This explains how to both check and change a disk UUID: http://linuxconfig.org/how-to-retrieve- ... d-on-linux

Btw. When I was dragged kicking and screaming into the Linux world I was taught that the correct way to use blkid was

Code: Select all
sudo blkid -c /dev/null


Which ignores the blkid cahe. Nobody seems to do this any more, but I wonder if that may account for your strange results??
Fujitsu Lifebook AH532. Intel i5 processor, 6Gb ram, Intel HD3000 graphics, Intel Audio/wifi. Realtek RTL8111/8168B Ethernet.Lubuntu 13.10,Ubuntu12.10 (Unity), Mint16 (Cinnamon), Manjaro (Xfce).
Image
User avatar
viking777
Level 14
Level 14
 
Posts: 5153
Joined: Mon Dec 01, 2008 11:21 am

Re: UUIDs not unique?

Postby SkinnySeal on Tue Aug 27, 2013 2:11 pm

Running blkid in the form "sudo blkid -c /dev/null" gives the same results as just running "sudo blkid". The full output is:

/dev/sda1: LABEL="PQSERVICE" UUID="C4EACD9BEACD8A5E" TYPE="ntfs"
/dev/sda2: LABEL="SYSTEM RESERVED" UUID="3C46CE7D46CE3802" TYPE="ntfs"
/dev/sda3: LABEL="Gateway" UUID="34BC46D1BC468CF4" TYPE="ntfs"
/dev/sda5: UUID="ff722e53-bb22-422a-82b9-391c747c6be3" TYPE="ext4"
/dev/sda6: UUID="30ab4ea2-2e8d-462a-ad71-79faf4f1cc73" TYPE="ext4"
/dev/sda7: UUID="55de1a29-71e5-474c-9ff7-284d2f44b0e4" TYPE="swap"
/dev/sda8: LABEL="Cronos" UUID="000288F00288EC4A" TYPE="ntfs"
/dev/sdb1: LABEL="Poseidon" UUID="DA5494F35494D41B" TYPE="ntfs"
/dev/sdc1: LABEL="Athena" UUID="844019F84019F222" TYPE="ntfs"
/dev/sde1: LABEL="Dionysus" UUID="803CC3A53CC39516" TYPE="ntfs"
/dev/sdf1: LABEL="Aphrodite" UUID="803CC3A53CC39516" TYPE="ntfs"
/dev/sdd1: LABEL="Gorgon" UUID="92846080846068A5" TYPE="ntfs"
/dev/sdl1: LABEL="Hermes" UUID="803CC3A53CC39516" TYPE="ntfs"

I'm running Mint 12 at the moment, but the UUIDs are the same whether I am running under MINT 12 or MINT 15 Mate, which is a good thing.

Thanks, viking77 for giving me two options. I'll have to ponder a bit whether it is better for me to use the label format in fstab, or manually change one of the UUIDs. I'll make my choice when I next boot up MINT 15 MATE.
SkinnySeal
Level 1
Level 1
 
Posts: 34
Joined: Mon Jun 20, 2011 11:00 pm

Re: UUIDs not unique?

Postby SkinnySeal on Tue Aug 27, 2013 2:26 pm

I see now that my external hard drives Hermes has the exact same UUID also. These are the 3 hard drives that I first access after the boot process finishes, so that seems to the common denominator. I can't even power up my external hard drive Hermes until the boot finishes, as it's one hard drive too many for my home PC to even get into Grub's menu. Actually it doesn't matter which external drive I leave powered off to begin with; any external hard drive will do.
SkinnySeal
Level 1
Level 1
 
Posts: 34
Joined: Mon Jun 20, 2011 11:00 pm

Re: UUIDs not unique?

Postby AlbertP on Tue Aug 27, 2013 2:50 pm

Probably the harddisks come from the same manufacturer which hasn't realized that UUID's have to be unique. It looks like one NTFS partition was cloned and put on all those disks.
Registered Linux User #528502
Image
Feel free to correct me if I'm trying to write in Spanish, French or German.
AlbertP
Level 16
Level 16
 
Posts: 6513
Joined: Sun Jan 30, 2011 12:38 pm
Location: The Netherlands

Re: UUIDs not unique?

Postby SkinnySeal on Tue Aug 27, 2013 3:27 pm

Five of my six external hard drives do come from one manufacturer, each pre-configured with the one NTFS partition. Two USB2 connected ones from this manufacturer are automatically booted and do have unique UUIDs. I have no entries for USB2 connected external hard drives in fstab as they may or may not be connected to my home PC at any time.
SkinnySeal
Level 1
Level 1
 
Posts: 34
Joined: Mon Jun 20, 2011 11:00 pm

Re: UUIDs not unique?

Postby Bolle1961 on Tue Aug 27, 2013 3:36 pm

I have also my doubts at UUID's
Asus EeePC900, 2 SSD dives
On 1 drive Debian and swap, other drive SolydX
UUID for swap is different in Debian and SolydX
Bolle1961
Level 3
Level 3
 
Posts: 198
Joined: Tue Jun 29, 2010 12:59 pm

Re: UUIDs not unique?

Postby srs5694 on Wed Aug 28, 2013 12:32 pm

viking777 wrote:you may well be better off manually changing the ID value of one of them with tune2fs.


The disks use NTFS, so tune2fs (which works on ext2/3/4 filesystems) will be useless. I don't see any obvious Linux ntfs* utility that might do the equivalent for NTFS, but it could be that one of these tools has such an option and I'm unaware of it. It might also be possible to make this change in Windows. If nothing else, a desperate last-ditch approach would be to back up the disk, create a fresh filesystem, and then restore it. In fact, if the disk is being used exclusively in Linux, this is what I'd recommend; NTFS is a very poor choice of filesystem for Linux. Its only advantage is that it's Windows' native filesystem, so exchanging files with Windows on NTFS is possible. Even for that, IMHO FAT is superior if you can cope with FAT's file-size limits.

AlbertP wrote:Probably the harddisks come from the same manufacturer which hasn't realized that UUID's have to be unique. It looks like one NTFS partition was cloned and put on all those disks.


I agree with this conclusion; however, on NTFS (and FAT), what Linux reports as a "UUID" is not a true UUID; a UUID is a 128-bit (16-byte) value, which is typically represented in 32 hexadecimal digits, thus:

Code: Select all
580d3322-2ff8-4692-a964-be996c2e9929


Most Linux filesystems use UUIDs as serial numbers to uniquely identify themselves. NTFS and FAT both use shorter numbers for this purpose. Although the goal is the same (to provide each filesystem with a unique serial number), the NTFS serial numbers that SkinnySeal reported are 64-bit values (half the size of a UUID), and FAT uses serial numbers that are only 32 bits in size. Thus, from a statistical perspective, it's much more likely that you'll run into duplicated NTFS or FAT filesystem serial numbers.

Getting back to the main point, if NTFS is really the right filesystem for SkinnySeal, either fixing the problem with a Windows tool or leaving it as-is and using the label to mount the filesystems is probably the way to go. IMHO, backing up, creating a fresh NTFS, and restoring is overkill to fix this problem.
srs5694
Level 6
Level 6
 
Posts: 1018
Joined: Mon Feb 27, 2012 1:42 pm

Re: UUIDs not unique?

Postby viking777 on Wed Aug 28, 2013 1:03 pm

The disks use NTFS, so tune2fs (which works on ext2/3/4 filesystems) will be useless


Excellent point srs and one I had overlooked. However gparted can set a new uuid on devices with windows filesystems, so that route is still open should the OP desire to use it. What kind of UUID it sets I have no idea though I guess it will be a 'Linux' UUID not a 'True' UUID.

However I also agree with your conclusion that label mounts are the way to go - providing he can remember which Greek God lives on which disk :lol:
Fujitsu Lifebook AH532. Intel i5 processor, 6Gb ram, Intel HD3000 graphics, Intel Audio/wifi. Realtek RTL8111/8168B Ethernet.Lubuntu 13.10,Ubuntu12.10 (Unity), Mint16 (Cinnamon), Manjaro (Xfce).
Image
User avatar
viking777
Level 14
Level 14
 
Posts: 5153
Joined: Mon Dec 01, 2008 11:21 am

Re: UUIDs not unique?

Postby grimdestripador on Wed Aug 28, 2013 1:10 pm

statistical perspective, it's much more likely that you'll run into duplicated NTFS or FAT filesystem serial numbers.


Probability of Collisions
EXT3,4: 1/16^128=1/1.340780793×10¹⁵⁴
NTFS: 1/16^64=1/1.157920892×10⁷⁷
Fat32: 1/16^32=1/3.402823669×10³⁸

Comparison of other rare probabilities
Number of stars in the Milky Way 2-4x10^11
Number of atoms in galaxy 1x10^69

Thus the chance that you had a colllision with a UUID with NTFS formatting is about 10^69/10^77 = 10^8 = 100,000,000 or a 100 million times more likely to randomly pick two atoms in out galaxy than you are to pick two UUIDs based on NTFS.
User avatar
grimdestripador
Level 6
Level 6
 
Posts: 1069
Joined: Fri Feb 16, 2007 2:26 am

Re: UUIDs not unique?

Postby viking777 on Wed Aug 28, 2013 1:30 pm

Lovely mathematics grimdestripador.

So based on the fact that the OP has three identical UUID's out of 13 partitions I guess he should definitely do the lottery this week :lol: (and probably shouldn't go out in a thunderstorm).
Fujitsu Lifebook AH532. Intel i5 processor, 6Gb ram, Intel HD3000 graphics, Intel Audio/wifi. Realtek RTL8111/8168B Ethernet.Lubuntu 13.10,Ubuntu12.10 (Unity), Mint16 (Cinnamon), Manjaro (Xfce).
Image
User avatar
viking777
Level 14
Level 14
 
Posts: 5153
Joined: Mon Dec 01, 2008 11:21 am

Re: UUIDs not unique?

Postby AlbertP on Wed Aug 28, 2013 2:47 pm

viking777 wrote:What kind of UUID it sets I have no idea though I guess it will be a 'Linux' UUID not a 'True' UUID.

The length of the UUID will stay the same since you don't change the partition type. Partitions of the same type also have the same kind of UUID and that should be independent from the operating system you use to create the partition.
Registered Linux User #528502
Image
Feel free to correct me if I'm trying to write in Spanish, French or German.
AlbertP
Level 16
Level 16
 
Posts: 6513
Joined: Sun Jan 30, 2011 12:38 pm
Location: The Netherlands

Re: UUIDs not unique?

Postby srs5694 on Thu Aug 29, 2013 2:42 pm

grimdestripador wrote:
statistical perspective, it's much more likely that you'll run into duplicated NTFS or FAT filesystem serial numbers.


Probability of Collisions
EXT3,4: 1/16^128=1/1.340780793×10¹⁵⁴
NTFS: 1/16^64=1/1.157920892×10⁷⁷
Fat32: 1/16^32=1/3.402823669×10³⁸


Your computation is flawed; you raised to different powers from a base of 16, when in fact you should have used a base of 2 -- the values are 32, 64, or 128 bits in size, not 32, 64, or 128 long words in size, as your computations show. Thus, the correct values are:

ext2,3,4: 1/2^128=1/3.40x10^38
NTFS: 1/2^64=1/1.84x10^19
FAT32: 1/2^32=1/4,294,967,296

For comparison, the number of FAT32 values is about 4 billion, whereas the number of people in the world is about 7 billion. Thus, the odds of getting two identical FAT32 serial numbers on two FAT32 disks would be greater than the odds of one person winning a hypothetical world-wide lottery twice in a row. Those odds are still pretty small, of course, and I never claimed otherwise -- I simply claimed that the probabilities of such a match were less for a true UUID than for an NTFS or FAT serial number. That claim is true, no matter how small the probabilities are in absolute terms.
srs5694
Level 6
Level 6
 
Posts: 1018
Joined: Mon Feb 27, 2012 1:42 pm

Re: UUIDs not unique?

Postby srs5694 on Thu Aug 29, 2013 2:44 pm

viking777 wrote:So based on the fact that the OP has three identical UUID's out of 13 partitions I guess he should definitely do the lottery this week :lol: (and probably shouldn't go out in a thunderstorm).


No; it just means that whatever produced those filesystems was not properly randomizing the serial numbers. I believe somebody suggested earlier that they were set to a static value by a disk manufacturer. This makes sense. Other possibilities including a bug in a script or disk partitioning program, filesystem cloning, and deliberate duplication by a person.
srs5694
Level 6
Level 6
 
Posts: 1018
Joined: Mon Feb 27, 2012 1:42 pm

Re: UUIDs not unique?

Postby AlbertP on Thu Aug 29, 2013 3:07 pm

Yes, I indeed suggested that the disk manufacturer could have done this. @ SkinnySeal: Have the affected disks indeed been produced by the same manufacturer? If you never formatted them yourself, just blame your disk manufacturer then. I guess it would be nice to know which manufacturer is doing such a bad thing.

And if it's not the disk manufacturer, then it'll be one of srs5694's other suggestions which are also well possible.
Registered Linux User #528502
Image
Feel free to correct me if I'm trying to write in Spanish, French or German.
AlbertP
Level 16
Level 16
 
Posts: 6513
Joined: Sun Jan 30, 2011 12:38 pm
Location: The Netherlands

Re: UUIDs not unique?

Postby SkinnySeal on Thu Aug 29, 2013 3:34 pm

I went with the LABEL mount option. Simplest solution. I'm triple booted, and my external hard drives are storage units that must be accessible from both Linux and Windows - hence NTFS is the only logical choice. They're Fantom drives.

I note a difference in directory path from Mint12 to Mint15 in the autoboot of USB2 connected drives. In Mint 12, all my external drives - USB2 and eSata - are mounted directly under the /media folder. In Mint 15, the eSata and my internal drive data partion are mounted directly under /media; but the USB2 connected drives were mounted under /media/<user-id>. It's not an issue for me, but I'm curious why the change and if this is something that is selectable. It didn't matter if I listed the USB connected drives in the fstab file or not, USB2 connected drives got mounted in /media/<user-id> either way.

Thanks for all the input!
SkinnySeal
Level 1
Level 1
 
Posts: 34
Joined: Mon Jun 20, 2011 11:00 pm

Re: UUIDs not unique?

Postby DrHu on Thu Aug 29, 2013 5:25 pm

https://en.wikipedia.org/wiki/UUID
    The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". Since the identifiers have a finite size, it is possible for two differing items to share the same identifier The identifier size and generation process need to be selected so as to make this sufficiently improbable in practice

http://linux.die.net/man/3/uuid_generate
    Description
    The uuid_generate function creates a new universally unique identifier (UUID). The uuid will be generated based on high-quality randomness from /dev/urandom, if available. If it is not available, then uuid_generate will use an alternative algorithm which uses the current time, the local ethernet MAC address (if available), and random data generated using a pseudo-random generator.
    The uuid_generate_random function forces the use of the all-random UUID format, even if a high-quality random number generator (i.e., /dev/urandom) is not available, in which case a pseudo-random generator will be substituted. Note that the use of a pseudo-random generator may compromise the uniqueness of UUIDs generated in this fashion.

Manufacturer's quality control issue for hard drives: they would have the option of ensuring randomness as far as it is possible.
User avatar
DrHu
Level 16
Level 16
 
Posts: 6611
Joined: Wed Jun 17, 2009 8:20 pm

Re: UUIDs not unique?

Postby srs5694 on Thu Aug 29, 2013 10:15 pm

DrHu wrote:https://en.wikipedia.org/wiki/UUID
    The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". Since the identifiers have a finite size, it is possible for two differing items to share the same identifier The identifier size and generation process need to be selected so as to make this sufficiently improbable in practice
...
Manufacturer's quality control issue for hard drives: they would have the option of ensuring randomness as far as it is possible.


I feel I must reiterate a point I made earlier: The "UUID" values reported for NTFS and FAT volumes are not UUIDs! They're serial numbers that play the same role on NTFS/FAT that UUIDs serve on most Linux-native filesystems, but they aren't UUIDs. I have no special insight into the minds of the programmers who wrote them, but I would speculate that Linux utilities report these filesystem serial numbers as UUIDs simply because using some other term would complicate the code -- using "UUID" for Linux filesystems that use UUIDs and "SERIALNO" for filesystems that don't use UUIDs would make the code (and associated documentation) more complex. Arguably, "SERIALNO" would be more generally correct, but the programmers went with "UUID," so we're stuck with that.

That said, the snippet you quoted from Wikipedia applies equally well to NTFS and FAT serial numbers, with the caveat that the probability of collisions goes up because NTFS and (especially) FAT serial numbers are smaller than UUIDs.
srs5694
Level 6
Level 6
 
Posts: 1018
Joined: Mon Feb 27, 2012 1:42 pm

Re: UUIDs not unique? <SOLVED>

Postby grimdestripador on Fri Aug 30, 2013 1:30 pm

TIL more about UUIDS
User avatar
grimdestripador
Level 6
Level 6
 
Posts: 1069
Joined: Fri Feb 16, 2007 2:26 am

Linux Mint is funded by ads and donations.
 

Return to Mounting Partitions

Who is online

Users browsing this forum: No registered users and 2 guests