Best File Format for File Preservation? [SOLVED]

Questions about hardware,drivers and peripherals
Forum rules
Before you post please read how to get help
Post Reply
User avatar
Amii_Leigh
Level 5
Level 5
Posts: 634
Joined: Fri Mar 25, 2016 10:58 pm
Location: Somewhere in the middle of nowhere, Missouri

Best File Format for File Preservation? [SOLVED]

Post by Amii_Leigh » Wed Jun 07, 2017 8:05 pm

Is there a 'best' file format for preserving file integrity over longer stretch of time?
Is this a matter of opinion, or are there actually some formats are better for this than others?
I know that three hundred gigablytes of data isn't such a great thing in these days of terabyte drives, but I'd still like to keep what I have managed to keep for the last eight years, yes, you saw that right, eight whole years I've been using this drive.
Any positive responses are welcome.

Thank you!
Last edited by Amii_Leigh on Wed Jun 07, 2017 9:07 pm, edited 1 time in total.
नमस्ते = Namaste
I honor the place in you in which the entire universe dwells.
I honor the place in you in which is of love, of truth, of light, and of peace.
When you are in that place in you, and I am in that place in me, we are one.

User avatar
WharfRat
Level 20
Level 20
Posts: 11436
Joined: Thu Apr 07, 2011 8:15 pm

Re: Best File Format for File Preservation?

Post by WharfRat » Wed Jun 07, 2017 8:43 pm

Use Ext4 if You’re Not Sure :wink:

I've experienced several sudden power outages over the years using Linux and the ext3 and ext4 file systems booted right up like nothing ever happened.
ImageImage

User avatar
Amii_Leigh
Level 5
Level 5
Posts: 634
Joined: Fri Mar 25, 2016 10:58 pm
Location: Somewhere in the middle of nowhere, Missouri

Re: Best File Format for File Preservation?

Post by Amii_Leigh » Wed Jun 07, 2017 9:07 pm

Thank you ever so muchly!

Namaste!
नमस्ते = Namaste
I honor the place in you in which the entire universe dwells.
I honor the place in you in which is of love, of truth, of light, and of peace.
When you are in that place in you, and I am in that place in me, we are one.

User avatar
jimallyn
Level 18
Level 18
Posts: 8297
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: Best File Format for File Preservation? [SOLVED]

Post by jimallyn » Thu Jun 08, 2017 7:07 pm

My experiences with EXT3, EXT4, and Reiser have all been about the same as WharfRat's: they just work. (Reiser isn't used much anymore.) As I understand it, it's the journaling that does it. Basically, it makes a record of what it's going to do, then does it, then records that it did it. A friend of mine who is a lawyer says when presenting a case to a jury, you have to tell them what you're going to tell them, tell them, then tell them what you told them. Works for filesystems, too.
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan

Mute Ant
Level 13
Level 13
Posts: 4903
Joined: Tue Sep 03, 2013 7:45 pm

Re: Best File Format for File Preservation? [SOLVED]

Post by Mute Ant » Sat Jun 10, 2017 10:28 am

Gently wish to point out that the file-system is not the file-format.

These people are in the business... http://www.nationalarchives.gov.uk/docu ... actice.pdf

It's not just opinion...'best' needs something to work with...

o Effects of damage. An old JPG photograph with a blue line through it (because the CD-R faded) is obviously more damaged than a BMP with just one pixel missing. But you can get 5 copies of the JPG (in five different devices) for the same storage cost as one robust BMP. The trend in SSD and HDD is obvious...halve the geometry...quadruple the capacity. Then you have to decide to use that extra storage for redundant copies.

o Does your file-storage device notice bit-rot? Mostly yes. SSD or HDD or USB-Flash or DVD or CD all use checksums to detect bit-rot. Even paper tape had a parity bit (yes I am that old).

o Does your motherboard notice bit-rot? Mostly no. A 'stuck' or 'fizzy' bit in RAM can affect what's read from a store (bad) or what's written (worse).

o Does your file-system notice bit-rot? Mostly no. VFAT NTFS EXT2 EXT3 EXT4 do not record the checksum of data stored, so they have no idea if what comes back from the store is accurate. ZFS and BTRFS do store checksums for data. BTRFS does so on a per-file basis and can make a redundant copy in the same file system, so a bad read can be repaired from the good copy (recommended for data...not so good for OS...it works, but it's 'gritty').

o Does your file-format notice bit-rot? ZIP and PNG and FLAC do, they have internal checksums. MPEG and JPEG, no.

JPEG can be made more robust by using the jpegtran command to lossless-recode it, compressing each 16x16 MCU block separately. Then, a bad bit only affects a small part of the image. The file size increases...sometimes to 160% of the original size...trivial compared to recoding as lossless PNG or BMP though.

Code: Select all

### Recode a JPEG photograph to be more resistant to bit-rot...
    cat pussy.jpg | jpegtran -copy all -restart 1B > tiger.jpg
Data stored in an encrypted volume will only be accessible with the original password. Without this key, all the data will remain scrambled beyond hope of recovery.

Post Reply

Return to “Hardware Support”