Linux Mint Live USB with Persistent Partition: how?

[vansloneker]

...

It was always the sqlite databases that got corrupted on my drives and it happened on three separate Kingston DataTraveler drives after 6-9 months daily use (bought a bunch on sale) suggesting wear leveling wasn't perfect. Granted, SanDisk probably uses a better controller (and mayhaps better NAND) than the one on the Kingston. After the third corruption, I switched to portable USB HDDs, then later 2.5" SSDs in USB enclosure for my PortableApps.

Wow..
I have 1 Kingston DataTraveler too,
and it's extremely slow, compared to all my other USB Flash Drives.
I will never buy Kingston again, especially after what you wrote about your 3 ones..
...

I only recently booted from a Lexar where I had cloned my Kingston onto and I was surprised at how much quicker the Lexar was compared to the Data Traveler. A good USB2 stick is maybe faster than the USB3 Kingston.
One big pro of the Kingston is it's solid construction, it has the best retractable connector of all sticks and still opens and closes as firm as when new. You can have it in your bag or your pocket and it will survive.

[spaceman5]

One big pro of the Kingston is it's solid construction, it has the best retractable connector of all sticks and still opens and closes as firm as when new.
You can have it in your bag or your pocket and it will survive.

Yes its casing is good, but many other Mfrrs have models that have a thick plastic casing too..

For example Sandisk usually makes 7-8 models simultaneously, with different casings, from thin and flexible (and cheap), to thick and strong.


BTW,

P.S. The Kingston drives worked fine for file storage. I expect Firefox sqlite db just isn't a normal load for these things.

Why did you put Firefox's sqlite db on a USB Flash Drive?
(was it just the SQLite DB on the USB Flash Drive,
or a full OS such as Mint, with Firefox, and the SQLite DB among all these?)

[rui no onna]

BTW,

P.S. The Kingston drives worked fine for file storage. I expect Firefox sqlite db just isn't a normal load for these things.

Why did you put Firefox's sqlite db on a USB Flash Drive?
(was it just the SQLite DB on the USB Flash Drive,
or a full OS such as Mint, with Firefox, and the SQLite DB among all these?)

PortableApps. That's one of the things I'm looking for a Linux alternative. I had LibreOffice, Notepad++, Calibre, etc. on there, too. Firefox was the most used (daily 8+ hr usage). Well, Notepad++, too but that one doesn't write to the disk much.

The Kingston I was using was just USB 2.0. By the time I made the switch to USB 3.0, I was already using SSDs in enclosure.
Kingston DataTraveler SE9

Right now, I use the Samsung BAR+ for persistent Linux. The SanDisk Extreme PRO is very nice but I don't need that small of a form factor and I'd rather spend the same amount on a bigger SSD+enclosure.

[spaceman5]

PortableApps. That's one of the things I'm looking for a Linux alternative. I had LibreOffice, Notepad++, Calibre, etc. on there, too.
Firefox was the most used (daily 8+ hr usage). Well, Notepad++, too but that one doesn't write to the disk much.

Oh..
BTW, is there a way to be sure that FF's SQLite DB file is the problematic item?
Or that is just an logic estimation?
(it would've been really useful to know what file ruined a USB Flash Drive/SSD)

[rui no onna]

Oh..
BTW, is there a way to be sure that FF's SQLite DB file is the problematic item?
Or that is just an logic estimation?
(it would've been really useful to know what file ruined a USB Flash Drive/SSD)

It was the same file (iirc, places/history?) that got corrupted on three separate occasions so it's the logical assumption. I had a robocopy job for backup and that's where it falters.

Mind, wear leveling is supposed to avoid cases like that. It just doesn't look like the Kingston controller does a good enough job of wear leveling. On good SSDs, the controller will remap frequently rewritten data to cells with less wear. That way, you don't get into a situation where you have a cell/block that's used up all its P/E cycles while you have other cells that have barely used their P/E cycles. I've seen this happen on gen 1 Indilinx SSDs.

[spaceman5]

It was the same file (iirc, places/history?) that got corrupted on three separate occasions so it's the logical assumption. I had a robocopy job for backup and that's where it falters.

Oh..
File Corruption is a nice way to isolate the cause..

But I am curious:
If a certain part of the Flash Memory got corrupted, it shouldn't make the whole USB Flash Drive broken,
it should just decrease the amount of usable space the user has..

Some USB Flash Drives and SSDs even contain a small hidden area, that is used to compensate for dying bits.
But even if there weren't such an area, so a 32GB USB Flash Drive will turn into a 31.5GB one.. it should still be working, no?

Mind, wear leveling is supposed to avoid cases like that. It just doesn't look like the Kingston controller does a good enough job of wear leveling.
On good SSDs, the controller will remap frequently rewritten data to cells with less wear. That way, you don't get into a situation where you have a cell/block that's used up all its P/E cycles while you have other cells that have barely used their P/E cycles. I've seen this happen on gen 1 Indilinx SSDs.

Right.
Seems that Kingston is a bad choice both in speed and in controller quality..

[rui no onna]

But I am curious:
If a certain part of the Flash Memory got corrupted, it shouldn't make the whole USB Flash Drive broken,
it should just decrease the amount of usable space the user has..

Some USB Flash Drives and SSDs even contain a small hidden area, that is used to compensate for dying bits.
But even if there weren't such an area, so a 32GB USB Flash Drive will turn into a 31.5GB one.. it should still be working, no?

First time it happened, I haven't identified the cause so the prudent thing to do was to stop using the drive. Second time, a pattern began to emerge but that was still just a theory. It wasn't until the third time that I was reasonably certain of the cause.

But yes, drives still work (they were just small though - 8GB). Like I mentioned, I stopped using them for PortableApps duty and only use them for non-essential write-once, read-many tasks (e.g. software and driver installers).

[spaceman5]

Interesting..
So you can still use them.
Did their controller mark the bad bits as non-usable, and you don't get any errors any more? (from the specific parts that were ruined already)
(did you try to run CHKDSK/F in Windows on them, to see that the bad parts were taken out of use?)


BTW,
maybe from this repeating incident, one should disable Firefox's History, Cookies, and all this..
(for example by setting it to always be in Private Browsing mode..)
We lose information, but help keep the drive for a much longer term..
What do you think?



And I am curious to ask you about a different thing that you wrote several days ago:

After the third corruption, I switched to portable USB HDDs, then later 2.5" SSDs in USB enclosure for my PortableApps.

This sentence that you wrote several days ago, made me think..

I wonder what will usually be better, for running Mint:
A USB Flash Drive (with Mint real-installed on it),
or an SSD with a USB Enclosure (with Mint installed on it)..


(let's leave the price difference aside for a moment, and focus only on technical advantages and disadvantages)


1 advantage for the SSD+USB Enclosure option, is that the SSD will have DRAM for Cache.
USB Flash Drives do not have DRAM, at least I have never heard of any yet..
SSDs on the other hand, have. Not all of them, not the extremely cheap models, but many do have.


On the other hand,
the pair SSD and USB Enclosure may not be optimized one with the other..
Those are 2 pieces of hardware that come from 2 different mfrrs, and will not necessarily work the optimal way together.
A USB Flash driveon the other hand should be optimized - the Mfrr chooses both the Controller and the Flash chips, and if he did a good job, it should give a good drive.

Despite what I wrote in this last paragraph, I think that in the end, many SSD+USB Enclosures will outperform most USB Flash Drives (maybe except the most expensive USB Flash Drives).


If someone has more advantages or disadvantages to each of the 2 options, please write

[rui no onna]

Interesting..
So you can still use them.
Did their controller mark the bad bits as non-usable, and you don't get any errors any more? (from the specific parts that were ruined already)
(did you try to run CHKDSK/F in Windows on them, to see that the bad parts were taken out of use?)

I don't think the controller marked the bits as unusable nor did it replace them with spare sectors. I did run chkdsk /f /r on it and let Windows mark the bad sectors.

BTW,
maybe from this repeating incident, one should disable Firefox's History, Cookies, and all this..
(for example by setting it to always be in Private Browsing mode..)
We lose information, but help keep the drive for a much longer term..
What do you think?

For me, keeping the history (at least for the last 2 months) is important. Hence, the switch to portable HDDs/SSDs. Ended up with a performance gain, too, so win-win.

I wonder what will usually be better, for running Mint:
A USB Flash Drive (with Mint real-installed on it),
or an SSD with a USB Enclosure (with Mint installed on it)..


(let's leave the price difference aside for a moment, and focus only on technical advantages and disadvantages)


1 advantage for the SSD+USB Enclosure option, is that the SSD will have DRAM for Cache.
USB Flash Drives do not have DRAM, at least I have never heard of any yet..
SSDs on the other hand, have. Not all of them, not the extremely cheap models, but many do have.


On the other hand,
the pair SSD and USB Enclosure may not be optimized one with the other..
Those are 2 pieces of hardware that come from 2 different mfrrs, and will not necessarily work the optimal way together.
A USB Flash driveon the other hand should be optimized - the Mfrr chooses both the Controller and the Flash chips, and if he did a good job, it should give a good drive.

Despite what I wrote in this last paragraph, I think that in the end, many SSD+USB Enclosures will outperform most USB Flash Drives (maybe except the most expensive USB Flash Drives).


If someone has more advantages or disadvantages to each of the 2 options, please write

Both Samsung and SanDisk make external SSDs if you're concerned about the separate controller. I prefer enclosure + 2.5" SATA SSD, though. More flexibility and options for recovery. Less expensive, too.

I ran VMs from my SSD+USB enclosure (connected via USB 2.0!) and funnily enough, the XP VMs were "snappier" than the native Windows XP install on a 3.5" 7200RPM HDD. This was back when SSDs were $3-4/GB. The SSD used was an Intel X25-V 40GB.

Really, the biggest downside to external SSD vs flash drive is size/form factor and potential BIOS boot compatibility issues. Apart from powerful controllers and DRAM cache, NAND used in SSDs also tend to be more cherry-picked than the ones used in flash drives.

https://www.anandtech.com/show/6459/sam ... f-tlc-nand

Furthermore, it should be kept in mind that all SMART values that predict lifespan are conservative; it's highly unlikely that your drive will drop dead once the WLC or MWI hits zero. There is a great example at XtremeSystems where a 256GB Samsung SSD 830 is currently at nearly 6,000TiB of writes. Its WLC hit zero at 828TiB of writes, which means its endurance is over seven times higher than what the SMART values predicted. That doesn't mean all drives are as durable but especially SSDs from NAND manufacturers (e.g. Intel, Crucial/Micron, Samsung etc.) seem to be more durable than what the SMART values and datasheets indicate, which isn't a surprise given that they can cherry-pick the highest quality NAND chips.

Granted, there's cost, too, but if you consider $/GB, the SSDs are actually pretty competitive.

[spaceman5]

For me, keeping the history (at least for the last 2 months) is important.
Hence, the switch to portable HDDs/SSDs. Ended up with a performance gain, too, so win-win.

I recently moved to SSD in my computer.
It's a good one (Samsung 860 EVO),
and altho I am sure it has a good controller with a good wear leveling mechanism,
it makes me worry abit, that my browser's (Firefox) history mechanism, will be the most intensive write-work on that SSD..
(and like you, I want to keep it, not disable it)

Maybe one should relocate it to a different SSD drive, a small and cheap one, that the user is not worried to lose and replace from time to time...
Or maybe I should trust the wear leveling enough..

Both Samsung and SanDisk make external SSDs if you're concerned about the separate controller.

Oh you're right,
completely forgot about that option

I prefer enclosure + 2.5" SATA SSD, though.
More flexibility and options for recovery. Less expensive, too.

Right.
Yesterday, I ordered this:

https://www.gearbest.com/hdd-enclosure/pp_866790.html


It's really cheap as can be seen.

The USB(2.0+3.0) <--> SATA chip that it uses is JMicron JMS567.
It should be good from results people posted,
but I will google to see if there's even a better USB<->SATA chip that is worth using.

I ran VMs from my SSD+USB enclosure (connected via USB 2.0!) and funnily enough, the XP VMs were "snappier" than the native Windows XP install on a 3.5" 7200RPM HDD. This was back when SSDs were $3-4/GB. The SSD used was an Intel X25-V 40GB.

Amazing..
Almost inconceivable that it was so good, when using an Enclosure, and on USB 2.0 even..
This is quite encouraging

[rui no onna]

I recently moved to SSD in my computer.
It's a good one (Samsong 860 EVO),
and altho I am sure it has a good controller with a good wear leveling mechanism,
it makes me worry abit, that my browser's (Firefox) history mechanism, will be the most intensive write-work on that SSD..
(and like you, I want to keep it, not disable it)

Maybe one should relocate it to a different SSD drive, a small and cheap one, that the user is not worried to lose and replace from time to time...
Or maybe I should trust the wear leveling enough..

I wouldn't worry about it at all. Samsung has one of the best controllers around and since they manufacture the NAND, they can also cherry-pick the ones for their SSDs.

I actually have Samsung's first consumer SSD with planar TLC NAND. Been in use since release (maybe 5-6 years?) and it's only used 54 of its 1000 rated P/E cycles (3000 actual per endurance testing) with write amplification at 1.8x. I don't baby it at all but at the rate it's going, it'll be 90+ years before its Media Wear Indicator hits 0 and another 180 years before it actually uses up all its P/E cycles. I expect it to die of other causes (or be completely obsolete) way, way, way before then.

Also note the 860 EVO is warrantied for 5 years (as are a number of other SSDs) while most HDDs are only warrantied for 2-3 years. Given RMA replacements cost companies $$$, that should give an idea of comparative failure rates. Granted, HDDs often tend to give early warning of impending failure. Meanwhile, I believe majority of SSD failures so far have not been due to NAND wear and thus were completely unpredictable.

[spaceman5]

I don't baby it at all but at the rate it's going, it'll be 90+ years before its Media Wear Indicator hits 0 and another 180 years before it actually uses up all its P/E cycles.

In this rate the SSD will outlive me...

I believe majority of SSD failures so far have not been due to NAND wear

Right

From what I heard from friends who had problems with their SSD,
it usually suddenly died, and not from using all the write cycles, but it simply "burnt".
Probably a controller problem.

It was not with Samsung, it was with SanDisk SSDs..


BTW,
do you have any recommendation regarding a best chip for a USB HDD/SSD Enclosure?

As written, the one I bought (the 5$ one) has JMicron JMS567 which should be good.
But maybe you (or anyone else reading this) investigated it more and found an even better chip?

[BG405]

Right.
Yesterday, I ordered this:
...
The USB(2.0+3.0) <--> SATA chip that it uses is JMicron JMS567.

Good find, indeed! Hope it works well.

I've found a supplier in Hong Kong selling them for £1.20 including postage & packaging! Think I'll order one (probably 2, so I can clone disks more easily), rather than using the bare board SATA ⬄ USB adaptor down the Pub. I'm taking risks using unprotected electronics in such environments. Single USB plug is fine on my kit & it'll free up my DVD burner's USB lead, for around the cost of a decent USB lead.

I do have a dock (with card readers & a couple of USB ports) but that requires a mains adaptor & isn't as practical on the move.

These USB adaptors in my experience usually don't fully support all the features of a hard disk (i.e. SMART data etc.) i.e. you still need to put it in a machine for full functionality, but are fine for general use.

[spaceman5]

I've found a supplier in Hong Kong selling them for £1.20 including postage & packaging

You're not talking about the exact same thing that I gave in the link,
but you talk generally about a USB<->SATA Adapter, right?

Yes, in eBay you can find such for less than the already cheap price that I paid.
The problem is that they don't state there what chip the adapter is using.

I'm taking risks using unprotected electronics in such environments.

What do you mean?

[BG405]

You're not talking about the exact same thing that I gave in the link,
but you talk generally about a USB<->SATA Adapter, right?

It's the same model you've just ordered hope it has the same chipset. Your supplier is offering these to the UK for £4.11 inc. P&P.

I'm taking risks using unprotected electronics in such environments.

What do you mean?

Risk of touching the components & upsetting things or damaging the board; knocks, spillage etc.. If it's in a case it's inherently less vulnerable. Can also add soft feet to help absorb vibrations.

ETA: Just realized the £1.20 price is an auction. The cheapest Buy-it-Now is £1.69, sorry for the confusion.

[spaceman5]

OK

[rui no onna]

I'm taking risks using unprotected electronics in such environments.

What do you mean?

Risk of touching the components & upsetting things or damaging the board; knocks, spillage etc.. If it's in a case it's inherently less vulnerable. Can also add soft feet to help absorb vibrations.

ETA: Just realized the £1.20 price is an auction. The cheapest Buy-it-Now is £1.69, sorry for the confusion.

In fairness, apart from accidental disconnection, the other stuff aren't really an issue for 2.5" SATA SSDs as those are already enclosed.

@spaceman5
Sorry, haven't really paid attention to recent SATA-USB chipsets. The JMicron is probably fine, though. I think my old USB 2.0 enclosure has JMicron. I think the only options are ASMedia and JMicron anyway so not really a large pool to choose from.

[BG405]

n fairness, apart from accidental disconnection, the other stuff aren't really an issue for 2.5" SATA SSDs as those are already enclosed.

True but I don't have (nor can afford) SSDs at present. I'm sure many others are also in this position. The board I'm using now would still be just as vulnerable, though.

[lsemmens]

Just a couple of "idle, silly" questions. Is there a BIOS that will permit booting from a USB connected HDD? (i've not tried, so don't know) I'd have thought that that would be a no-brainer given that USB flash memory can be bootable. How would I convince Linux to install and run from said drive? Could I configure linux to be as secure as TOR or will I need to download and install tails?

EDIT: Wife's 'puter just up and spat the network dummy, so thought I'd try. So Far, so good am Installing Mint to an external drive as I type.

If it works as I hope, I might even give tails a spin to see how that works. My issue with something like tails was that, I believe, it was meant to run from a memory stick which in many cases would be, then, unavailable to store data unless somehow you could configure persistence. It that even possible in tails?

Stay tuned!

EDIT2: I thought I'd keep it all in one post so people can find the results easily
YES you can load to, and boot from an external HDD with one caveat! It stuffs up Grub on the primary drive so now I have to work out how to repair it!!! 'nutha thread coming up!

EDIT 3: BUGGA! wouldn't boot on another machine BACK TO THE DRAWING BOARD!

[spaceman5]

The JMicron is probably fine, though.
I think my old USB 2.0 enclosure has JMicron. I think the only options are ASMedia and JMicron anyway so not really a large pool to choose from.

Yeah those 2 repeat themselves..
With some people preferring the first, and some people preferring the second