SSD can't be stopped after backup and TRIM

[Lady Fitzgerald]

I'm between desktop computers right now but I'm still maintaining the data drives and their backups. The desktop computer drives are ext4 but the desktop backup drives are NTFS (they need to be readable by a windows computer). The data drives/partitions in my laptop are identical to the desktop computer drives so I update the desktop drives and their backups the same way I update the backup drives for my laptop.

I use FreeFileSync (FFS) to backup my data drives (files get copied and deleted on the backup drives as needed to make them identical to the data drives in the computer). I normally manually TRIM the laptop drives and their external USB backup drives once a week. I plug all four external USB backup drives in at the same time and have an alias that will TRIM all of the drives one after the other until done. After TRIMming, I safely remove the backup drives without a problem.

I update the laptop data drives/partitions as needed (one data drive in the laptop has two 4TB partitions and the other two are 4 TB drives). The main data drive's external backup drive gets updated at the end of the day (sooner if I add, change, move, etc. a lot of data during the day). The other data drives rarely get written to so I update their backup drives as needed or monthly, whichever comes first.

I treat the desktop data drives (kept in USB enclosures) the same as the laptop data drives and update them as needed. I also TRIM the desktop data drives once a week. If I've added, changed, etc. a lot of data to the laptop data drives, I will update the backups, using FFS, before TRIMing. Usually, I can safely remove the drives without any trouble.

The backup drives for the desktop data drives get the same treatment however I often run into problems when trying to safely remove them after updating and TRIMming. When I right click on the left panel of a drive's open window and click on safely remove, I often will get an error message saying the drive can't be stopped and all four of the external drives will be unmounted and have to be remounted and, after waiting a while, try again. Sometimes, waiting awhile after TRIMming before trying to safely removing them will avoid the problem. Other times, I have shutdown the computer, then unplug the drives.

This is driving me insane(r). Does anyone have any idea why this happens and what can I do avoid the problem?

Code: Select all

jeannie@Laptop1:~$ inxi -Fxxxrz
System:
  Host: Laptop1 Kernel: 5.4.0-81-generic x86_64 bits: 64 compiler: gcc 
  v: 7.5.0 Desktop: Cinnamon 4.4.8 wm: muffin 4.4.4 dm: LightDM 1.26.0 
  Distro: Linux Mint 19.3 Tricia base: Ubuntu 18.04 bionic 
Machine:
  Type: Laptop System: System76 product: Serval v: serw11-b serial: <filter> 
  Chassis: type: 10 serial: <filter> 
  Mobo: System76 model: Serval v: serw11-b serial: <filter> 
  UEFI: American Megatrends v: 1.05.25-1 date: 07/10/2019 
Battery:
  ID-1: BAT0 charge: 57.9 Wh condition: 80.7/79.9 Wh (101%) volts: 15.9/14.8 
  model: Notebook BAT type: Li-ion serial: <filter> status: Unknown 
  Device-1: hidpp_battery_0 model: Logitech Wireless Mouse M525 
  serial: <filter> charge: 100% rechargeable: yes status: Discharging 
CPU:
  Topology: 8-Core model: Intel Core i7-9700K bits: 64 type: MCP 
  arch: Kaby Lake rev: D L2 cache: 12.0 MiB 
  flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 57600 
  Speed: 800 MHz min/max: 800/4900 MHz Core speeds (MHz): 1: 800 2: 800 
  3: 800 4: 800 5: 800 6: 800 7: 800 8: 800 
Graphics:
  Device-1: NVIDIA vendor: CLEVO/KAPOK driver: nvidia v: 460.39 
  bus ID: 01:00.0 chip ID: 10de:1f51 
  Display: x11 server: X.Org 1.20.8 driver: nvidia 
  unloaded: fbdev,modesetting,nouveau,vesa 
  resolution: 1920x1080~60Hz, 1920x1080~144Hz 
  OpenGL: renderer: GeForce RTX 2060/PCIe/SSE2 v: 4.6.0 NVIDIA 460.39 
  direct render: Yes 
Audio:
  Device-1: Intel 200 Series PCH HD Audio vendor: CLEVO/KAPOK 
  driver: snd_hda_intel v: kernel bus ID: 00:1f.3 chip ID: 8086:a2f0 
  Device-2: NVIDIA vendor: CLEVO/KAPOK driver: snd_hda_intel v: kernel 
  bus ID: 01:00.1 chip ID: 10de:10f9 
  Device-3: N/A type: USB driver: hid-generic,snd-usb-audio,usbhid 
  bus ID: 1-5.4.3:9 chip ID: 21b4:0082 serial: <filter> 
  Sound Server: ALSA v: k5.4.0-81-generic 
Network:
  Device-1: Qualcomm Atheros Killer E2500 Gigabit Ethernet 
  vendor: CLEVO/KAPOK driver: alx v: kernel port: d000 bus ID: 6e:00.0 
  chip ID: 1969:e0b1 
  IF: enp110s0 state: up speed: 1000 Mbps duplex: full mac: <filter> 
  Device-2: Intel Wireless-AC 9260 driver: iwlwifi v: kernel port: d000 
  bus ID: 71:00.0 chip ID: 8086:2526 
  IF: wlp113s0 state: down mac: <filter> 
Drives:
  Local Storage: total: 14.73 TiB used: 7.47 TiB (50.7%) 
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 PRO 512GB 
  size: 476.94 GiB speed: 31.6 Gb/s lanes: 4 serial: <filter> rev: 1B2QEXP7 
  scheme: GPT 
  ID-2: /dev/nvme1n1 model: Sabrent Rocket Q size: 7.28 TiB speed: 31.6 Gb/s 
  lanes: 4 serial: <filter> rev: RKT30Q.2 scheme: GPT 
  ID-3: /dev/sda vendor: Samsung model: SSD 860 PRO 4TB size: 3.73 TiB 
  speed: 6.0 Gb/s serial: <filter> rev: 1B6Q scheme: GPT 
  ID-4: /dev/sdb vendor: Samsung model: SSD 860 PRO 4TB size: 3.73 TiB 
  speed: 6.0 Gb/s serial: <filter> rev: 1B6Q scheme: GPT 
Partition:
  ID-1: / size: 452.31 GiB used: 27.40 GiB (6.1%) fs: ext4 
  dev: /dev/nvme0n1p2 
  ID-2: swap-1 size: 15.91 GiB used: 0 KiB (0.0%) fs: swap 
  dev: /dev/nvme0n1p3 
Sensors:
  System Temperatures: cpu: 50.0 C mobo: N/A gpu: nvidia temp: 46 C 
  Fan Speeds (RPM): cpu: 1331 
Repos:
  No active apt repos in: /etc/apt/sources.list 
  Active apt repos in: /etc/apt/sources.list.d/additional-repositories.list 
  1: deb http://liveusb.info/multisystem/depot all main
  Active apt repos in: /etc/apt/sources.list.d/brave-browser-release.list 
  1: deb [arch=amd64] https://brave-browser-apt-release.s3.brave.com/ stable main
  Active apt repos in: /etc/apt/sources.list.d/official-package-repositories.list 
  1: deb http://mirror.os6.org/linuxmint.com/packages tricia main upstream import backport
  2: deb http://mirror.enzu.com/ubuntu bionic main restricted universe multiverse
  3: deb http://mirror.enzu.com/ubuntu bionic-updates main restricted universe multiverse
  4: deb http://mirror.enzu.com/ubuntu bionic-backports main restricted universe multiverse
  5: deb http://security.ubuntu.com/ubuntu/ bionic-security main restricted universe multiverse
  6: deb http://archive.canonical.com/ubuntu/ bionic partner
  No active apt repos in: /etc/apt/sources.list.d/oguzhaninan-stacer-bionic.list 
  Active apt repos in: /etc/apt/sources.list.d/system76-dev-stable-bionic.list 
  1: deb http://ppa.launchpad.net/system76-dev/stable/ubuntu bionic main
  Active apt repos in: /etc/apt/sources.list.d/teejeetech-aptik.list 
  1: deb https://packages.teejeetech.com/aptik/S7tBVjLBzw stable main
Info:
  Processes: 293 Uptime: 33m Memory: 15.46 GiB used: 1.40 GiB (9.1%) 
  Init: systemd v: 237 runlevel: 5 Compilers: gcc: 7.5.0 alt: 7 Shell: bash 
  v: 4.4.20 running in: gnome-terminal inxi: 3.0.32 

[Lady Fitzgerald]

Anyone?

[Lady Fitzgerald]

Seriously? No one?

[t42]

Sometimes, waiting awhile after TRIMming before trying to safely removing them will avoid the problem. Other times, I have shutdown the computer, then unplug the drives.

awhile - the measure or time that could be of any length
It's obvious that the drive is busy and depends on the volume of storage to be processed.
Just open terminal and issue sync command. Then wait till cursor appears.

Code: Select all

time sync

You can test a measure of delay copying some file about 8 GB to such drive and executing sync after copy starts.

[Lady Fitzgerald]

Sometimes, waiting awhile after TRIMming before trying to safely removing them will avoid the problem. Other times, I have shutdown the computer, then unplug the drives.

awhile - the measure or time that could be of any length
It's obvious that the drive is busy and depends on the volume of storage to be processed.
Just open terminal and issue sync command. Then wait till cursor appears.

Code: Select all

time sync

You can test a measure of delay copying some file about 8 GB to such drive and executing sync after copy starts.

Thank you. I'll give it a shot the next time I'm in this situation.

[Lady Fitzgerald]

Sometimes, waiting awhile after TRIMming before trying to safely removing them will avoid the problem. Other times, I have shutdown the computer, then unplug the drives.

awhile - the measure or time that could be of any length
It's obvious that the drive is busy and depends on the volume of storage to be processed.
Just open terminal and issue sync command. Then wait till cursor appears.

Code: Select all

time sync

You can test a measure of delay copying some file about 8 GB to such drive and executing sync after copy starts.

Thank you. I'll give it a shot the next time I'm in this situation.

I just had 24.9GB worth of updates to make to my backups. I updated them in batches of four drives, then, after FreeFileSync had finished each batch, I ran

Code: Select all

time sync

in a terminal. It finished almost instantaneously and I was able to immediately start safely removing the backup drives without any problems, even with the troublesome NTFS formatted drives. So, that appears to have solved the problem.

Thanks again!

[Lady Fitzgerald]

Well, it didn't solve the problem.

I copied 844GB of data from my laptop's Data1 drive (my Music folder) to my Data2 drive earlier in the day, then started updating my laptop's four backup drives, the four desktop data drives (currently, they are external drives; my desktop data drives and my laptop data drives are identical except for drive name and label), and the first set of four desktop data drives without any problems. I ran TRIM after each set of operations, ran time sync, then successfully safely shut down before removing that set of drives.

However, when I had updated the second set of desktop backup drives, ran TRIM, ran time sync, then tried to safely shut down each drive one at a time as before, all the backup drives unmounted and I got that *&^%$#@! message saying the drive couldn't be stopped. What the heck?

[t42]

So you have time sync output showing delay and bash prompt waiting for next command. Next thing to do is to shut down all programs. I understand you are using file manager to unmount partitions. Close Nemo as well. Open the Disks program from Accessories, select the disk on the left and click Power off button at the top on the left. Before that you can select disk individual partition(s) and click Stop button ("Unmount selected partion").
If no luck then look for background processes involved with your disks, such as Timeshift of some other backup program -- open System Monitor -> Processes tab and kill the suspect, then try Disks program again.

[Lady Fitzgerald]

So you have time sync output showing delay and bash prompt waiting for next command. Next thing to do is to shut down all programs. I understand you are using file manager to unmount partitions. Close Nemo as well. Open the Disks program from Accessories, select the disk on the left and click Power off button at the top on the left. Before that you can select disk individual partition(s) and click Stop button ("Unmount selected partion").
If no luck then look for background processes involved with your disks, such as Timeshift of some other backup program -- open System Monitor -> Processes tab and kill the suspect, then try Disks program again.

Won't using power off potentially cause data loss due to data that hasn't been written to the drive yet?

One thing I haven't mentioned is the last batch of backup drives this happened to are formatted NTFS (I know, not ideal for Mint but they need to be NTFS so, the executor of my will, who uses Windwoes, will have access to my data should the only drives left after my demise, such as from a house fire, are the backup drives in my safe deposit box).

Btw, the previous batch of four backup drives, also NTFS, closed down just fine and dandy so this problem is just weird.

[t42]

Won't using power off potentially cause data loss due to data that hasn't been written to the drive yet?

You can just use Stop button of Disks app for each partition. No data loss is expected if you click Stop as well as Power Off during data transfer - you will get error or wait icon.

[rene]

Well, it didn't solve the problem.

I was already somewhat surprised when you earlier reported that it did...

That is, with sync you instruct the Linux kernel to flush any and all outstanding data to the device but here it seems the issue is one level down from kernel <-> device; that it is the device itself, its firmware, being done (enough) or not with TRIM operations which it handles fully internally without any form of software involvement.

When you submit a bunch of TRIM commands to the device as part of that fstrim run you are telling the device's firmware about blocks that it may consider to be free. Even with synchronous TRIM there's almost certainly no form of guarantee that said firmware will act on that information immediately and with the normal asynchronous fstrim form of TRIM certainly there isn't --- although you'll need to forgive me for not having read the relevant spec at that point so as to know what is and is not in fact guaranteed.

But your SSDs are telling you to hang on while busy doing some of what you just told them to do. Perhaps only in the sense of flushing the information as to free state of blocks which you just told it about from its own internal RAM to flash before it can power down (i.e., power down said RAM); maybe in the sense of it in fact already being busy erasing some blocks and needing to wait for at least a currently proceeding batch to have finished. I once again doubt that this is in fact in any sense guaranteed/specified by a standard.

And that's then in the end to say that I would feel that you'll probably just have to wait on the device; that you probably can't do anything about this: you instruct the device to go do a lot of work and can't then subsequently tell it to power down immediately, potentially even before it has recorded all said work to be done.

[Lady Fitzgerald]

What has me baffled is the previous batch of backup drives that are identical to the last batch gave me no trouble but the last batch did.

Is there any way to for me to know how long after I run TRIM, etc. before I can safely remove the drives without any problems? The problem only happens when a large amount of data is being added, etc. which means I have to deal with it a minimum of once a month after I swap my onsite backup drives with my offsite backup drives. I also only happens on the second set of four NTFS backup drives I've done a massive update on.

I would think that, when telling Mint to safely remove the drive, the notice that often pops up saying to wait because the drive is still being written to would cover the situation instead of unmounting all the drives.

[rene]

I would think that, when telling Mint to safely remove the drive, the notice that often pops up saying to wait because the drive is still being written to would cover the situation instead of unmounting all the drives.

That's the thing most fundamentally; as far as Linux (or Windows, or any piece of software) is concerned all's readily flushed out and acknowledged as having been so by the drive; it's seemingly only on Disks sending an actual STOP command that the drive tells it "Hey now, hang on, not done with all you told me to do yet" but that "not done" state existing only internally in the SSD.

Does it ever happen when you only backup without an immediate subsequent fstrim? If not it's indeed that situation of the drive recording free-block state and/or trimming already; if so it may even be the drive legitimately flushing things from a RAM and/or SLC cache to main storage I suppose.

The unmounting thing doesn't make sense to me either (but Disks generally doesn't make sense to me) and, yes, sure, it would make sense to me if there were some manner to poll the drive for it in fact being really-completely done but I'm not finding anything relevant, and am not able to reproduce locally, undoubtedly due to not in fact having enormous amounts of external USB-connected SSD storage to play around with. Because USB is also a complicating issue here; who knows what one's average USB controller relays and does not relay...

I'd personally be trying if sudo hdparm --idle-immediate /dev/sdz or perhaps sudo hdparm -Y /dev/sdz for the proper /dev/sdz has you wait for a bit before it returns after you write said bunch of data or write said bunch of data and fstrim, or it returning immediately, or it perhaps simply giving an error. It it waits that might mean it's a valid polling method...

[Lady Fitzgerald]

I would think that, when telling Mint to safely remove the drive, the notice that often pops up saying to wait because the drive is still being written to would cover the situation instead of unmounting all the drives.

That's the thing most fundamentally; as far as Linux (or Windows, or any piece of software) is concerned all's readily flushed out and acknowledged as having been so by the drive; it's seemingly only on Disks sending an actual STOP command that the drive tells it "Hey now, hang on, not done with all you told me to do yet" but that "not done" state existing only internally in the SSD...

I'm having "fun" wrapping my mind around this (the sinus headache I have right now is adding to that "enjoyment"). I also may be (more like probably) doing a lousy job of describing what is going on and when.

I'm not using Disks to safely remove the external drives (while Disks is a valuable tool, I infrequently need to use it). As I plug the USB cables connected to the backup drives' enclosures (SSDs) into the computer, the partition on the drive is mounted and it's window opens. To update the backup, I use a folder/file syncing called FreeFileSync (FFS, a glorified file copy program although it does a bit more than just that) that, in Mirror mode (not RAID 1) copies and deletes files as needed to make the backup drive essentially a clone of the drive being backed up. I usually only update the backups of drives I know to have had changes or additions made but sometimes I connect up to four backup drives to the computer at once (one reason I like this laptop is it has the equivalent of four USB ports on the left side) and "tell" FFS to backup all four drives (only one external drive is actually "active" at any one time), then leave it alone to do its until it alerts me with a loud "ding" that it has finished. It will then also alert me to any files that failed to copy over (that hasn't happened in a long time). If the amount of data involved in the update is massive (such as the 800+GB one most recently), I'll also run the fstrim command for those drives (I have aliases made for each combination of drives needing TRIMing rather than laboriously typing them all out, risking screwing up with a typo, the latter being something I excel at).

I safely remove the external drives, one at a time, by right clicking on the drive's name in the left panel of that drive's open window, then click on Safely Remove Drive. Normally, a notification window will appear (I have them set to appear in the lower right hand corner where it's easier for me to see them) that will say that the disk is still being written to and to not unplug it. A few seconds later, the notification window will be replaced with one saying it is now OK to unplug the drive. Sometimes, it jumps straight to the second window. The open drive windows are stacked full screen on top of each other so I start with the open window on top and, as a drive is removed and its window disappears, I repeat the process with the next window on top until all the drive have been safely removed, then I unplug them all.

I normally TRIM all my laptop drives (internal and external) on Monday mornings when I first fire up the laptop. For smaller backup updates, say up to 100GB, usually much smaller, I don't bother with TRIMing afterwards until Monday rolls around again.

....Does it ever happen when you only backup without an immediate subsequent fstrim? If not it's indeed that situation of the drive recording free-block state and/or trimming already; if so it may even be the drive legitimately flushing things from a RAM and/or SLC cache to main storage I suppose...

The problem happens infrequently enough I don't recall if I ever skipped TRIMing immediately after an update (at my age, the only thing I retain well is water ). I usually have massive updates only after I have swapped out my onsite backup drives with my offsite backup drives, then update the latter drives; that happens only once a month or so (or I have a massive number of CDs or DVDs to rip; that happens infrequently). I'm due to make the swap in the next week or so (mayhap even tomorrow) so I'll make it a point to just update the backups (and that will be a doozy of an update!) and skip TRIMing afterwards until the following Monday when I normally do it. I'll post back here when that happens.

So far, this problem has never happened when doing my normal daily backups of my laptop drives (and I update the backups only on drives I've modified during the day, usually only one or two of them). The desktop drives and their backups that I'm maintaining while between desktops usually get updated only just before I do a backup swap or after I've added or otherwise modded a large amount of data. Ironically, it's the last four of the desktop backup drives (both sets of four being NTFS) where this problem rears its ugly head (the desktop drives themselves are ext4).

[rene]

I'm not using Disks to safely remove the external drives. [ ... ] I safely remove the external drives, one at a time, by right clicking on the drive's name in the left panel of that drive's open window, then click on Safely Remove Drive.

Ah, okay. Unfortunately doesn't yet change anything about this probably being a lost cause though; only disallows me dumping on Gnome Disks a bit more. Which is clearly a shame but which I shall live with.

It's not hugely important whether or not this happens only after trimming or not; the point is just that on being instructed to in fact stop the SSD is telling the computer that it can't yet and/or at that precise moment since it needs to finish some part of the tasks that you told it to do first.

As said, I doubt you can do much about it other than wait "long enough" but you may try if instead of sudo sync, saying

Code: Select all

sudo hdparm --idle-immediate /dev/sdz

or sudo hdparm -Y /dev/sdz helps; if after that returns to a prompt you can always safely remove the drive /dev/sdz without that issue.

"/dev/sdz" in this is the identifier that the external drive was assigned by Linux when you plugged it in; being not very GUI-oriented I can't in fact compactly tell you how/where to see what that is graphically if you don't know. You'd also eventually want to do this for all four drives with a loop or some such, but given that I quite doubt that it works at all let's not go there yet.

In essence then, I do believe you may simply be out of luck as to doing something fundamental about this, i.e, other than "waiting long enough", or at the very least, certainly I personally don't know how to poll an SSD for shutdown readiness and Google isn't telling me either.

[t42]

I personally don't know how to poll an SSD for shutdown readiness and Google isn't telling me either.

It seems you're in a good company with the kernel.

[rene]

Well, actually, the kernel as such seemingly has in the end little issue with this; it needs to of course on shutdown make sure that the SSD is ready to have its power delivery yanked out from under it. Even if only perhaps by trying to stop it until it no longer objects which is also more or less the point/idea of my hdparm notion.

Anyways; before this gets too far let's wait and see if I managed to make enough sense to have OP still interested.

[t42]

it needs to of course on shutdown make sure that the SSD is ready

and that's my point, controllers are lying too often and kernel developers being aware still have not much choice (I agree it's off-top here)

[Lady Fitzgerald]

I'm not ignoring you guys (despite this stupid headache from Hell). I'm just waiting until I do the next massive update, which will be after I swap out my offsite and onsite backup drives. I do appreciate the comments even if some of it is going over my aching head right now.

[rene]

In that case, let me add that Disks in fact seems to display the device nodes (weeee, a use for Disks....) so where I said "/dev/sdz" I aim for you to replace that with the device node(s) as shown by Disks as "Device", with a trailing number stripped. I.e, if it shows an external drive as /dev/sda or /dev/sda1 you use /dev/sda.