Removable USB storage

Forum rules
Before you post please read this
mmmna
Level 1
Level 1
Posts: 7
Joined: Sun Jul 30, 2017 3:22 pm

Removable USB storage

Postby mmmna » Sun Jul 30, 2017 4:32 pm

Hey, hello!
My first post here.

I've been using Linux maybe 18 years, many distros, but not really deep technically speaking, more as a user.

Just installed and am running Mint 18.2 Mate 64bit.

For the record, I have 2 USB to SATA setups, a USB to IDE box, External USB DVD drive, External USB CDRW, some SD cards, some Compact Flash cards, 3 xD cards and I think 10 USB Flash drives. I work with multimedia fairly often, such is my personal life

I have just removed a fledgling Debian derived distro where the USB devices suddenly became no longer user accessible, after a month of what seemed to be normal permissions. I can't tolerate a distribution randomly changing policies on my devices when all I did was perform a security driven system update, and I was already dissatisfied with the desktop environment options they had. I had tried Mate on that distro, but theirs was Mate 1.08, from 3 years ago. Not a great situation to be so far behind.

I had many good reports about Mint in another forum and Mint offered a current release of Mate on a 64 bit release, so ... how could I lose? I got my files relocated from the hard disk onto my flash drive despite the sudden resistance of the former distro. I ran dd to put the Mint ISO onto another flash drive and proceeded to test the Mint 18.2/Mate 64 bit live session. Very appealing, visually, although I do prefer multi-color icons over monochromatic icons.... I'll tweak that later.

After installing Mint/Mate and logging in as a normal user, I wanted to get used to the new environment and soon, I inserted one of my USB flash drives.

I got pretty scared. It mounted automatically, according to mount command.

Umm, what about those USB devices which have a dirty bit set? Mint/Mate just mounted the dirty device, so I'm worried: will system atime operations screw my data? I'd REALLY prefer to not have the device mounted upon insertion, since I already know the former distro may have borked some of the filesystems.

I'm thinking, lets look into administering this, lets see what tools I have at my disposal in the crisp new distro!

I visited the Mate Control Center, found the disks section, and began testing if I could shut off the magic.

I'm looking at what is happening and what is presented to me, and thinking: "Dear god, when and why did this matter suddenly get so flipping complex?". The best I was able to discover SO FAR, was that I could reconfigure each individual device to NOT automount, but I can only configure after the device has been automounted for the first time to create the entry in the control center for me to edit it. No broad policy statement. Device by device. Same dirty bit issue, now for every writeable device I own.

Rather than continue on my own, I came here hoping there might be direction to a concise, on point discussion (or maybe a how-to) regarding properly and topically adjusting these policies. More, I'd REALLY prefer to not have to SUDO at a commandline or CHOWN every mount point just so I can write my work files to the device I just inserted. Yes, I heard rumblings about removable devices being a security topic maybe 8 years ago (we could open the discussion well after this situation gets addressed), but today, in Mint/Mate, I now have what looks like a great tool and I'd like to get the configuration set right, straight right away. I am the ONLY user of this system, no accounts (other than root are configured) and my plan is there never will be more than these.

I see this topic viewtopic.php?f=90&t=250280&p=1345737&hilit=removable#p1345737 and can do that, as a start.

Ok, so here it is, this is the main point of my post: Anywhere I can turn to get an overview of how to get my removable USB device access under control?

I will also be looking for labelling to not say silly stuff like UUID but I think the control center tool should be enough.

So, if anyone has a better clue than what I have, I'd really appreciate making these issues right, but specifically, the writing permissions.





-Just an observation: when the installation script asked me if I wanted to install proprietary drivers (I installed from a live session that used ethernet for functional internet access), I was kinda hoping my systems Broadcom BCM4318 WiFi card would get set up for me. Not so. Kubuntu around 14.04 used to set it up. Just saying I was hoping.

-Another thought: After a fresh install, I've learned it is good practice to update the newly installed packages for security fixes.... Yesterday was a bit tough to take, as Synaptic seems to have lost a feature labelled 'Select All Updates', meaning I had to manually set 161 files for updating. Ahh, yesss! Thank you, whoever, for the less than useful modification.

mmmna
Level 1
Level 1
Posts: 7
Joined: Sun Jul 30, 2017 3:22 pm

Re: Removable USB storage

Postby mmmna » Mon Jul 31, 2017 8:12 pm

Ok, 2 dozen views, no comments.

Here is what I have presently done and am actively doing.

I have added the suggested USB rule as found here but, after adding the rule, it seemed to have had no effect for the next USB flash drive I added (the device is owned by root, cannot be written by user). The UDEV rule (well, actually the file and the rule in it) remains, but the following occurs in spite of the rule. Will probably remove the UDEV rule in a bit.

Here is what I see....
Insert USB media, wait for device activity LEDs to cease blinking.
Open a terminal as user.
dmesg | tail to see if any device was recently seen.
[I could lsblk, if needed]

I open the Control Center - Hardware - Disks tool.
I click on the device I just inserted.
I click the gears button, select the 'Edit Mount Options' menu item.
I adjust entries until I get something close to the following:
  • Automatic mount options button = OFF
  • Mount at startup = unchecked
  • Show in user interface is checked
  • Require additional authorization to mount is unchecked
  • Greyed out are
    • Display name,
    • Icon Name
    • Symbolic Icon Name
  • beneath the 'Symbolic Icon Name' is an unlabeled field with 'nosuid,nodev,nofail,x-gvfs-show,noauto'
  • Mount point entry will be determined from the following 'Identify As' selection
  • Identify As section is set for LABEL=%Label from the device filesystem% as per what I entered when I partitioned the device
  • Filesystem Type is set to auto
I click the ok button.
I then have to return to this screen to MANUALLY mount the USB device (activating the Automatic Mount Options button simply greys the rest of the setting window, meaning the OS will do what it wants; no, Automatic isn't going to be turned on any time soon).

Just one step too cumbersome, but at least partially functional.

Problem is, the mount points are created as root.root, and that is where the next problem hits me.

To gain read/write access to the device, I need to go to a terminal, and 'sudo chown -R me.me /media/me/mountpoint'. For every new device I insert.
Yes, once I get them all through this process once, then they will all be good to go..... Only 12 more USB devices left to edit...Only 11 more USB devices to edit... Oops, bought a new 512GB for the next project... Only 12 more USB devices to edit...



Any suggestions so far?

mmmna
Level 1
Level 1
Posts: 7
Joined: Sun Jul 30, 2017 3:22 pm

How do I disable the automount for removable USB storage?

Postby mmmna » Tue Aug 01, 2017 6:34 pm

Hmm.

Only lasted until I rebooted, all USB flash drives are auto mounting again.

I've tried using what appear to be the tools which Mate has provided for this exact purpose, the tool which creates settings does not persistently modify the USB device mounting behavior, even though I must 'authenticate the changes' with root password.

I do not want this behavior.

User avatar
coffee412
Level 5
Level 5
Posts: 690
Joined: Mon Nov 12, 2012 7:38 pm
Location: Indiana, USA
Contact:

Re: Removable USB storage

Postby coffee412 » Tue Aug 01, 2017 7:57 pm

Ok, I had to re read your first post a few times to get what your trying to do. Forgive me if I still do not understand what the problem is.

You have multiple devices with dirty file systems on them. So, In fear of losing data you do not want the O/S to automount the devices in case it screws up what you have.

Instead of going thru all this why do you not just backup your data on the devices and just blow out the partition and lay a new one down? Then its clean and no problems and it would not matter if it auto mounted them?

To me, The real problem is that you have dirty file systems that need to be corrected and I will venture to say that you also do not backup the data.

I would grab a couple of mechanical drives and just mirror them and back up all the devices. Then I would simply reformat.

IMHO,
Ryzen x1800 Asus Prime x370-Pro 32 gigs Ram RX480 graphics
IceWarp * Samba AD * Mint 18.1 * RAID 1/5 * OpenVPN * Linux since kernel 2.0.36

Mute Ant
Level 12
Level 12
Posts: 4116
Joined: Tue Sep 03, 2013 7:45 pm

Re: Removable USB storage

Postby Mute Ant » Sun Aug 06, 2017 10:40 am

Mint starts a daemon udevd which sends messages to the GUI when devices appear or vanish. Your desktop responds by showing a CDROM is present or a USB store has been plugged in. You can disable all offers of automatic mounting in any desktop by disabling the udev daemon...
sudo udevadm control --exit
...leaving you in full-manual control. You can re-start the daemon with the command...
sudo udevd & disown
KDE and XFCE offer equivalent functionality with tick-boxes to signal Do Not Automount. Devices you plug in will still appear and vanish in the /dev/ folder, the kernel and udev do that without the daemon running. You have to mount them and unmount them manually. With udevd switched off, you have a chance at check-and-repair without any writing to the store. Using gparted is good for that, it has the abilities of the console commands with point-and-click selection.

Support for damaged NTFS is very poor in Linux, little more than a read-only consistency check.

By design Linux won't make a file-system more damaged unless you do something provocative, like forcing a fsck on a mounted file system. If a file-system is inconsistent, it flips to read-only regardless of how you mount it.
Armageddon only happens once, you know. They don’t let you go around again until you get it right.

mmmna
Level 1
Level 1
Posts: 7
Joined: Sun Jul 30, 2017 3:22 pm

Re: Removable USB storage

Postby mmmna » Tue Aug 08, 2017 7:03 pm

coffee412 wrote:You have multiple devices with dirty file systems on them. So, In fear of losing data you do not want the O/S to automount the devices in case it screws up what you have.
Close. My preference for NOT automounting them is because I want to be informed if they are dirty! Mint/Mate silently denying me access to the device with no mention of a problem, That is about 3 steps short of Mint/Mate being what I would say was 'clearly beneficial'.
1] I need to be told they are dirty,
2] I need to be asked if I want the devices magically mounted AT ALL (sure, if you want to, Mint/Mate, go ahead and display the icon in the desktop environment, but do NOT officially mount it), and
3] I need to be offered the opportunity to repair the devices. Don't just pop them open and silently refuse to let me add more records. I'm saying the OS needs to explain the situation!

Instead of going thru all this why do you not just backup your data on the devices and just blow out the partition and lay a new one down? Then its clean and no problems and it would not matter if it auto mounted them?
Easy fella. You would think I only had a hundred files? One device held something like 16 thousand files. 4-5 megabits per second, this took MANY hours to copy the files off the damaged filesystem, then you have the not really trivial effort of repartitioning, reformatting and then.... 4-5 megabits per second to put the 16 thousand files back onto device, which is taking hours more. The problem happened mostly because it was Linux Mint 18.2 which made some of the devices dirty. Another distro did cause some of the problems, but Mint/Mate certainly caused at least 2 bad ejects, We will quickly begin the discussion where some devices and some screens in Mate were NOT offered a 'safely remove' option, and instead the only offer from Mint/Mate is 'Eject', and THAT made 2 of my devices dirty. Not sure why, but some were honestly safely removed in Devuan before I installed Mint 18.2, and then they are dirty in Mint Mate but I'd never have known that by reading Mint/Mate's manual.... there is NO discussion about the read only aspect of automounted removable media. Maybe you should contribute the explanation process for us, coffee? Maybe read this post but don't respond.

To me, The real problem is that you have dirty file systems that need to be corrected and I will venture to say that you also do not backup the data.
Wrong guesses. A 320Gig USB hard disk IS my backup, but with the way things were going with the removable USB devices in Mint/Mate, I was NOT about to mount THAT device and have issues with it; no, not until I got help about the way Mint Mate was supposed to work handling removable media. Intelligent rule #1: if you think your OS is screwing with your disks, you do NOT attach your primary archive. 'Venture' all you want, coffee, but this part of your post is incorrect and borderline caustic.

I would grab a couple of mechanical drives and just mirror them and back up all the devices. Then I would simply reformat.
And thus I repeat the dirty eject problem ad infinitum? Eject, insert, read only, dirty, fsck repairs but OS doesn't adjust permissions, copy the thousands of read only files, repartition, reformat, write thousands of files....eject, insert, read only, dirty files, fsck repairs but OS refuses..... are you catching on yet, coffee?

I have since recovered the dirty media, and I had to wipe the partitions on the devices, because 'fsck -p' issued as root had initially reported the matter of being dirty, fsck fixed the problem and yet I was never again able to mount the device as read write; the read only state remained after several rounds of 'fsck -pv' which reported no problems. The OS is not recognizing the fsck repairs. When I changed the partitions to EXT3, I was able to read and write, but I only tried accessing the devices ONCE until now.

Next....

mmmna
Level 1
Level 1
Posts: 7
Joined: Sun Jul 30, 2017 3:22 pm

Re: Removable USB storage

Postby mmmna » Tue Aug 08, 2017 7:18 pm

Mute Ant wrote:Mint starts a daemon udevd which sends messages to the GUI when devices appear or vanish. Your desktop responds by showing a CDROM is present or a USB store has been plugged in. You can disable all offers of automatic mounting in any desktop by disabling the udev daemon...
sudo udevadm control --exit
...leaving you in full-manual control. You can re-start the daemon with the command...
sudo udevd & disown
Fantastic, Big help there. It will be nice to have a strong arm tactic to 'adjust' udev. Was always afraid of modifying that thing, and anybody I asked until now was unaware of what to do.
KDE and XFCE offer equivalent functionality with tick-boxes to signal Do Not Automount. Devices you plug in will still appear and vanish in the /dev/ folder, the kernel and udev do that without the daemon running. You have to mount them and unmount them manually. With udevd switched off, you have a chance at check-and-repair without any writing to the store. Using gparted is good for that, it has the abilities of the console commands with point-and-click selection.
The disks tool in Mint/Mate seems to offer some of that, but right clicking the desktop icon and the disks tool do not both offer safely remove under all circumstances for a given device in a single mount.
Support for damaged NTFS is very poor in Linux, little more than a read-only consistency check.
In regards your mention of NTFS, you wouldn't know from my posts, but I have never allowed NTFS on any system including Windows, not on any device I've partitioned, never formatted any partition as NTFS.

My removable media has been mostly FAT32 file systems as they are delivered FAT32 when new, with my few non-flash devices (the USB based spinning platter hard disks) which were formatted EXT3 or 4.

By design Linux won't make a file-system more damaged unless you do something provocative, like forcing a fsck on a mounted file system. If a file-system is inconsistent, it flips to read-only regardless of how you mount it.
Yet nothing explained that was what had happened, especially for the devices which Mint/Mate affected.

Good discussion in your post, I appreciate it.

User avatar
coffee412
Level 5
Level 5
Posts: 690
Joined: Mon Nov 12, 2012 7:38 pm
Location: Indiana, USA
Contact:

Re: Removable USB storage

Postby coffee412 » Tue Aug 08, 2017 8:23 pm

mmmna wrote:
coffee412 wrote:You have multiple devices with dirty file systems on them. So, In fear of losing data you do not want the O/S to automount the devices in case it screws up what you have.
Close. My preference for NOT automounting them is because I want to be informed if they are dirty! Mint/Mate silently denying me access to the device with no mention of a problem, That is about 3 steps short of Mint/Mate being what I would say was 'clearly beneficial'.
1] I need to be told they are dirty,
2] I need to be asked if I want the devices magically mounted AT ALL (sure, if you want to, Mint/Mate, go ahead and display the icon in the desktop environment, but do NOT officially mount it), and
3] I need to be offered the opportunity to repair the devices. Don't just pop them open and silently refuse to let me add more records. I'm saying the OS needs to explain the situation!

Instead of going thru all this why do you not just backup your data on the devices and just blow out the partition and lay a new one down? Then its clean and no problems and it would not matter if it auto mounted them?
Easy fella. You would think I only had a hundred files? One device held something like 16 thousand files. 4-5 megabits per second, this took MANY hours to copy the files off the damaged filesystem, then you have the not really trivial effort of repartitioning, reformatting and then.... 4-5 megabits per second to put the 16 thousand files back onto device, which is taking hours more. The problem happened mostly because it was Linux Mint 18.2 which made some of the devices dirty. Another distro did cause some of the problems, but Mint/Mate certainly caused at least 2 bad ejects, We will quickly begin the discussion where some devices and some screens in Mate were NOT offered a 'safely remove' option, and instead the only offer from Mint/Mate is 'Eject', and THAT made 2 of my devices dirty. Not sure why, but some were honestly safely removed in Devuan before I installed Mint 18.2, and then they are dirty in Mint Mate but I'd never have known that by reading Mint/Mate's manual.... there is NO discussion about the read only aspect of automounted removable media. Maybe you should contribute the explanation process for us, coffee? Maybe read this post but don't respond.

To me, The real problem is that you have dirty file systems that need to be corrected and I will venture to say that you also do not backup the data.
Wrong guesses. A 320Gig USB hard disk IS my backup, but with the way things were going with the removable USB devices in Mint/Mate, I was NOT about to mount THAT device and have issues with it; no, not until I got help about the way Mint Mate was supposed to work handling removable media. Intelligent rule #1: if you think your OS is screwing with your disks, you do NOT attach your primary archive. 'Venture' all you want, coffee, but this part of your post is incorrect and borderline caustic.

I would grab a couple of mechanical drives and just mirror them and back up all the devices. Then I would simply reformat.
And thus I repeat the dirty eject problem ad infinitum? Eject, insert, read only, dirty, fsck repairs but OS doesn't adjust permissions, copy the thousands of read only files, repartition, reformat, write thousands of files....eject, insert, read only, dirty files, fsck repairs but OS refuses..... are you catching on yet, coffee?

I have since recovered the dirty media, and I had to wipe the partitions on the devices, because 'fsck -p' issued as root had initially reported the matter of being dirty, fsck fixed the problem and yet I was never again able to mount the device as read write; the read only state remained after several rounds of 'fsck -pv' which reported no problems. The OS is not recognizing the fsck repairs. When I changed the partitions to EXT3, I was able to read and write, but I only tried accessing the devices ONCE until now.

Next....


Easy Fella?? You mean asking simple questions is not allowed with you?

You have mistaken the tone of my message. My post was to get a better understanding of your problem. I had a few questions pertaining to your problem. You know, Like when you first posted expecting someone to help you out. So, Fine. Best of luck with your problem I hope you get things fixed.

I really do not care for your tone when someone has a question and is going to attempt to help you. I will just leave this thread and let others work with you.
Ryzen x1800 Asus Prime x370-Pro 32 gigs Ram RX480 graphics
IceWarp * Samba AD * Mint 18.1 * RAID 1/5 * OpenVPN * Linux since kernel 2.0.36

User avatar
MintBean
Level 8
Level 8
Posts: 2476
Joined: Fri Aug 07, 2015 6:54 am
Location: Blighty

Re: Removable USB storage

Postby MintBean » Tue Aug 08, 2017 8:34 pm

The original post is too long and rambling. That's why you're not getting much response. Clear concise questions get answers.

mmmna
Level 1
Level 1
Posts: 7
Joined: Sun Jul 30, 2017 3:22 pm

Re: Removable USB storage

Postby mmmna » Fri Aug 11, 2017 6:27 pm

Since I rambled too much by asking one question in my first post, I've uninstalled.

User avatar
catweazel
Level 10
Level 10
Posts: 3334
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Removable USB storage

Postby catweazel » Fri Aug 11, 2017 11:09 pm

mmmna wrote:Since I rambled too much by asking one question in my first post, I've uninstalled.


lol
If your problem is fixed, please mark your thread as [SOLVED] by editing the title of the first message in the thread.

Mute Ant
Level 12
Level 12
Posts: 4116
Joined: Tue Sep 03, 2013 7:45 pm

Re: Removable USB storage

Postby Mute Ant » Sat Aug 12, 2017 8:14 am

This page is quite useful as background-reading...
https://github.com/rbrito/usbmount
o Auto-mounting and unmounting devices without the udevd running.
o Using udevadm to re-detect a device that has been safely removed, without needing to unplug it.
...although I don't use usbmount or pmount myself.

Changing distributions will probably not help...they all use Linux and udev...whatever is going wrong will follow you...Debian and ext4 are two of the best. I suspect it's simply the default kernel parameters for disk-write cache, which you can investigate on a running system...

### How big is a disk-write cache (percent of the total physical memory)...
sudo cat /proc/sys/vm/dirty_ratio
...I get '20'. With a 3GB RAM that means writing to my USB Flash is allowed to cache 20% of 3GB == 600MB before data must be sent to the drive.

I use USB3 devices through a USB2 port and get a sustained-write rate of 10MB/s. So after a large copy or writing an ISO to the USB store, there's still 60 seconds of writing to do before the data is actually in the drive...and the drive itself has a RAM buffer. Are you allowing minutes for a device to 'eject'. It's not reasonable, but it is the default kernel setting...which you can change...

### Set the cached write-to-disk data size to 10MB
sudo su -c "echo 10000000 > /proc/sys/vm/dirty_bytes"

...and that change can be made permanent with a line in /etc/sysctl.conf if you like. Even USB1 devices will be safe in a reasonable time...count 10 seconds if there's no activity-lamp on your stick.
Last edited by Mute Ant on Sun Sep 10, 2017 11:13 am, edited 1 time in total.
Armageddon only happens once, you know. They don’t let you go around again until you get it right.

mmmna
Level 1
Level 1
Posts: 7
Joined: Sun Jul 30, 2017 3:22 pm

Re: Removable USB storage

Postby mmmna » Sat Sep 09, 2017 5:32 pm

Mute Ant wrote:This page is quite useful as background-reading...
https://github.com/rbrito/usbmount
o Auto-mounting and unmounting devices without the udevd running.
o Using udevadm to re-detect a device that has been safely removed, without needing to unplug it.

Thanks, will probably be useful in the future.

Changing distributions will probably not help...they all use Linux and udev...whatever is going wrong will follow you...
I'll have to ramble a bit: I need to use my OS, not get a bachelors degree in administering it. Years of getting installations done right, and from what is written here Debian suddenly makes a configuration change, the change propagates to nearly all derivatives just because Debian said so, and now we are the ones with issues? Tens of thousands of users of Debian and derivatives need to recover? Well, be that as it may, I asked for understanding from the forum of the distro I had installed, once upon a time. Or maybe my question related to Mate - Control Center - Discs?


Did not ask:
### How big is a disk-write cache (percent of the total physical memory)...
### Set the cached write-to-disk data size to 10MB


As an observation: using a 3 year old distro on the same hardware, when issued at a command prompt, sync had rarely needed more than 2 or 3 seconds before restoring the prompt. Maybe syncing after writing isn't at the heart of what I wrote about.

But I've rambled again.

Part of the recent problems I've seen have to do with how Mate is the cause of mounting the USB disks. The Mate control center defines the parameters for deeper system functions as you suggest, and I wanted to get the Mate control center to suit my needs. After all, power is nothing without control.

Thus, getting back onto my topic, I guess forum readers have already discovered how to configure USB mounting under Mate - Control Center - Disks. I still don't quite grasp the answer, though.


Return to “Mounting Partitions”