Best Practice - storing large amounts of data

[TaterChip]

Lady Fitzgerald and TaterChip - Thank you both.
I started using a separate internal drive in my laptop, and will add an external drive for storage.I'll just keep newly downloaded images on the internal drive, moving them after I get around to processing them.
Both of you have some great insight. I'm not new to computers or photography, but any other trade or skill, I'm always learning!
Thanks!

You're welcome 3pinner

From a photography standpoint here is the file structure that I have found that works best for me. I have tried several different file labeling formats, it seems that for at least the last several years I have settled on this one. I am lucky enough that I get to travel the country as a nature photographer, so my folder labeling may be a bit different from others.

I start off with a year folder "2008", then drill it down from there. Below is an actual shoot folder.

2008-04_IL_SnakeRoad_CottonmouthPortrait

I start with the year-month_State of shoot_Location of shoot_description of shoot. Starting with the year-month puts all the shoot folders into a loose chronological order within the main year folder.

This naming convention allows me to see at a glance what's inside the folder. Since I am in the process of switching to Linux full time, I have started using DigiKam (instead of lightroom) to pull keywords from the images, and help me find the exact images that I am looking for. My file naming system, and now using DigiKam seems to be a great combination for finding images fast.

As it stands, I am currently working with 53,988 Files and 1,023 Folders. I doubt I could be as efficient without my system of naming and DigiKam sorting by keywords. Those numbers used to be a lot higher, but I recently went and removed all the RAW files that would never get processed for one reason or another.

I USED to do my photography like you're doing yours. I would put new shoots on my internal secondary drive. Then one day I forgot to backup and to compound things tragedy struck. I went to boot my computer (winXP) and it wouldn't boot. After reinstalling the OS, I went to access my secondary drive and it was unusable. I couldn't get the computer to read it. I can only speculate what happened. What I do know, is after I copied the shoot from the 3.5 floppy used by my Sony Mavica digital camera I formatted the disk. So now that shoot was lost forever. Back then, photography was a hobby instead of part of my business.

That's when I started looking for other ways to do things... lesson learned the hard way.

Now I have an external "working" photography drive where I start off with two folders. Processing and Portfolio. All new shoots go into processing and when done they get moved over to portfolio. The portfolio folder houses the file naming structure shown above.

When I am done for the day, I now use FreeFileSync to mirror the "working" photography drive (with versioning enabled) over to the first external backup. Once complete, I run it again over to the second external backup. For me this simplifies things, since I don't have to change the source location and I can backup everything in one step.

Hope this helps.

Thank you.
I use a similar folder system. I use Rapid Photo Downloader, it creates folders by date for me. I can also add location and subject that becomes part of the folder name. I use Darktable for processing. When I load the RAW images into Darktable, I can there also add ratings and description tags that are added to the image data. That way I can search by date, subject, location, or whatever tag I generated. I don't have any where near as many photos as you do, but I'm planning on it. What I was looking for is how do people like you handle a potentially vast number of files, and this system looks good

freeFileSync - between you and Lady Fitzgerald, I will look into that this week.

As for separate storage ,i am convinced now. I am going to see how I can implement that into my system. Problem is - I just spent a boatload on drives for my wife's desktop. I'll probably steal one of the new 4TB drives from her, make that my exterior Processing/Portfolio drive, and just get an exterior drive for her and backup whatever she has.

I don't want hijack my own thread any further so we could discuss this elsewhere - have you tried Darktable for sorting & processing?
thanks again!

will send you a DM so we don't hijack any further.

[red-striped-zebra]

I agree with your opinion that the default OS setup procedure should cater to a worse case scenario however I do not believe the current procedure of mixing personal data files with System files is the way to do so. I don't see why the default setup procedure can't include an option to set up a separate data partition during installation. And where did you get the idea I suggested disregarding the capacity of a drive? I never said any such thing.

This.

Makes it so much easier. For some reason, they don't mention /add a tutorial at the download page instructing the user to separate the root from personal files once the installation is done. I am now reading in places, it is preferable to at least have 20 gb separated for the OS alone. And now I don't know what to do (besides removing older kernel packages) once the single drive I have begins to fill up. Should I back up/reinstall or is there a way to partition the existing set up?

[mikeflan]

By 'mounted', you mean here in layman terms, your external is always connected by wire to the main computer.

By 'mounted' I mean it is an internal auxiliary drive (one of three) that is connected to the motherboard by a sata cable. I don't automount my auxiliary drives. I manually mount all 3 every time I reboot. It takes about 5 seconds and I reboot very rarely - probably less than 20 times a year.

[3pinner]

I agree with your opinion that the default OS setup procedure should cater to a worse case scenario however I do not believe the current procedure of mixing personal data files with System files is the way to do so. I don't see why the default setup procedure can't include an option to set up a separate data partition during installation. And where did you get the idea I suggested disregarding the capacity of a drive? I never said any such thing.

This.

Makes it so much easier. For some reason, they don't mention /add a tutorial at the download page instructing the user to separate the root from personal files once the installation is done. I am now reading in places, it is preferable to at least have 20 gb separated for the OS alone. And now I don't know what to do (besides removing older kernel packages) once the single drive I have begins to fill up. Should I back up/reinstall or is there a way to partition the existing set up?

I agree with both of you!
The installation instructions should include the options to add an additional partition or an additional hard disk.
As well as a very clear and simple instructions on how to set the permissions for an exterior or interior hard disk.
I don't make those kind of changes often enough to remember all the steps, and when I go to research them I find way too much information.

As far as hard drives filling up- in my case my wife had a ton of photos. It was a 1 TB drive. Adding a new partition onto an existing drive that has data on it is very dangerous. It can be done but there's a real danger of losing data in the process.
I just bit the bullet went and bought a new one terabyte drive and a new 4tb Drive. The one terabyte drive will be for the operating system with a 50 GB partition, and the rest of it will be used for /Home. In her case she doesn't generate much data other than the photographs. The 4 TB Drive will be used exclusively for storing photos videos etc. What I was looking for here was to see how more experienced photographers organize and store potentially thousands of images.

When you install Linux mint with the defaults, it usually sets the home as a separate partition. The beauty behind it is if you need to upgrade the OS you just format and install the OS on the root Drive and leave the home partition alone.

To keep my home partition clear, I go through it on a regular basis and just get rid of stuff. In your case if you have a lot of valuable stuff on your home folder that you need to keep you could always get another Drive, either internal or external, and move items to it, just regularly clear out space on your existing home folder.

I posted a question in another thread on moving the home folder to a separate drive. It seems some people do this and I have mixed feelings about it now. I've heard both it works well, or it causes an issues with how the system runs.
So I thought it was better just get a larger drive for my root and home partitions, and have my photos live elsewhere!

[mikeflan]

When you install Linux mint with the defaults, it usually sets the home as a separate partition.

I'm thinking when you install Linux mint with the defaults it always puts the home folder in the root, so at /home.

[DisturbedDragon]

I personally put all critical data on a separate redundant RAID array as well as back that up somewhat regularly to a USB3 four drive enclosure. Getting ready to move from the USB storage to a new backup solution which is a separate TrueNAS box with zRAID drives. Data will autosync between those devices on a schedule. All in all we're talking around 12TB of family pictures, videos and documents. Any data on /home for me at least is not all that important but still backed up semi-annually to a USB SSD. Timeshift takes care of / regularly to a separate, small internal SSD.

No matter where the data is stored. I suggest at least one (regularly run) backup copy on a separate drive.