This is the first of a series of articles or blog for that matter that looks into how to manage the ever increasing amount of data from cameras be it photos and videos which clogs up our desktops, laptops and smartphones.
I wrote this as a wider educational sharing with my photography class participants as there are questions pertaining to data management of photos and videos. If there are any questions I am ready to answer. Just give me a message through here or connect through Facebook.
Quick Overview – How I Save and Archive Image Files
Whenever I save the files from the camera or the phone, I will typically start by finding space in my secondary (Desktop) or external hard disk drives (Laptops) and then create the folders and copy those images to the folders. Do note that I am writing from the perspective of using Lightroom as the app to import from the Desktop, edit and export them to the respective folders. You can do the same through other Lightroom alternatives like the free Darktable App for desktop.
Find space in your Secondary Drive (NOT C:\ drive) or External HDD drive
Create the main folder that describes the time, genre and place or even clients’ name (see naming convention below)
The sub-folders that hold my still images and videos separately (more of that below)
COPY and Paste (not move) the files to the respective sub-folders
Import (add) these images to the database within Lightroom.
Choose the photos and videos within Lightroom by ranking or labelling them
The chosen files are EXPORTED as the first unedited copies into a “Selection” folder (optional as some clients wants unedited photos)
The same chosen files are edited and EXPORTED into “Selection.Edited” Folder
The “Selection” files are copied to NAS drive for save keeping/archive from brands such as Qnap and Synology.
The “Selection.Edited” files are uploaded to Flickr Cloud Service as archive as well as portfolio use.
Simplified explanations to the steps. Diagram from Wilson Wong
First Line of Defence – Acquiring Additional Hard Disk(s)
In a computer system there are always a main working storage device that has the operating system such as MacOS or Windows. In the past, manufacturers used a Hard Disk Drive or HDD as their main drive (it is C:\ drive because A:\ and B:\ drives used to be reserved for removable media like floppy disks and CD drive. If you don’t know, is okay because that means you are young!)
SSDs are similar to your typical SD memory card, they are much faster and are preferred over the HDDs when using it as the main operating system drive. Good news is that most laptops and desktops now comes with SSDs. The newer types of SSDs is called Non-Volatile Memory Express NVMe that takes up even lesser space in the desktop and is faster than SSDs as they use a faster connection within the desktop.
The HDD or Hard Disk Drive is still the prefered medium to save huge files because it is cheaper. Photo by Pixabay on Pexels.com
It is good to look at HDD for your on-site (means at home or home office) storage that is separate from your main operating system drive. Due to the cost of SSD and NVMe, it is more economical to save your image files in a separate physical drive using the HDD.
Tip 1: To make sure that your images and videos are safe, store them in another HDD in the desktop or in the case of the laptop, an external drive or into a NAS.
Besides, if anything happened to your main drive, the secondary drive with the images and videos will have a higher chance of survival. If the drives are in two separate locations, it will be even better.
This is perfect for portable use but with 1TB storage it does not has quite enough space for images and videos for archiving purposes. This is by far safer than using a portable HDD Photo by Luis Quintero on Pexels.com
Different Types of HDDs
To make matter complicated, the manufacturers have designed HDDs to match particular needs of computer users in different segment.
General Use – Typically used on workstations in the office. Usually of cheaper quality.
Gamer – Designed with data speed in mind and longevity is placed second in priority.
CCTV – these HDDs are used for Closed Circuit TV systems as the system keeps updating the data as the security cameras are kept on to record. The focus is on reliability.
NAS or Network Attached Storage – these HDDs are meant for NAS systems such as the Synology DS220+. The focus is on read speed.
Data Center HDD – Used in data center and rated to have best reliability. Too expensive for consumers.
Tip 2: Even if you are using the HDD in an enclosure or a secondary storage in your desktop, choose NAS HDDs that is in the sweet spot of reliability, cost effectiveness and read speed.
My personal favourite HDD is from Western Digital RED designed for NAS but I used it in my Desktop as well.
My choice of HDD is the NAS HDDs. While it is designed to be used for NAS servers within the home, I used it for my desktop storage as well. It is far more reliable than the normal desktop HDDs without the need to pay for faster speed (Gamer) or reliability (CCTV). Besides the cost of the HDD at the physical level, you can also choose the capacity of the HDD based on per Terabyte cost. For now the sweet spot is around the 6TB and any higher the cost per unit is quite high. In absolute cost, the 4TB are the cheapest now.
For the more technically inclined on which brand of HDD to use, I rely on Backblaze blog to have a sense of which brand and model to look out for. Backblaze is an online storage solution company and their report is based on real-life usage of the hard disks. I will usually look out for 2 statistics, Mean Time Between Failures (MTBF) and Annualized Failure Rate (AFR). The longer the MTBF or smaller AFR percentage means the drive is more reliable in the long run. I use Western Digital’s RED and Toshiba NAS HDD for my storage needs. While the the Backblaze company may have different model and batch as what you may want to buy (it is after all historical data), the manufacturing process of their HDD should not differ that much.
This is a 3.5-inch HDD enclosure. From Orico
Storage Needs for Laptops If you are using a laptop, it is best to get a 3.5-inch Desktop HDD with an external enclosure and NOT the smaller 2.5-inch off the shelf HDDs. The main reason is that I get to choose what HDD goes into the enclosure rather than leaving it up to the manufacturer. The smaller 2.5-inch HDDs are more susceptible to disk failure as users put them in the bag where it gets knocked around or worse, dropped on the pavement. If you need mobile storage, you will do better with a NVMe memory stick with an enclosure or buy a larger capacity thumb drive.
HDDs are not meant to be brought around and should stay with your desk. You can just plug them in whenever you want it or you can just install a NAS in your home where I will talk about it in part 3 of this series. The downside is that you need to plug it in every time when you need to access the data and having a NAS may be a better option for ease of use and access.
Tip 3: Choose a 3.5-inch HDD over the 2.5-inch HDD as it is more robust. Besides, the external HDD should be on the desk rather being used as a mobile storage solution. For that, look at mobile SSD dongles instead.
Second Line of Defence – Uploading the Data up to Online Data Storage Services
Having a secondary HDD in the desktop or external drive is a good thing but you will only have 1 copy of the image data in your possession (technically second one if you did not delete the data from the memory cards but we know that will be formatted eventually).
To ensure there are at least 2 copies, at least another set should be uploaded to an online data storage services such as Google Drive, Dropbox, Send Anywhere etc. For photographers 500px, Google Photos and Adobe Portfolio are good choices too but I still went back to the grand daddy of online photo sharing site, Flickr, that has unlimited storage space and easy to organise to see your images as a portfolio.
The downside is always about subscription costs and these solutions means you have to pay more for more storage space and need to have a speedy Internet connection to upload huge files. This is the reason why I would only upload my best image files to save the time of uploading and keep most of my older image files in the NAS.
Qnap TS-251D Network Attached Storage
Third Line of Defence – Network Attached Storage (NAS)
There will be times when saving sensitive files such as private photos online or digital documents is not wise and yet a backup copy of those files are to be kept separately from your computers and mobile devices. You can use an external enclosure just like the solution for laptops but can be a hassle for some if you need to access the data while working around the house.
The most obvious choice is to have another computer in the home to hold the backup files in and share the data through not only your home network but can be accessed via the Internet as well. This is where the NAS comes in.
Just think of the NAS being your main data storage server. The Operating System is designed around storage and user access. Some models may even have 4 bay to accommodate 4 HDDs so that a RAID (Redundant Array of Independent Discs) storage system can be installed but can be an overkill. The best solution (compromise?) is to have a 2 bay NAS such as the Qnap TS-251D so that whatever data you have saved is automatically mirrored or copied to the 2 HDDs residing within the NAS. That would mean you need to buy a pair of HDD when you first started so can be a hefty expenditure.
A Network Attached Storage allows the backing up and file access without the need to fire up your computer. Mobile devices such as smartphones, laptops, tablets and even TVs can be linked via the home network to expand its use for easy access to files.
The articles in this series:
Data Storage Strategy https://wilzworkz.wordpress.com/2021/07/18/data-storage-strategy-part-1/
Naming files for faster search https://wilzworkz.wordpress.com/2021/07/19/organise-and-quickly-find-your-files/
Storage Strategy for Content Creators https://wilzworkz.wordpress.com/2021/07/21/storage-strategy-for-content-creators-part-3-network-attached-storage/