Your First A4 Print is On Us. Learn More

Please make an appointment before visiting the studio - other than your order collections.     PH: 03 9329 4522

Welcome to the new Image Science website!

Backup and Storage Overview

Multi-bay Synology NAS unit representing a RAID-based local storage option covered in this backup strategy overview

Introduction

In this article I'm going to give you an overview of backup strategies and storage solutions, including physical technologies. While it's a very factually grounded topic, the precise way you implement it in your workflow is idiosyncratic, so it's vital that you develop a system that makes sense to you and that you can maintain over time, using the principles and best practices discussed here.

For a more in-depth dive into the specifics covered in this article, see Jeremy's article Setting Up Effective Backup Systems for Digital Images, where he talks through his own workflow and the specifics of different solutions.

Don't assume it won't happen to you. Consider Olegas Truchanas, one of Australia's most famous 20th-century photographers, who lost the bulk of his images to bushfire in 1967 — a lifetime's work gone in minutes.

With any luck, bushfire won't be an issue in your life, but almost everyone is affected by media failure at some point — whether it's the all-too-common dead hard drive, a virus infection, or cheap burned CDs that start to rot away after just a few years. It will almost certainly happen to you eventually.

A good backup system has four features:

  1. It must be safe — that is, truly redundant.
  2. It must be easy to use, or you won't use it.
  3. It must offer fast access to your files while you're working on them, and if you need them again years down the track.
  4. It must be affordable.

Backup Strategy: 3-2-1

Redundancy is the name of the game.

You need three separate copies of your data: two different drives or media locally, and one in a geographically separate location. At a minimum. If you don't have at least this, you don't have a backup.

Two local copies of your data guard against hardware failure (drive failure, controller malfunction, some firmware bugs), corruption (data rot, software bugs), and some human errors (deleting or overwriting by mistake — a journalled file system helps here). But you can have a dozen local copies and still not have a truly redundant backup. A cataclysmic event like a bushfire or flood, or a burglar walking off with your drives, can render multiple local copies useless at once. That's why you need a geographically separate copy of your data somewhere else — to guard against something happening to your local copies.

For photographers, you should back up your images immediately when you get home from a job, but your backup strategy should really extend back as far as capture. Shoot and write to dual cards for an in-camera backup, and keep one card in your pocket on the way home from a job — just in case your gear gets stolen or you're in a car accident, you've still got a copy on your person if the worst happens.

RAID

Multi-bay Synology NAS unit, an example of a RAID-based network attached storage device

RAID is not a form of backup. RAID (redundant array of independent disks) guards against one type of hardware failure. There are plenty of failure modes it doesn't prevent, including file corruption, human error (deleting or overwriting by mistake), catastrophic damage (fire, flood, theft), viruses and malware, software bugs, other hardware issues (controller malfunction, voltage fluctuation, firmware bugs) — and drives can fail during a rebuild, too.

The Cloud

Illustration of cloud storage, representing an offsite backup destination
Image: Tumisu / CC BY

The cloud, in its many forms, is going to be the most viable option for most people's geographically isolated backup copy. It's an excellent, mostly safe, reliable, cost-effective option, and while I use it myself, I personally don't consider it robust enough on its own to be my only "part 3" geographically separate backup — I keep other cold, offsite backups too, for redundancy and protection against malware and viruses.

As with anything, it's worth checking it's genuinely the best option for your use case — particularly if you're a working professional with large volumes of files, or you work in video, where bandwidth, internet speed and restoration time (and the resulting business downtime) matter a lot. If you're a professional and a lengthy restore time because the cloud is your only offsite backup would mean unacceptable downtime, that's worth factoring in.

For most people, though, the cloud is a wonderful, cost-effective option for offsite backup, and we wholeheartedly recommend it as the single most important first step toward a proper backup system.

The Problem With Synchronisation

Two folder icons connected by a sync arrow, illustrating file synchronisation

Synchronisation is not the same as backup. The problem with syncing data from one place to another is the risk of propagating erroneous changes: if you're syncing drive A to drive B and you accidentally delete a file off drive A, the version on drive B gets deleted too. It's far too easy to accidentally spread corruption or a human error this way. Avoid relying on plain sync — at the very least, use a journalled file system so you have a record of changes and can roll back the data structure if you need to.

Data Rot

Close-up of disc rot damage on an optical disc surface
Image: at / CC BY-SA

That leads nicely into data rot. "Data rot", or data degradation, is the gradual corruption of data on a storage device from an accumulation of non-critical failures. To guard against it, periodically go through your stored data and compare it to a known-good copy (comparing an initial checksum hash of the original against the current copy) — there are programs designed to detect and guard against various forms of data rot that can help automate this. ZFS and Btrfs are file systems specifically designed to guard against various types of data corruption, popularly used in NAS units like Synology (Btrfs) and QNAP (ZFS).

Physical Technologies

Let's talk about the actual physical drives and solutions, and how to set it all up. This is all scalable to meet your needs — keep your 3-2-1 backup strategy in mind: a minimum of three copies of your data, two locally on different drives (and preferably different media), and one geographically separate offsite.

There are five broad categories of media and technology in common use, each with its own pros and cons. Here's a quick summary of the major options.

Removable Hard Drives

Lineup of removable external hard drives used for local backup

These are fast, and can actually be a surprisingly cheap option in the long term. However, HDDs rely on moving parts, and the mean time between failure (MTBF) for a single hard drive is measured in years, not decades. Leaving hard drives unused for long periods also tends to affect their reliability, so keep that in mind. Good for a semi-regularly accessed local copy of your data, for example.

Online Storage

In many ways an ideal solution to backup for most people: keep a local copy or two, and upload another copy to the other side of the world for storage on a massively redundant disk farm. Recent developments in the Australian internet market have made this an attractive option, with cheap, unlimited plans readily available. A good choice for a reliable, geographically isolated offsite backup for most people.

Multiple Disk-Based Servers

QNAP network attached storage unit, an example of a multiple disk-based server

Often called RAID machines, or NAS ("network attached storage"). These can be effective, fast and relatively cheap, but also complex to set up. Modern NAS systems, though, can offer genuine ease of use alongside excellent data reliability, high storage capacity and fast access.

Burnable Media

Two types of writable DVD discs
Image: ocrho / CC BY-SA

Burnable media isn't recommended as a viable backup solution anymore — remember the point about a backup needing to be easy to use, or you won't use it. Burnable media used to be cheap, easy and common, but today you'd struggle to find a modern computer with a CD/DVD burner built in. Availability and cost work against it too. The bigger issue is media quality: cheap CD and DVD media often lasts only a year or two, while high-quality media can last upwards of 100 years. If you do use burnable media as part of your backup solution, choose a good brand of media and be disciplined with your process.

Tape Drives

The classic corporate approach to data backup is tape. Old, slow, and notoriously unreliable over time, it's generally a poor choice these days — it can take hours of tape spooling to retrieve a specific file you've lost.

My Approach, as a Photographer

I take this further than most, with more layers of redundancy — call it pedantic or paranoid, but before I got serious about backup strategy, I lost photos that couldn't be replaced, to hardware failure. I go a bit over the top now so I never have to feel that again. Having slower spinning HDDs as deep storage is a lot less of an issue when you have a fast SSD as your primary editing drive — you don't need a separate drive for that; your computer's internal SSD works fine. All of this scales up or down depending on how much data you're working with. My own setup is an internal computer SSD, an external four-bay drive enclosure (not RAID), a cloud drive, and two separate offsite backups. The external enclosure holds a live file SSD, an HDD Time Machine backup of my computer, and two large identical HDDs that form my archive and archive backup (deep storage).

Internal computer SSD: holds a master folder (I call it "FINAL MASTER") of all my final edited, exported JPEGs from both the personal and business sides of my workflow, sorted by project folder.

Cloud drive: I use OneDrive, set up to periodically sync my internal SSD's FINAL MASTER folder to it.

External Time Machine HDD: a dedicated drive in my external enclosure running Apple's Time Machine, giving me both incremental, versioned backups and individual file version history to guard against human error. It also means I have a copy of FINAL MASTER on another drive.

External "live file" SSD: a dedicated SSD in the enclosure for fast access while editing RAWs and exporting finals. It's optimised purely for speed and for what I'm currently working on. When I ingest camera cards, all my RAWs get copied here first, and when I'm done editing, all my JPEGs export back here first — everything organised using the project folder naming and structure covered elsewhere. This drive also carries a copy of FINAL MASTER for redundancy.

External "archive" HDD: a large-capacity drive that's my primary store of all captured RAW files, with Personal and Business folders in the root. When I ingest a camera's card, all the RAW files go into a RAW folder within the relevant project folder, at the same time my editing copy goes to the live file SSD.

External "archive backup" HDD: a manually managed backup of the archive drive above (no RAID mirroring here), giving redundancy against drive failure for my RAW files. It's identical to the main archive drive but kept one step behind in my workflow, until the previous step is verified okay.

Offsite cold finals HDDs: two drives stored in two different geographically separate locations from my local storage, giving me two layers of redundancy against catastrophic events. As mentioned above, I don't back up all my RAW files offsite — only a copy of FINAL MASTER goes to these two drives. Being cold (offline) storage, they're also impervious to malware and viruses while sitting on the shelf, so if something of that nature ruined my local copies, I can restore from a clean source that couldn't have been infected.

Offsite RAW backup NAS: something I don't currently run, but am evaluating — essentially the setup this blog covers elsewhere as the recommended approach for offsite RAW backup. Despite the limitations of internet bandwidth and some exposure to viruses and human error, having a redundant, geographically isolated backup of my full RAW archive is appealing.

My Methodology

Technically, I keep six separate local copies plus three offsite copies of all my final edited JPEGs. I know that's overkill. I keep all my captured RAW files too, including rejects, but only two separate local copies. My reasoning: realistically I'll never need to look at 95% of them again, but as a just-in-case safety net — if there's ever a copyright question, or a client really needs an image for a specific purpose (a funeral, in one case that actually happened to me) — it's there. I keep them local only, still with two redundant copies, mainly because of the cost and, more importantly, the upload time involved in moving terabytes of RAW data offsite; for my use case, a remote offsite backup of that volume isn't currently justifiable. For RAW files, I'm guarding against drive failure and similar, but not a bushfire wiping out everything locally at once. What I'm mainly protecting, thoroughly, is my finished, exported JPEGs — against drive failure locally with multiple separate copies, against virus or ransomware with cold, air-gapped drives, and against a major event like a bushfire or flood with geographically separated drives.

Summary

Redundancy and consistency are the name of the game.

You need three separate copies of your data: two different drives or media locally, and one in a geographically separate location. At a minimum. If you don't have at least this, you don't have a backup.

At a basic level, for most people this looks like two separate hard drives locally and a cloud drive for the offsite copy. It doesn't have to be as elaborate as everything covered here — if you've currently got no backup at all, something, anything, is better than nothing. Start small and build up over time.

Next Steps

For the specifics of setting one of these systems up, see Jeremy's companion article, Setting Up Effective Backup Systems for Digital Images, and our guide to choosing the right disk format for any external drives you use as part of your setup.

If you'd rather not manage an offsite copy yourself, Image Science's Client Libraries service can hold a backup copy of your files for you as part of your print workflow. Get in touch if you'd like to talk through what would suit your volume of data.