Your First A4 Print is On Us. Learn More

Please make an appointment before visiting the studio - other than your order collections.     PH: 03 9329 4522

Welcome to the new Image Science website!

Setting Up Effective Backup Systems for Digital Images

Wall of stacked external hard drives, illustrating the scale of hardware needed for a proper digital photo backup system

If your house burnt down right now, and assuming everyone in the house was safe, what would you try to retrieve before the walls came down? For most people the answer is your family's photographs. Everything else can be re-bought, but you can't get back the record of your life.

So let's look at how to make sure you never lose your irreplaceable history — more specifically, your digital history. Backing up digital data is much easier and less expensive than making physical copies of everything, provided the system you use is actually a backup and not something that only looks like one.

How Big Is the Problem?

Let's use some back-of-the-envelope mathematics. Say you're a hobby photographer — you might generate a few hundred photos a month: some family shots, some nice landscapes from a weekend away.

Multiply that out by a few years, say five years since you started shooting digital, with a typical modern camera of 16 to 20 megapixels, shooting RAW. That works out at roughly 150 GB of images. Not a lot by modern standards, but add a scanned archive of thousands of historical family photographs and the raw data can soon reach a full terabyte or more. My own family archive, more extensive than most because we run a scanning service, is about 1 TB at this point.

If you're a professional photographer, your volumes are higher — often quite a bit higher. You might generate several hundred images each week, and most working professional photographers have in practice around 10 TB of genuinely valuable RAW data.

So how do we cope with this sort of data? 150 GB isn't so hard, but several terabytes is a different problem.

What Is Backup, Really?

There is only one good definition of backup: multiple copies of your important data in significantly different physical locations, with regular retrieval checks.

No other system really qualifies. Things like RAID 5 are not backup on their own — more on this below. The only reliable path to data safety is making sure you have identical, checked and complete copies of your data in different locations.

These are the things most likely to cause you data grief:

  • Device failure (easily the most common issue)
  • Human error (accidental deletion or an editing mistake)
  • Theft
  • Fire

Backup prevents outright loss of your data under any of these circumstances. All backup systems work through redundancy — having multiple identical copies of your data. If any one copy is destroyed, another copy becomes your working copy, which should immediately be backed up again if you're down to just one copy at that point.

Having any data as just one copy puts you at significant risk. Hard drive failure is common and can be very sudden — from working fine to total, irretrievable failure in the blink of an eye, especially with SSDs.

Data in only one physical location puts you at risk too. Even multiple copies spread across your house carry real risk — someone could steal both your hard drives, or your house could burn to the ground, which does happen in Australia. Fire has destroyed the life's work of several photographers of world importance.

A Good Backup System

A good backup system:

  • Is as near to automatic as possible — the more work it involves, the less likely you are to do it.
  • Lets you get back to work within a few hours of a negative event, restoring everything or individual files easily.
  • Is inexpensive — the more it costs, the more reluctant you'll be to keep it running.
  • Prevents catastrophic, outright loss under all circumstances.
  • Ideally prevents any loss at any time.
  • Is checked regularly to make sure you really can restore from it.

That's the goal: a totally safe, inexpensive, easy-to-run system.

Traditional Approaches

The traditional approach is to use backup software to make backup sets. This is slow and laborious, and generally requires a single storage system equal in size to the data being stored. You can do incremental backups — after the initial long, slow process of backing everything up, it only backs up recent changes — but restoring a single accidentally deleted file is tedious, and this method typically requires the backup drive to be connected locally, so people often end up with their backup sitting right next to their computer. Which means someone can walk off with the computer and the backup in one go.

Another approach is file synchronisation: rather than creating one big backup set, you synchronise one library of files to another — say, your master photo library to an external hard drive. This is effective and easy, but it isn't really backup on its own. The problem is that synchronisation copies mistakes too. If you accidentally delete a folder in your master library and then synchronise, you've just deleted your backup as well. For this reason, synchronisation needs to be paired with version control and fail-safes:

  • Journalling — keeping older versions of files so you can roll back to a previous version if you realise you've accidentally damaged or deleted something.
  • Fail-safes — if more than a set percentage of your files would be changed in one sync, the process stops and warns you, so you can't accidentally wipe both ends at once.

Synchronisation with journalling and fail-safes is probably the most reliable method overall, because it's very easy to automate. The backup you actually make is always better than the backup you don't.

A Better Approach

It helps to split the problem into two parts:

  1. Preventing outright catastrophic loss.
  2. Preventing any loss at all.

The first step stops the truly terrible from ever happening: it makes it possible to safely store and retrieve every image you produce, simply and cheaply, by reducing the problem and thoroughly protecting only the data you genuinely can't live without. The second step is considerably harder and more expensive, and involves protecting everything.

Preventing Catastrophic Loss

Preventing catastrophic loss is the main goal. Losing one precious image is bad, but losing your life's work or your entire family history is genuinely awful, so the most important system to put in place is one that prevents outright loss of everything.

Two technologies make this achievable: one reduces the size of the problem, the other removes the need to worry about backup hardware and geography. They are compression and the cloud.

Compression

Compression of images is a long-established practice. Mathematical algorithms reduce the data required to store an image. The most basic method is run-length encoding — rather than store every pixel individually, groups of identical pixels are stored together (instead of fifteen black pixels in a row, you store "15 × black"). There are many variations on this theme, but all of them reduce data size significantly at the cost of a very slight loss in quality.

Even when you shoot RAW, the end product of your photographic and editing effort is ultimately a single rendered image, and images compress very well with little to no visible quality loss at conservative compression levels. A typical 120 MB edited 16-bit TIFF — about what you need for a 300 DPI A3 print — can be reduced to 4 or 5 MB as a high-quality JPEG. Store enough uncompressed TIFFs and you'll quickly have gigabytes, even terabytes, of data; compression brings that pool back down to something manageable.

Your first concern about compression will likely be quality loss, and yes, JPEG is a lossy technique. But much like a high-quality MP3 file, in most circumstances the loss is imperceptible — and in the context of a disaster, complete retrieval of even slightly compressed data is a fantastic result compared with losing everything.

As an exercise, try this: save one copy of an image as a TIFF and another as a high-quality JPEG. Open both in Photoshop, lay one as a layer over the other, and set the blending mode to Difference. You'll quickly see how small the change really is.

Here's an example with a complex, detailed image. The differences are shown in the second image — black means identical, other tones mean some change. The difference is negligible in practical terms, and certainly no real concern next to the alternative of losing everything.

Fine art print titled Acedia by Jeremy Geddes, used as a compression comparison example
Acedia by Jeremy Geddes
Difference-blend comparison showing negligible visible change between a TIFF and a high-quality JPEG of the same image
Blending mode difference example — mostly black, meaning almost no visible change between the TIFF and the compressed JPEG

So compression helps us reduce the size of the problem by a factor of ten to twenty times or more — and that reduction is what makes the second technology feasible.

The Cloud

The cloud is really just a fancy name for a large number of server computers distributed around the world in various data centres. The technical implementation is irrelevant to the end user; what matters is the result: a readily available, universally accessible pool of inexpensive, safe storage managed entirely by someone else.

Cloud storage is offered by major IT companies who know how to build massively redundant, geographically distributed storage systems, and you simply buy space on their systems. All you then need is a reasonable internet connection with decent upload speed. Compress your files, upload them to your cloud storage of choice, and let the connection do the work — even a modest home connection is workable if you leave the upload running for a few hours a week.

Some cloud services worth considering, depending on how you like to work:

  • Zenfolio — offers large-volume photo storage plans and doubles as a private or public gallery and print-sales system, with good mobile and tablet clients so your library travels with you.
  • Dropbox — general-purpose cloud storage. Pairing it with a journalling/version-history feature means an accidental delete or overwrite can be rolled back later, which is worth having for anything irreplaceable.
  • Amazon Glacier — the cheapest option for really large volumes of data, but retrieval takes hours rather than being instant. Fine for a deep, rarely-touched archive copy; not for anything you might need back in a hurry.

The cloud is essentially low-risk, though it isn't unheard of for a smaller cloud storage company to shut down. For this reason, stick with established, well-known names, and consider using two independent services rather than one — it's now affordable to do so, and it removes the risk of any single provider disappearing.

Preventing Loss of All Your Data

This is harder to achieve. Here the goal is to store everything — all the raw data in full, uncompressed form: your RAW files plus any resulting TIFFs, with all their layers, after editing. This can mean you need a genuinely large amount of storage space.

One effective technique is to run two machines — your own and one kept somewhere else, such as your studio or a colleague's premises — connected by a simple internet connection, synchronising to each other overnight. Each machine is essentially just a large box of disks. It takes some initial effort to set up, but isn't difficult to run day to day.

For most people, now that large hard drives are cheap and readily available, the easiest approach is to buy a couple of large single drives and set up a simple synchronisation backup with reminders: once a week or so, back up your recent changes and new files to one of these drives, then store it off-site — at work, or a friend's house. Once you have more data than fits on a single drive, a sensible division — family photos as one set, client work as another — keeps the process manageable.

The temptation, as your data grows, is to look at more expensive multi-disk storage systems that join several drives into one large pool. These have real advantages, but also real problems for backup purposes specifically:

  • They're a single device, so they do nothing on their own to protect you from fire or theft — you need two of whatever unit you buy, with one kept off-site.
  • They're expensive — not just the unit and drives, but the extra drives used for internal redundancy.
  • They're often slower than single-disk systems, particularly over a network.

Multi-drive systems typically offer some form of RAID (RAID 5 or similar), which claims to protect your data — if one drive fails, you swap in a new one and the system rebuilds itself. In practice, RAID has a well-earned reputation among experienced IT people as a source of exactly the kind of disaster it's meant to prevent. RAID arrays have notoriously high failure rates, because the constant write strategy stresses the disks far more than normal use. Rebuilds ("re-sync" operations) after a drive replacement are a particularly common failure point — another drive fails mid-rebuild, or an internal RAID issue occurs, and the whole array goes down irretrievably.

Multi-bay Synology network storage unit capable of holding tens of terabytes of data in a single enclosure
A unit like this will happily store tens of terabytes of data — in one easily-stolen box, which is exactly why it shouldn't be your only copy.

Multi-disk systems encourage poor "all your eggs in one volatile basket" backup strategies and should, where possible, be avoided for backup purposes specifically — though they're genuinely useful as a working drive, keeping all your active files accessible in a single pool of storage.

A better multi-disk approach is "RAID over any file system" (RAID-F), similar to traditional parity RAID techniques like RAID 3 and RAID 6. Here, your data is stored on individual drives as normal, with a separate drive holding parity information. This offers the same recovery benefit as traditional RAID — replace a failed drive and the system rebuilds — but if the rebuild itself fails for some reason, you only lose that one drive's data, because each drive in the array is still a normal, independent drive in its own right. The all-too-common catastrophic RAID failure simply doesn't happen with parity RAID in this form, and it also tends to be faster.

It's generally cheaper to build a parity RAID system yourself than to buy a convenient pre-packaged disk aggregation unit, though it's more work: a server-style case, a stack of hard drives, and a Linux parity RAID system on top. It sounds harder than it is in practice — these builds are very modular these days.

General Tips on Traditional Disk-Based Backup Systems

If you do choose a pre-packaged multi-drive path, look for a vendor with a strong track record for reliability and support.

Whether you go single- or multi-drive, it's good practice to vary the brands and batches of hard drives you buy. This helps avoid the common problem of batch failure — if you buy several drives from the same batch and put them in an array, they all accumulate wear at roughly the same rate, so when one fails, the others may well be right on the point of failure too. Different batches fail at different times.

Finally, remember that a consumer hard drive has a useful life of around four years of powered use. Hard drives don't last forever and need regular replacing. Check your backups, and replace your disks entirely every few years.

How do we put this into practice? It needs to be as automated as possible, or you're not likely to keep doing it. Here's one workable approach:

  1. Copy RAWs from the camera to a temporary storage drive on your PC as soon as possible after shooting. (Leave the images on the camera card while you do this — that's already a backup in itself.)
  2. Import the RAWs into your master library via copy, not move, so you now have two copies on your PC and still one on the card.
  3. Edit your images as normal — visual edits, tagging and so on.
  4. Catastrophic failure prevention: immediately trigger two automated exports — one set of high-quality JPGs into a local "fast album" folder that automatically syncs to cloud storage and other machines, and a second set of JPGs uploaded to an online gallery service, which doubles as another backup and a way to share or sell prints.
  5. Full data backup: overnight, your RAW files automatically synchronise to a second machine using a synchronisation tool with fail-safes built in — for example, "don't synchronise if more than a set percentage of files would change" — which prevents an accidental deletion or edit from propagating to both copies at once. If you don't have a second machine of your own, this is worth arranging with a friend or colleague: set it up so your files synchronise from your machine to theirs overnight. Alternatively, back up your RAWs to a portable hard drive or drive array and physically move it somewhere else, cycling between two units so one is always off-site.

The end result is a lot of copies: high-quality JPGs on more than one cloud service for emergencies, so there's no realistic way to lose everything even in a major disaster, plus two independent copies of all RAW files, processed files and your catalogue — effectively your own private mini-cloud. Once set up, the process is almost entirely automatic.

Each year this gets cheaper, faster and easier as connection speeds and storage costs improve, making it steadily more practical to extend the same automatic, robust approach to video as well — which is another order of magnitude again in data terms.

Next Steps

Setting this up yourself is entirely achievable with the approach above, but if you'd rather not manage the hardware and monitoring side at all, Image Science's Client Libraries service can hold a secure, professionally managed backup copy of your files for you.

For related reading, see our overview of Backup and Storage and our guide to choosing the right disk format for external storage devices. If you have questions about backing up your own library, get in touch — we're happy to talk through what will work for your volume of data.