My Linux-based, Multi-platform Photography Workflow

Lightroom screenshot blended with photoOne of the chief struggles for an amateur photography enthusiast, apart from developing your skill as a photographer, is figuring out how best to manage your ever-growing collection1. Importing, naming, securing, processing, and filtering thousands of photos often requires sophisticated planning and a stable workflow.

Linux—solution, or problem?

For various reasons, I’ve settled on Linux as my platform of choice, which has its benefits and occasional deficits. As a continual tinkerer, I tend to run more up-to-date but less stable versions of software, which means the tools I use for certain tasks sometimes have to change while a project shakes the bugs out.

For example, I have gone through Picasa, F-Spot, DigiKam, and Shotwell, only to find that they all have bugs, problems, or fundamental design flaws that make them unacceptable to me.

In my opinion, F-Spot is the best photo manager on Linux, but the last few major versions have been worryingly unstable. Shotwell is good, but has also been somewhat unstable—though this may be attributable to the aggressive development cycle of earlier versions—and I have generally been opposed to their versioning model and the inability to save metadata directly to photo files2. DigiKam has a respectable feature-set for a photo organizer, but isn’t user-friendly. And while Picasa runs under Wine, Google is no longer producing or supporting a bundled Linux build.

Essentially, there is no native Linux photo management software on which I can rely.

A further complication—multi-user, multi-platform

My wife is also an amateur photographer, and since most of our pictures are of joint interest (family, children, vacations, etc.), we’d like to maintain a unified collection, rather than having multiple copies of the same photographs floating around on multiple machines in various states of processing. She’s also a die-hard Mac fan at this point, so even if I found a suitable tool for myself on Linux, it might be difficult to get that software to play nicely with what she’s running.

The third complication is that my wife is most decidedly not a computer geek, and she has a very demanding career, so any solution she adopts must be easy to understand and as transparent as possible. And should she ever have an issue, I would be coming to her rescue, so our solution for her must be something with which I can stay familiar.

The workable, if not ideal, solution

After months of research and consideration, it became clear that the easiest way to meet my major criteria—a unified collection, stable software, and easy enough for my wife to actually use—was for us both to use the same software. From here, it wasn’t hard to settle on Adobe Lightroom. I was already familiar with Lightroom after years of watching Photoshop User TV, and though it doesn’t run on Linux, it does run on both Mac and Windows. Since I generally buy computers with Windows pre-installed, it’s a pretty safe bet I’ll always have some way of running it.

Currently, I run Lightroom in a Windows virtual machine, or boot from my laptop’s original Windows partition3. It’s a bit of a cheat, but I decided I’d rather “cheat on” Linux with Lightroom, than spend hours a week fighting broken software. Also, it meant my wife had one less thing to complain about me wasting my time on.

So, how do I salvage my pride and keep this photo workflow Linux-based? I keep Linux in charge of what Linux is good at—single-purpose tools and server stuff.

The Software

Rapid Photo Downloader
[Linux] Handles file import.
Adobe DNG Converter
[Wine on Linux] Converts camera RAW files to Adobe’s DNG format.
Unison
[Linux] collection backup/synchronization
VirtualBox
[Linux] Hardware virtualization
Adobe Photoshop Lightroom
[Mac, Windows VM on Linux] The photography workflow workhorse

The Workflow

  1. Import:

    I use Rapid Photo Downloader to pull RAW and JPEG files from my camera’s cards. It is Linux only, and it is the single best app for the purpose that I’ve found, on any platform. It handles import from the card, metadata based file renaming and directory structure creation, and simultaneous backup of originals to a separate location. It is blazingly fast. The only issue I have with it is that it doesn’t automatically recognize my iPhone, nor will it read images directly from it, but this is more a problem of the underlying OS than of RPD itself.

    I’ve set RPD up to import photos to my collection in the following directory structure:
    <CollectionDir>/YYYY/YYYY-MM/YYYY-MM-DD/
    I didn’t use the third level directory (YYYY-MM-DD) originally, but Lightroom’s importer isn’t as configurable as RPD, so the above structure was the closest Lightroom preset to what I was already using.

    For file renaming, I use the exiflow file naming convention, which looks like this:
    YYYYMMDD_cXX65789_mr000.jpg
    I realized long ago that attempting to maintain a collection with unique filenames, or keywords in filenames was simply untenable, so I have committed to putting good metadata in filenames and to rely on IPTC/XMP keywords for filtering. The exiflow convention gives me the date, camera model, original image number, photographer’s initials, and a code to manage versioning. Another reason I picked up the exiflow convention is that, before RPD was created, I used exiflow’s tools to import and automatically rename photos, and an exiflow plugin to fix F-Spot’s versioning.

    My wife will use Lightroom to import photos on her machine. As I mentioned above, Lightroom’s importer is not as configurable, but it can at least approximate the file and directory naming conventions I use.

  2. Pre-processing:

    I convert all of our RAW photos to DNG because, at least theoretically, it’s a more open format than the various proprietary camera RAW formats. Unfortunately, Linux has poor DNG support4, so I have settled on running the official Adobe DNG Converter for Windows under Wine.

    After importing with RPD, I run the camera RAW files through DNG Converter, then delete the originals (since RPD has already backed them up).

    Lightroom automatically converts the camera RAW files to DNG during the import process, so that’s one less step my wife has to worry about.

  3. Backup:

    As I mentioned, Rapid Photo Downloader automatically copies the original images to a backup location—in my case, to a folder on my NAS box. Presently, I only keep one copy of the originals, while i have multiple backups of my main photo collection. Unfortunately, I only started keeping these originals last year when I shot my brother’s wedding, so there are several years worth of photos for which I no longer have the original-off-the-card files.

    To my knowledge, Lightroom doesn’t offer simultaneous backup of the original files, so it’s not perfect, and though I intend to do most of the importing myself, it means I need to find a solution for my wife to easily make such backups.

    To maintain backups of the active collection, I use Unison to sync between my machine and the local backup, and between the local and remote backups. Unison can be a little fiddly, and mistakes can be catastrophic, but it is the best multipoint synchronization option I’ve found (or was when I was actively researching it, anyway).

  4. Processing:

    There are a few options for RAW processing on Linux—UFRaw, RawTherapee, and Bibble to name a few—but none of them have the usability or versatility of Lightroom. It offers full library management, keywording, “smart collections”, and more editing, developing, and processing features than I can figure out how to use. And when I’m finished with processing, it has exporters & uploaders to move the final images into my various online photo services, and a print module to handle physical output as well. So, everything in this step is handled by Lightroom, and I don’t have to make any concessions or adjustments since we’re both using it.

  5. Collaboration:

    I keep the “master” collection on my laptop, which is where I import photos and do file/directory maintenance. My collection is synced to a local backup, which serves as the central repository, and changes that I make get moved to that repository every couple hours.

    Rather than maintain a separate collection on my wife’s laptop (which hasn’t the disk space for it), she accesses the repository directly, via a network shared folder. She imports to, reads from, and saves edits directly to this network share. When she launches Lightroom, she first has to synchronize the collection’s folders, which automatically imports any new images into her local library, and updates any existing images with edits I’ve made elsewhere.

    I have Lightroom setup identically on my work laptop (a Mac), so if for some reason I need to access our collection at work, I have to take the same synchronization steps. This also helps keep me familiar with the process, and aware of any issues my wife might run into on her own machine.

    There are a couple major fail points with this method of collaborating or sharing our collection: first, it requires manually triggered folder synchronization, and second, there is a possibility that we could both be working on the same files, introducing a sync conflict, and likely resulting in one of us losing our edits. The first issue requires us both to establish a consistent routine with the software, and the second requires us to communicate effectively if we’re both working on the same collection.

    Hopefully Adobe will introduce “shared library” support at some point in a future version, which might help alleviate some of the potential issues.

Some final thoughts

Even on a single platform, it isn’t easy to find good software that does everything you need, and to establish a photographic workflow that makes sense. Even the best end-to-end workflow application available doesn’t offer every feature I would like, but thankfully the workflows it supports can be supplemented by the judicious application of some narrow-focus tools. And, with only a few concessions in the realm of personal principles (such as never running Windows again), I can stay rooted on my platform of choice, while enjoying the benefits of better commercial software support. Perhaps one day Adobe will get off their lazy tailbones and make Photoshop and Lightroom the crown jewels of software development that they could be, by making them truly cross-platform.

Until then, or until F-spot or Shotwell become more stable and full-featured applications that better support a full end-to-end workflow, or until Bibble gets off their high-horse and supports DNG, I guess I’ll just have to make do.

If you’ve struggled with setting up a photographic workflow on multiple or just a single platform, and have any insights, I would love to hear them. Especially you Linux photographers (I know you’re out there). How do you manage your photos? How do you facilitate sharing your collection with others? What’s your backup strategy? Answers to any of these questions would be welcomed in the comments below.

Footnotes

1. For the purposes of this post, we’ll use the following terms:

collection
The “physical” files and directories which comprise your photo… collection.
library
The software database that contains references to, and meta & edit information on some or all of the photos in your collection.

2. Lately, however, I’ve been leaning toward using XMP sidecar files for metadata, as it would reduce bandwidth overhead for remote backups—using in-file metadata means having to backup the entire file each time you add/change a keyword or make an edit, while using sidecar files means only the updated sidecar file would have to be backed up, saving megabytes, possibly gigabytes of bandwidth.
3. I’ve found that, installed on 64-bit Windows 7 running natively, Lightroom is excruciatingly slow and completely unusable. Surprisingly, once the VM settings are tweaked appropriately, Lightroom running under 32-bit Windows XP in a virtual machine is perfectly fast and usable. (I do have a rather beefy machine at this point, which makes this easier.)
4. The only native DNG converter I’ve found—part of the KDE project’s kipi plugins—killed the in-camera white balance setting during conversion. There are other Raw processors for Linux, but I’m not aware of any others that have a batch conversion feature.

8 Responses

  1. Lisa says:

    Dude, your digital assets mgmt & photo workflow puts mine to shame! I applaud you. :) I’m dealing w/hundreds of images at my main job, where I manage a website that’s updated every weekday, hundreds of images for my other main client, my own images, plus all the other stray ones for smaller clients. It’s all I can do to make sure they’re backed up. Hee!

    • coffeemonk says:

      Well, since I know you're a Mac user, you should have it somewhat easier… Lightroom should be all you need! ;)

      Maybe add a TimeCapsule + TimeMachine (or other NAS device) for local backup, and something like JungleDisk for remote backup, and you're covered.

      Three major recommendations I have are…

      1) keep your buckets big: I put all the pictures Sara & I shoot in one top-level folder, and images I receive from other people (family, friends, etc) in another folder. Both of these have the sub-directory structure i outline above, however, you could get away with just a sub-folder per year. Ultimately, the directory structure is a personal choice, depending on how much fiddling you plan on doing directly with the filesystem.

      2) keyword, keyword, keyword: with big buckets, your primary source of organization will be your keywords. For you, i'd recommend some top-level stuff, like "personal," "freelance," "for hire"… whatever seems appropriate. You can have Lightroom add these on import (along with IPTC metadata as appropriate). My keywording goes so far as "portrait" or "landscape" and "indoors" or "outdoors" … something that gives me things I can discretely search on to get at just the right image.

      3) consistent & stable file naming incl. metadata: i love the exiflow convention. It's no replacement for proper keywording, but at least you can look at a filename and know exactly when, by whom, and with what it was taken… which is, at least, better than the off-the-camera names. In your case, you could extend that with an additional code to indicate "personal," "freelance," or "<employer id>".

      in any event, glad you enjoyed the post!

  2. Hendy says:

    Have to say, I was a bit excited… but then disappointed, only because after seeing such an awesome title, it amounted to using Linux to access functionality designed for other OSs. Not a big deal — those other OSs have that software developed for them for a reason (they're more popular than Linux!).

    Anyway, I'm interested in this and haven't worked out my own solution yet. I've thought about just using tags or metadata to categorize somehow, but I've worried about if I lose all the metadata/effort if I switch programs. I'm planning to look into f-spot/shotwell for management, though a while ago I looked through this POST on alternatives to ACDSee and kind of settled on gqview (now geeqie). Simple viewer, but allows for tagging/metadata stuff. I never followed through and tagged my stuff, though! I just remember seeing all the others and not liking something about them.

    I particularly don't really want my manager to be my editor. I just want it to do one thing well. For editing, I recently decided I'd stick to shooting RAW and ran across some plugs for Rawstudio, if you're interested. Though… I guess you're using LightRoom, so who cares what Linux has to offer in that department!

    For what it's worth, I'll share two links on Linux management/editing that I just found today from the photo.stackexchange site: Linux RAW processing and Linux photo management.

    • coffeemonk says:

      My experience with Linux photo management/workflow software has been ultimately disappointing, so I feel your pain. On the other hand, if you're NOT looking for an end-to-end workflow in a single application, and are committed to one of the RAW formats that is well supported, I have to say that both F-Spot and Shotwell are quite capable as photo managers.

      They both have (or will have) the capability to write keywords directly to photo files, or to external XMP "sidecar" files if desired, which should allow you to move between applications more easily, in theory. In practice, I've found no applications that will read-from and write-to ALL the different buckets (XMP, and IPTC) that are commonly used to store keyword data within the photo files, meaning switching between programs may cause your keywords to get out-of-sync.

      I actually cobbled together a script that synched keyword data between multiple buckets, but I haven't maintained it. I could possibly clean it up and post it if anyone is interested, with no guarantees of utility or safety.

      If DNG support on Linux gets better, I'd love to re-evaluate my options in the future, and establish a truly Linux-based workflow.

      • Hendy says:

        Thanks for the reply — I actually re-read your title and retract any inkling of being disappointed about it not being purely linux based… you said so in the title! I just had selective reading issues at that moment :)

        I'm all for a program to be just a photo manager, and another to be just a RAW editor, so I think that helps my options. I don't have to ask a lot from F-Spot or Shotwell; just to manage. I may revisit geeqie for that, though.

        I just installed rawtherapee, rawstudio, photivo, and darktable last night and played around with them for maybe 5-10min each. Photivo was very non-intuitive, but I've heard great things. Darktable looks like a direct attempt to be the Gimp equivalent to Lightroom. I've never used Lightroom, but the screencasts I've seen lead me to think that darktable looks quite similar. It even has the "snapshot" button in the upper left. They aim to be both a manager (light table) and developer arena (darkroom), which is unique.

        Seems to have rapid development, so that's a plus. Rawstudio was my next favorite, and seems to have just been resurrected after about a 2 year development stagnation.

        I contemplated using filenames-as-tags at one point and wrote a script to parse through all my files in a given directory and create a nested symlink structure from the tags. I'm no shell-fu expert; you can see my thoughts/request for help on StackOverflow. I never got it where I wanted it, but it did work at one point! The cool thing is that your files never have to move, so you just browse the symlink directory tree and rebuild it every so often as things change.

        Anyway, glad your blog is here. Neat stuff and it's great to find other users interested in these types of things.

        • coffeemonk says:

          I like geeqie as a photo viewer, but haven't been impressed with it as a manager. Perhaps I haven't dug far enough, but it seems that keywording is an incidental functionality, rather than a true organizational tool…

          I hadn't heard of Photivo before, but it looks promising… I'll have to check it out. Being cross-platform is great, and it seems that it *might* support DNG, though it's not directly stated in the feature-list.

          Darktable seems nice, and also shows promise, but in my experience, it's not yet full-featured or stable enough for day-to-day work.

          Your symlink script is very intriguing… i had a similar idea a while back, but never pursued it for various reasons (lack of shell-fu being one). I'll have to look back at your SO stuff, and I'd love to see your code if you're willing to share. I mainly wanted such a beast so that I could have something to point my DLNA server and PS3 at. The Year/Month/Day folder structure is fine for housing a collection, but lousy for finding photos from a keyword-blind interface.

          Thanks for the comments!

          • Hendy says:

            Yeah, I haven't actually used geeqie much, and you may well be right that it's not very good with keyword stuff.

            I kind of like the light table mode of darktable for organizing. I've always been hesitant about anything uses a database or stores metadata somewhere that I cannot be sure I'll get it back from. That's what I don't like about a lot of managers.

            The code at SO right now is about as far as I got. I was really lacking a "spontaneous nested loop" structure that could accommodate any number of tags per file. Perhaps it needs to do something like (complete mockup)

            - for $file, tags = string
            - for tags in string, mkdirs (this would be really tricky, as you need every permutation)
            - in each dir, symlink the file

            That's about it. I would have gone to a naming convention like file-name_[tag1-tag2-tag3].ext, so looking in between brackets would have been simple enough. I just don't know how long that would take to do. with a lot of files. SO suggestions didn't actually help all that much, either. There's some other things that do this already — you might look into them?

            - Oyepa
            - tagsistant

            Enjoy!

          • Hendy says:

            Found another one which looks like about exactly what I tried to do! tagxfs. Looks like it might be cumbersome to setup (not sure if one has to manually add tags to everything), but one could probably write a script to churn through files if you put tag names in the file names as well? In any case, it looks really neat. I might play around with it sometime in the next few weeks.