WTF, Lightroom

So, I've been trying out Adobe Lightroom 1.4.1. It's an amazingly good piece of software, and I really want to use it all the time... but the god damned thing uses all available memory. I'll import a few hundred photos, start tagging them, and within minutes it's using 4GB of RAM, locks up on me with a hypnowheel, and I have to force quit. Restart it, and the same thing happens again ten minutes later. WTF, Lightroom? Is there any way to make it have less insane memory consumption?

(The Lightroom catalog contains all of my photos -- 75,000 of them -- but if the answer is, "you can't have that many photos in your catalog" then I don't understand why I would use Lightroom in the first place, because being able to search on photos by metadata seems like at least half of the reason this program exists. Without access to the whole catalog, Lightroom is just Bridge.)

But anyway, memory usage goes through the roof if I'm just tagging newly-added photos, without touching old files at all.

Any tips?

Tags: , , , ,

52 Responses:

  1. What makes Lightroom 'amazingly good'?

    • jwz says:

      Metadata searching and editing are great, rating workflow is easy, raw handling is great, and all of the image-manipulation tools that I normally use when editing photo galleries (white balance, levels, cropping) are built into Lightroom itself and are fast, so I almost never have to actually launch Photoshop at all.

      It's what iPhoto would be if iPhoto was also Bridge and, oh, if iPhoto wasn't a complete piece of shit.

      • interesting, thanks.

        (Right now I'm using stuff in linux (gphoto, ufraw, gimp) to handle all my photography, but I'm getting sick of crap like gphoto crashing on every other raw file. I'm considering other options, but they generally would require an investment in mac hardware. Meh.)

        • jkow says:

          On Linux I can really recommend F-spot. You can do tagging, rating and editing, and most of it very easy as different workflows are possible with many shortcuts, tap-completion on tags and stuff like that. And best is: It stores the meta data in the files and in an sqlite-database which you can query from the command line, php scripts, etc. and do all kind of stuff that the gui won't allow you to do. Also backing up your pictures including the databasefile is amazingly easy, you just copy pictures and database somewhere and can keep using them there as the database is freely accessible with free tools.... I could go on like this forever. :-D

          • bifrosty2k says:

            In case you forgot, LINUX IS NOT THE SOLUTION bro!
            GNOME is also 4tl.

          • hey neat, f-spot fails in an even more epic way than gphoto does! The cursor goes busy for a moment when I try to start it, and then... nothing else happens.

            And no, I'm not really interested in trying to troubleshoot this.

            • jkow says:

              Oh well.. nothing is perfect, it's version 0.4 after all. It's working great for me, that's what I was trying to tell.

              • gryazi says:

                Yeah, seconded. It has its quirks but the concept is right and recent versions work well enough.

                Because so much of it has been committed so recently, you want to make sure you have an actual recent version, at least one from the past 6 months, vs. whatever came packaged if your distro is not the newest.

      • With Lightroom 2.0 you also get the ability to paint exposure changes, also non-destructively of course.

        I think Adobe doesn't believe anyone would want that many photos in one catalog or something. I tried, and gave up - I've got about 20 different catalogs, organized by trip, or topic, etc...

        It's built on top of *sqlite*, for gods sakes. They probably have to load the entire database into memory all the time.

        • volkris says:

          Oh, sqlite... one of those software packages that makes me wonder why nobody else seems to notice how crappy it is.

          • laptop006 says:

            Because for all its crappiness it's *SO* *MUCH* *BETTER* then the crap that most programmers (and I include myself in that list) can write.

            That and most people hope that it gets swapped out for MySQL embedded or a real DB once the data file gets significant (which for sqlite is < 100MB).

            • volkris says:

              I've seen about a dozen applications move to sqlite, and never have I seen it enable a single new feature or capability, but I have seen it cause numerous problems even outside of the migration itself. For example, maybe it's been "fixed" by now, but at one point sqlite forced a disk flush every time data was modified, causing laptop harddrives to spin up annoyingly every five seconds. I put fixed in quotation marks because sqlite developers insisted that this was absolutely the correct behavior, and allowing buffering is evil.

              In my experience (and yes, that's a pretty big caveat), sqlite is a bullet that all too many developers list as a feature while deriving nothing but problems from its use.

              • emn13 says:

                Having recently used sqlite, I can tell you that it's very fast and uses little memory. It's almost certainly faster and smaller for simple cases than mysql or any such alternative. On a 10.7GB database file, it quite happily runs various joins and so on without using more than a little bit of memory (which, by default is something like 10MB, but I generally tweak to be much much higher to improve performance).

                I'd be willing to bet that sqlite is not the cause of the performance problems in lightroom. You can, incidentally, open lightroom's catalogs in sqlite admin programs without issue, though if you want high performance, you'll want to tweak the caching settings before doing so (10MB is just too little).

                Sqlite is a disk-backed, reliable data store, with all the drawbacks that entails. It's not tightly coupled to a particular language. But within those boundaries, it's excellent. You are aware, for instance, of the fact that firefox 3 uses it as a backend for storing history and profile data? And that this "enables" the rather handy title-bar search? And that this disables the "starting firefox takes forever with a large history" feature?

                • volkris says:

                  This springs to mind.

                  When databases are the answer, sqlite is a strong competitor, but all too often developers have been using it where it just doesn't belong.

  2. bifrosty2k says:

    My unfortunate experience is that most people who write imaging software don't really know what memory management is. Their fix is "oh, memory is cheap, buy more" which is BS. They should fix their sloppy ass shit.

    • miriku says:

      confirming that he knows most people who write imaging software personally. and knows their beliefs on memory management.

      • bifrosty2k says:

        Yes, I know all of them.

        Its actually, way too many developers don't know shit about memory management. I once found some code in a bit of server SW that never expired things out of a queue. So the process just grew and grew and grew and grew... I told the developer about it and they asked me what was wrong with that, wouldn't the operating system take care of that for them?

        Its one of the other reasons why I dislike a lot of RoR developers...

    • artkiver says:

      Well, memory is cheap right now. In fact, after lunching with my DRAM designing friend earlier this week, it's too cheap - his company couldn't pay staff for four months.

      Sadly, buying more memory is not the answer - shitty programs will just keep gobbling. Moreover, what if you've got a laptop - short of a tadpole, you won't be able to cram more than 4GB's into it still.

  3. vordark says:

    My photog friend reports to me that Lightroom (even the newest version which I guess is around 2.x) expects the phrase "large library" to be something less than 20,000 images. More than that and he says it just starts crashing and burning when trying to do seemingly trivial things.

    I asked how he solved this problem and he told me, and I hope this makes sense to you, "Divide your big library into multiple libraries, either by category or just arbitrarily, and do most of your work with that. Then, keep one big library with all of your images so you can do searches."

    Hope this helps you.

    • asw909 says:

      The thing is, I recently started using LR, added a folder of just 14 photos, and sure enough, that crashed on me too.

      It's a brilliant program, from what I can tell of using it so far, but it's memory "management" almost makes it utterly unusable!

  4. bitpuddle says:

    Is it still doing its import-and-process-metadata thing? That thread ran for days when I imported a good chunk of photos.

  5. jope says:

    A year old, but the following bits from this thread sound relevant:

    "LR does 'accumulated background work' when nothing else is going on"


    "During the conversion from a library file to catalog file, it used 750+MB of RAM and 1.3+GB of VM, but those dropped to reasonable limits after the startup after conversion. I just started LR and it began with 102MB and quickly dropped to 32MB."

    You're only tagging, yes, but the fact that you're doing so on recently added photos may be the key point. Have you watched it long enough to see whether memory usage goes down eventually?

    If that's the problem, I guess the manual should be amended to say 1) add photos, 2) go make coffee, 3) resume your LR experience.

  6. phreddiva says:

    I've asked, but I probably won't hear anything until Monday.
    Hope whatever I do hear, helps.

  7. jcurious says:

    Metadata searching and editing are great, rating workflow is easy, raw handling is great, and all of the image-manipulation tools that I normally use when editing photo galleries (white balance, levels, cropping) are built into Lightroom itself and are fast, so I almost never have to actually launch Photoshop at all.

    I belive Aperture does all this:

    • jwz says:

      Being Apple software, I believe Aperture does what iPhoto does: sucks all your photos into a database and doesn't let you decide what physical folders on disk each photo lives in. Once your data is in iPhoto (or, I believe, Aperture) it's rocket science to get it back out again. That's a deal-breaker.

      • jcurious says:

        I thought one of the features of aperture was the ability to "work" with images even if they are offline, ie. on removable volume. That concept would not work ifit kept all the photos in one location.

      • ckd says:

        While I haven't used Aperture, I've never had problems getting things out of iPhoto. Drag and drop (grab picture in iPhoto, drop picture in Finder) will DTRT.

        • discogravy says:

          It's hard to drag and drop via script, for example. Scripts (and CLI in general) tend to need to know where the file actually is. I haven't got any horses in this race, so I don't really care, but it's worth pointing out that drag and drop isn't valid all the time and jwz's deal-breaker is a valid one for a lot of people.

          • pfrank says:

            also, does your metadata follow the file when dragged and dropped, or does it get locked away in the iPhoto database as well? I assume you can't get that stuff out of it, I really need to fire iPhoto but I hav no idea what else to go with.

      • m4dh4tt3r says:

        That's been my experience with iPhoto and Aperture, which is why I ditched iPhoto and have shied away from Aperture. If LR fixes the memory problems, then it might be worth buying. Until then, I'll hobble along with Bridge.

      • danomite55 says:

        I've seen Aperture 2.0 perform very well on a MacBook Pro with a library of nearly 300,000 images. Images can live where ever you want, and can even be on external disks that aren't necessary plugged in. Aperture keeps a local preview so you can still see them.

  8. fieldsnyc says:

    Use Lightroom 2, not 1.4.1 - it's a significant improvement in every way. Among the huge number of things they changed, it's now crystal clear in the folder display which remote drive your pictures are on.

    I have about 30k pictures in my catalog (all on a linux fileserver), and while it took a while to import them all, it runs in around 800MB - 1.5GB of RAM under normal usage.

    • feren says:

      Crystal clear, yes.

      ... but (at least in the WinDoze world) I still haven't found a way to force LR2 to use the drive & folder where I want the folders with my DNGs to be on. So what ends up happening is I import all my RAW files off E:, it creates a new file with all my DNG files in the "My Pictures" folder on C: and then I have to manually drag and drop the DNG-containing folder back to D: where I want it with the rest of its brethren. Which then causes LR to be busy for like 5 minutes even though everything I have is SATA-300.

      That frustration being mentioned, I am still using LR because it sucks the least out of all the OSS/commercial options out there.

      • cetan says:

        So your workflow process is to create DNG files automagically during import?

        When I attempt that in Lightroom 2 I'm presented with a dialog box that includes a path and a "Choose" button allowing me to point LR to a folder where I want the DNG's created.

        I must be missing something with regards to your workflow because it seems to work fine for me.

  9. movingfinger says:

    Um, is it rendering previews as you go along? What size previews did you tell it to make? That may be what's choking it. On our 10.4 Powerbook G4, that is the most expensive usage. On our 10.5 iMac, processing can be a little slow as it renders, but there's more memory and processing power and the only problems have been related to using a database on an external USB disk. I don't think I've ever locked it up on the 10.5 iMac the way I can on the older, slower 10.4 Powerbook.

    I am told Lightroom 2 is much better, but after the 1.4 disaster will wait a couple more weeks for bug reports to come in, before upgrading. Apparently it doesn't still fix the screwing-up-dates bug, which angers me constantly.

  10. fhfdotorg says:

    Are your photos on internal hard drives or external?

    I tried storing my images on an external NAS, and the performance in Lightroom was pretty crappy. Once I added more internal hard drives and moved the photos there, Lightroom became usable. This is for about 50k images.

    Also, are you always "ejecting" your removable media? OSX has some issues with removed-but-unejected media that can lead to the spinny pizza of death.

  11. terryray says:

    So, someone wrote software that downloads all meta-data into memory. As if they don't believe in or like database software for such a thing, and they believe that nobody will have enough data that memory will be an issue.

    Why does this sound so familiar?

  12. rafasgj says:

    I used Aperture, it's not like iPhoto in the file management, it's much more "open" in that regard. Anyhow, it's slow as hell. I now use LR 1.4.1, and I'm "ugrading" for LR 2, mostly for its editing features.

    I've been a software developer, now I'm a social and sports photographer (mostly weddings and tennis), and each of this events easily goes for 2-3 thousand photos in a weekend.

    No photo software today live up to dozens of thousands of photos in a single library. What I do (and it works better for me anyway) is to have a catalog for each event. It's easier on my old PowerBook G4 machine (1,25 GB RAM), and it's easier on me for selecting photos, backing them up, etc.

    It wold be very nice to have a simple application that would just catalog hundreds of thousands of photos with tags, EXIF/IPTC and thumbnails, that could handle different projects, offline images, etc. And uses "only disk" as storage. Retrieval need not to be instantaneous. A few seconds are just fine, why keep the whole catalog in memory? I don't get it.

  13. wfaulk says:

    Just an idea, but what happens if you open a, run "ulimit -v 1048576" and then "open /Applications/"? I don't know that applications opened that way will inherit the ulimit environment from the shell, nor do I know that it will not do the same thing, only at 1GB instead of at 4GB, but I figure it's worth a shot.

    • gryazi says:

      Not experienced, but enforcing a ulimit might either help it pick saner defaults or just OOM-kill it faster.

      I assume it's not the issue, but the older Adobe apps were all born in the early Mac's culture of non-memory-management (remember how OS 9.x still required pseudo-manual allocations per process?). Thus, say, Photoshop still has its own cache and its own memory preferences. LR was probably developed by people who assume "OSes are real and RAM is cheap," but maybe there're still some tunables lurking.

    • kehoea says:

      If the ulimit doesn't work with open, it's normally possible to start the app directly, inheriting ulimit as normal; you just need to be in the appropriate directory in the bundle. So (I don't have Lightroom, of course):
      % cd /Applications/
      % ./Safari &

  14. violentbloom says:

    I've been using the Lightroom 2.0 beta (which ends on the 31st and the real one is on sale) and it seems ok. I had previously used LR when I only had 2gb and it was ok, but I had far fewer pictures. With the lightroom 2 beta I've got 4gb seems pretty zippy

    I have about 7,000 in my catalog that lightroom opens at once. I seem to be able to use it photoshop and also firefox with several windows and I'm not having issues. I leave it running all the time. (I have to restart the machine every couple weeks or things, firefox mostly, get screwy)

    maybe using multiple catalogs would speed things up for you? I probably should start doing that sometime soon.