The Great Recodening

I just had to re-encode ~700 music videos that ITunes 11 has decided it won't play. That was about 14% of them, 20GB. So that was annoying.

I knew there were codec problems with some of my video collection, because a few years back iTunes 10 decided to stop playing them after some upgrade or other. There was a fix, though, which was simply to check the Finder box that caused iTunes to launch in 32-bit mode instead of 64. Apparently without that it wasn't able to load whatever part of Quicktime rendered these videos playable.

Well, iTunes 11 still has that problem, but now with the extra bonus feature that it is no longer launchable in 32-bit mode at all. Joy.

Everything is terrible. Here are the details, possibly more for my benefit than for yours.

The names of the recently-unplayable codecs, according to qt_info, are:

  • FLV1 -
  • H264 -
  • SVQ1 Sorenson Videoª Compressor
  • SVQ3 Sorenson Video 3 Compressor
  • VP6F -
  • XVID -
  • cvid Apple Cinepak
  • h263 H.263

These still work:

  • avc1 H.264
  • avc1 -
  • mp4v -
  • mp4v Apple MPEG4 Compressor

I don't particularly understand what any of those names mean.

I recoded the files using my script: --force --video --quality 20 --retry

which eventually runs something like:
HandBrakeCLI --encoder x264 --encopts cabac=0:ref=2:me=umh:bframes=0:weightp=0:8x8dct=0:trellis=0:subme=6 --aencoder faac --ab 160,160 --arate Auto,Auto --previews 30 --custom-anamorphic --modulus 16

The --retry option means "if the output file ends up being more than 30% larger than the input file, keep changing --quality and trying again until it is not." Possibly there's an easier way to accomplish this, but I'm not sure I'd trust it.

Then I re-imported them using Munge Videos.scpt, since iTunes loses its mind if you just change the file out from under it.

Perhaps if I build an airplane tail section of out palm fronds, the cargo will return.

Previously, previously, previously.
Tags: , , , , ,

14 Responses:

  1. John says:

    Just gotta say, dropping codec support and not supporting user-supplied codecs is a bit draconian. I do worry a bit about the direction OS X is going sometimes.

    • Tim says:

      Just gotta say, it seems like jwz's Apple related posts invariably attract facepalm-inducing herpderp comments.

      (hint: Apple has not in fact removed support for user-supplied codecs from OS X)

      • John says:

        I read your other post. I stand corrected. Apparently my cursory googling lead me to jump to conclusions.

  2. Dusk says:

    My crystal ball tells me that you have Perian installed, and you used to be relying on it to decode some of these formats. (One of those formats - Cinepak - used to be internal to QuickTime, but it's so incredibly ancient that I believe they've just dropped it entirely.)

    Perian is old and is now abandoned by its developers. A 64-bit version was never released. Unfortunately, I don't know as there's any real replacement.

    • jwz says:

      This all may be the case, but for QuickTime Player to be able to play videos that iTunes cannot can have no explanation that is undeserving of cockpunches.

      • Dusk says:

        Keep in mind there are two QuickTime Players - there's the new one (the one with the dark metal sphere-y logo), and you may also have a copy of the old QuickTime Player around (with the all-blue logo).

        The old QuickTime Player is a 32-bit application, and runs using an old version of the QT libraries. So it doesn't have any trouble playing old videos, but the approach it's using isn't forwards-compatible (it can never go 64-bit).

        The new QuickTime Player has to go through a conversion process (screenshot) to play some older videos. I suspect it's doing something crazy to make this happen, like launching a 32-bit subprocess that transcodes the video to H264 - it's not particularly fast, and it prompts me to resave the file before I close it. This certainly isn't something that it'd be reasonable for iTunes to do every time it opened an old video file.

        • jwz says:

          I could easily believe that part of the problem is "we couldn't be bothered to re-compile the old codecs in 64-bit mode". A very stupid part of a stupid problem.

          And... having a video player or music player that runs in 64 bit should be a goal for me because why?

          • Tim says:

            You know your central hypothesis that everything is terrible? Well...

            QuickTime was a disaster area in OS X for a long, long time. Apple originally did a somewhat straight port of the "classic" version, which was designed around circa-1991 classic Mac system internals (handle based memory allocation, single address space, cooperative multitasking, one CPU core, etc.). OS X apps often had to work around QT's limitations, such as not understanding the concept of preemptively scheduled threads.

            Later they introduced Cocoa wrappers, but it was still the same 1990s bullshit underneath. And they kept upgrading it instead of rewriting it. Remarkably they kept it polished enough so most users didn't notice, but it was a sore point for application developers for a long time. It also kept Apple and 3rd party devs from properly taking advantage of modern hardware in their pro apps (see: threading).

            In 2009 they finally shipped the ground up rewrite, QuickTime X. They didn't reimplement every feature (classic QT is a bloated kitchen-sink API, almost does email). Instead they pared it down to the essentials, committed to restoring other features which made sense, and kept shipping the old QuickTime 7 library for legacy compatibility.

            The 64-bit thing is also housecleaning. The 32-bit i386 ABI and Objective-C runtime were both products of 1990s NeXT design decisions. The problems weren't as insane as legacy QuickTime, but still the 64-bit versions are faster and clean up some cruft, and they've been making a lot of new stuff 64-bit only.

            What I've been working up to with all this: most likely, clicking the 32-bit checkbox in iTunes 10 was causing iTunes to load 32-bit QT 7 + Perian instead of 64-bit QT X. iTunes 11 is probably 64-bit only because they've decided to start taking advantage of newer framework features (not necessarily in QT).

            You may want to look at "man qtdefaults". Looks like you can turn on some legacy codecs even without the benefit of Perian. But if you've already got good recodenings of everything there's probably no point, since I'd bet they're going away completely sooner rather than later.

            • 32-bit QuickTime X has a wrapper which will invoke crappy old QuickTime 7 when QTX can't natively handle a format.

              They could have feasibly implemented this for 64-bit QTX (by spawning a "codec host" process), but for whatever reason didn't.

              FWIW SVQ1, SVQ3, cvid and h263 are the only codecs in that list that QT7 supported natively as I remember

          • phuzz says:

            While "we couldn't be bothered to re-compile the old codecs in 64-bit mode" is probably the real reason, it feels more like "we can't be bothered to support these codecs so fuck you".

  3. phuzz says:

    Still, it's nice occasionally to really put some pressure on a modern CPU and see how much grunt it's got.
    I remember trying to rip a DVD to AVI about 10-15 years ago and it took a good 4-5 hours to run through at about 2-3 fps (and then the inevitable fuck ups which mean you have to start again).
    Nowadays Handbrake would probably do the same job (in better quality with less fucking around) in less than 10% of the time, and you can still use the computer at the same time.

  4. Edouard says:

    All the names mean that OS X supports one codec: MPEG 4. The "H264" in unsupported list should, in theory, be supported, but MPEG 4 has 100 parameters, of which everyone supports a slightly different subset, and range of values. It's at roughly TIFF levels of interoperability these days.

    How's the quality afterwards? I turn the bitrate way up on reencodes, because I always worry I'm ruining the picture by letting the compressor try to recompress the first files block artifacts with new, different block artifacts, etc.

    • jwz says:

      Well, it turns out that most of these particular files were already pretty low resolution or otherwise shitty rips -- predictably so, given the age of the codecs -- so it's hard to tell whether the quality has changed much. E.g., most of them are ones that still only have a 320×240 version on the Youtubes, so even "redownload it!" doesn't materially improve matters.

  5. FLV1 - No idea what codec this is (probably some old Flash codec, from the name). That it worked at all is some kind of miracle (at a guess, Sorenson Spark)
    H264 - H.264/AVC. This is probably just mislabeled avc1. If you fancied getting involved, you could probably convince some media munging software to replace the "H264" tag with "avc1" and it would start playing.
    SVQ1 Sorenson Videoª Compressor - Really old QuickTime codec (Really fast decode, heinously slow encode, poor quality these days)
    SVQ3 Sorenson Video 3 Compressor - Old QuickTime codec. A better version of the above
    VP6F - ON2 VP6, the common YouTube format back in the Flash 7 days
    XVID - XVID. XVID is just MPEG-4 ASP, so relabeling this as "mp4v" would probably work.
    cvid Apple Cinepak - QuickTime ancient history codec (SVQ1 but crappier)
    h263 H.263 - H.264/AVC's ancestor. Probably has a resolution along the lines of 352 × 288 or half that. Really old; for a time was relatively common for Flash video; otherwise rarely seen outside of crappy video conferencing software.

  • Previously