youtubedown

If you use youtubedown, please grab the new version.

I think I've gotten a slightly better understanding of what Youtube is up to with this enciphered signature nonsense, and I'm trying a new method of dealing with it.

If you send me the errors printed for any videos that it can't download, that will be very helpful.


I think that what's going on is not that the ciphers are keyed off of the length of the signature, but rather, than they are just periodically changing the cipher algorithm, so the only way to know what algorithm to use is to have hardcoded knowledge of what is implemented in whatever version of "html5player.js" is getting loaded today (currently "html5player-vfl_ymO4Z.js".)

This means that every time they change the algorithm, I'll have to update the code in youtubedown. I don't know how frequently they're doing that, but that's some bullshit.

Maybe there's a way to parse this out from the Javascript, but since they've obfuscated and minimized it, the name of the decipherment routine changes.

I still have no idea how the signatures in get_video_info are to be deciphered. If there's a clue in there as to what algorithm is in use, I haven't spotted it.

Tags: , , , , , ,

24 Responses:

  1. tkil says:

    Thanks for the update -- it's working great here. (Sorry if I sounded ungrateful in the previous posts; I was just trying to be terse and helpful in a "report the bug" kind of way.)

    Your efforts are definitely appreciated. :)

    Thanks again!

  2. I reported a failure a couple of days ago, and it's now working. Thanks!

  3. Sylvain says:

    Now works with http://www.youtube.com/watch?v=sXQVicNodMw

    Great work, thanks a lot !

  4. andy says:

    Are you still using get_video_info to get download links? Could you please explain more

    • jwz says:

      Only some videos use ciphers. First I try get_video_info, but if the signature is enciphered, I scrape the HTML instead. This video has no cipher, so get_video_info works:

      % youtubedown --size -v 'http://www.youtube.com/watch?v=711bZ_pLusQ'
      youtubedown: 711bZ_pLusQ: found url_encoded_fmt_stream_map in JSON
      youtubedown: 711bZ_pLusQ: available formats: 45, 22, 44, 35, 43, 34, 18, 5, 36, 17; picked 22.

      711bZ_pLusQ 1280 x 720 62M Epic - Slinky on a Treadmill (2012)

      This video is enciphered, so I have to scrape the HTML:

      % youtubedown --size -v 'http://www.youtube.com/watch?v=ktoaj1IpTbw'
      youtubedown: ktoaj1IpTbw: found url_encoded_fmt_stream_map in JSON
      youtubedown: ktoaj1IpTbw: signature is enciphered.  Scraping HTML...
      youtubedown: ktoaj1IpTbw: found url_encoded_fmt_stream_map in HTML
      youtubedown: ktoaj1IpTbw: available formats: 46, 37, 45, 22, 44, 35, 43, 34, 18, 5, 36, 17; picked 37.
      youtubedown: ktoaj1IpTbw: deciphered and replaced signature
      youtubedown:      cipher: vfl_ymO4Z
      youtubedown:    86 42.43: 2424D9E05D9145E8203BB9D9D880DDAF991FDD7E8F.
      youtubedown:              42F10B14517CE39CA1ABEB7493EC470CB514A337337
      youtubedown:    81 40.40: 24D9E05D9145E8203BB9D9D880DDAF991FDD7E8F.
      youtubedown:              42F10B14517CE39CA1AB7B7493EC470CB514A33E

      ktoaj1IpTbw 1920 x 1080 154M CHVRCHES - Gun

      And this one is a real problem, because it is both enciphered and marked as age-restricted, so currently my script can't download it at all. In that case, the download URLs do not exist in the HTML but only in get_video_info -- but the ones in get_video_info don't work because of the encipherment:

      % youtubedown --size -v 'http://www.youtube.com/watch?v=7wL9NUZRZ4I'
      youtubedown: 7wL9NUZRZ4I: found url_encoded_fmt_stream_map in JSON
      youtubedown: 7wL9NUZRZ4I: signature is enciphered. Scraping HTML...
      youtubedown: 7wL9NUZRZ4I: Content Warning: This video may be inappropriate for some users
      Exit 1

      • Tim says:

        Have you looked at ClickToPlugin's youtube killer before? It's a Safari extension which does the a lot of the same things as youtubedown, except (a) it's multi-site and (b) it's centered around presenting videos in-browser in a HTML5 player rather than the site's own Flash based player. Here's the Youtube-specific part of the source:

        https://github.com/hoyois/clicktoplugin/blob/master/ClickToPlugin.safariextension/killers/YouTube.js

        I haven't taken the time to understand his implementation of ciphered signatures, but at first glance it appears to be algorithmic rather than a table as in youtubedown, and it works on every enciphered video URL from the youtubedown source that I've tried. Might be worth checking out.

        (Sadly, he doesn't appear to believe in comments, whether in code or revision control logs. At least the code looks fairly clean.)

        • jwz says:

          Wow, he is actually parsing out the algorithm by doing a regexp match on the Javascript source. That's some mad science.

      • David says:

        This specific video downloads just fine with the latest youtube-dl, in case that's of any use.

        • jwz says:

          If you have a video that works with youtube-dl but not youtubedown, send me the link and error message.

          • David says:

            Well, the second video you mentioned (http://www.youtube.com/watch?v=7wL9NUZRZ4I), which you say youtubedown can't get, is successfully retrieved by youtube-dl:

            [youtube] Setting language
            [youtube] 7wL9NUZRZ4I: Downloading video webpage
            [youtube] 7wL9NUZRZ4I: Downloading video info webpage
            [youtube] 7wL9NUZRZ4I: Extracting video information
            [youtube] 7wL9NUZRZ4I: Encrypted signatures detected.

  5. Jeff Clough says:

    As has become my habit, I just want to say thanks for this. You do a lot of great, hair-pulling work to make this thing go, and it's very much appreciated.

  6. nooj says:

    I used your mixtapes for another party last weekend! Last time I just featured the audio. This time, I hooked up my biggest screen and aired the videos too! I went through fifteen or sixteen of the mixtapes and compiled my favorite five hours of video.

    People loved it! I got a lot of compliments about how cool the videos were, and several requests for my playlist.

    I also declared a few new favorite bands (M83, Niki & the Dove).

    Thanks!

  7. I eagerly await the version of your script that implements a JavaScript interpreter in Perl in order to evaluate Google's cipher.

  8. snoj says:

    I was intriged by the idea of finding the cipher so I hacked this together.

    It's not as fancy as the Safari plugin and requires mucking up the source, but it does find the cipher function by executing all functions found in html5player.js. I think as long as they use something like call or bind and don't wrap the result in some obscure way, it should be able to find the cipher routine no matter what the obfuscator does.

    Relevant javascript can be found at the bottom of the source for the so inclined.

  9. Sylvain says:

    Is the --title option working ? I'm too noob in perl to figure out what's wrong, but when I try this command line with latest version, it doesn't... which is bad as I don't like Korean characters :D
    It does get suffix though

    perl youtubedown.pl 'http://www.youtube.com/watch?v=RzSAO8op0ow' --title "55" --suffix
    youtubedown.pl: downloading "[MV] MYNAME(마이네임) _ Baby im sorry"

    youtubedown.pl: wrote "/temp/[MV] MYNAME(마이네임) _ Baby im sorry [RzSAO8op0ow].mp4", 326M, 1920 x 1080

    • jwz says:

      --title has to go before the URL, because otherwise things like --title T1 URL1 --title T2 URL2 don't make sense.

  10. gah says:

    > wget http://www.jwz.org/hacks/youtubedown
    HTTP request sent, awaiting response... 403 Forbidden

    Christ. The Internet is doomed when even jwz fucks up basic shit like this.