I have worked around Youtube's latest obfuscation. You can again download all videos and playlists with my youtubedown script and bookmarklet.Let me know if you find any that still don't work.
Great googly-moogly, that was a pain in the ass!
There's a get_video_info that tells you the URLs for the various resolutions of video data of a Youtube clip, in JSON form. Back in 2012, they changed it so that you had to take the provided "sig=" parameter and append it to the video URL before it would work. This signature appears to be a hash of some of the parameters to the URL (similar to what OAuth does) and the video-data URLs give you a 403 without it.
Within the last few months, they've started rolling out a new glitch where sometimes the signature (which is now "s=" instead of "sig=") is tagged with "use_cipher_signature", and... it's enciphered. Just enciphered. No actual crypto: simply a character-position-swapping cipher! These enciphered sigs come in lengths from 82 to 88 bytes, and you have to re-order and drop characters in particular ways to get them down to the 81-byte original signature. Each length seems to have a different algorithm, unless there's an overarching pattern that I haven't spotted.
It's purely security-through-obscurity! What a dick move.
Anyway, there's another kink on top of this: the enciphered signatures in the JSON don't work. The info about URL sources exists in both the get_video_info JSON, and also inside the HTML page itself, embedded as JavaScript. It's the same information, but with slightly different URLs and corresponding signatures. Obviously parsing the JSON is easier and saner than scraping JavaScript out of an HTML page, but... the URL/signature pairs in the JSON don't work, while the URL/signature pairs in the JavaScript do. So when you load the JSON, and find that the signature is enciphered, you have to punt on that data you already have and fall back on re-parsing the HTML instead.
What is this I don't even.
For laughs, check out the decipher_sig() function.
Now I have to wait for Miro to play catch-up and deploy a similar fix. I'm not holding my breath, because they still haven't released even a nightly build that includes the fix for downloading restricted Vimeo videos that I explained to them over a year ago. Sigh.
This sounds like the nonsense Y! did with their messenger log-in protocol to try to block third party clients from the service. (Because having ads shoved down your throat via the official client was so wonderful that obviously no one would have a problem with that.) The mechanism they came up with wasn't really secure, it was just ridiculously convoluted. They eventually gave up, but they didn't have the MPAA and RIAA breathing down their necks over it.
I wonder about the problem with the JSON signatures though. Is it a bug in the implementation and it happens not to matter to the official youtube player? Is a different decipher algorithm needed? Or is it an intentional attempt at obfuscation? I'm putting my money on it being a bug. Given how many layers of crap they've piled on, I'd be amazed if it worked right all the time.
I dont see the point to all this encrpytion why dont they just offer the god damn video as a simple link...they could even watermark it...Is it really so awful to be nice to the people that make your site even worth a damn?
I mean think about the people who dont always have internet.
there are some dumb fucking people working at this big tech compainies sometimes i have to sit back and just kinda let it all sink in that programmers arnt running the show...its the bigwigs up in corporate :(
Because then they can't track who watched it and when and for how long.
I don't think an internet company cares as much about people who don't always have the internet, much like Exxon isn't really concerned with people who only travel via bicycle.
I kinda disagree with a link they could tell not only who watched it but also who downloaded it so not only could they track views but also downloads
because if a user views a video theres no telling if they actually watched the whole thing or just viewed the comments but with a download there is some sort of guarentee that the user is somewhat interested in the video itself.
yea that does make sense...but its always that 0.01% that are more important then the 99.1% of the normal users
How do you embed easily-dismissible "targeted" ads (or display "targeted" pre- or post-roll ads) in an MP4 that someone downloaded from a link? Moreover, how do you determine that they later only watched the middle 10% of that MP4, or skipped around between points X, Y, and Z W times?
Wrap that entire comment in devil's_advocate XML tags for me, would you? :/
They can only tell that you downloaded it; they can't tell if you ever watch it afterwards, or who else watches it, either.
Again - there's no guarantee that I watched the whole thing after a download, either - with an embedded version, they can at least see whether I skipped around, etc.
Plus, one download does not equal one view - maybe I'll toss it up as a torrent or something.
Your math doesn't even add up. And YouTube - which is basically an advertising factory wrapped around a creamy video center - does not care about the 0.01% of consumers.
Plus, whatever Anonymoose said about serving targeted ads. YouTube can't easily serve ads to someone watching an mp4 file in VLC.
Nice program, thanks!
Just one minor thing, perl prints the warning:
otso:/home/ilya/flv > youtubedown.perl "https://www.youtube.com/watch?v=hZ9VHQizQok"
Wide character in print at /home/ilya/bin/youtubedown.perl line 1630 (#1)
(S utf8) Perl met a wide character (>255) when it wasn't expecting
one. This warning is by default on for I/O (like print). The easiest
way to quiet this warning is simply to add the :utf8 layer to the
output, e.g. binmode STDOUT, ':utf8'. Another way to turn off the
warning is to add no warnings 'utf8'; but that is often closer to
cheating. In general, you are supposed to explicitly mark the
filehandle with an encoding, see open and "binmode" in perlfunc.
youtubedown.perl: downloading "Ути едят куски булок"
Wide character in print at /home/ilya/bin/youtubedown.perl line 1653 (#1)
youtubedown.perl: wrote "Ути едят куски булок.mp4", 177M, 1920 x 1080
otso:/home/ilya/flv >
Hm. I'm still doing something wrong (or maybe I've got some CDN crap cached between me and YT from previous failures).
Trying to download just the Chvches video, and it 403'd. Downloading the whole mixtape again, we'll see if that works better... Nope, still errors on #4.
I'm using
youtubedownrev 1.159, downloaded just a few minutes ago. "Report this to jwz" data at:http://foiani.home.dyndns.org/~tony/jwz/youtubedown-errors-20130626.txt
If I were to generate an HTTP dump for this, what tool do you recommend? (I suppose I can shut down everything else and use
tcpdump, but if there's something more targeted, or a SOCKS4 / SOCKS5 option, that would be convenient...)Similar, while re-youtubedown-ing the 2012 best-of, every one that failed before v1.151 still fails v1.159:
I tried downloading the 2012 best-of for the first time this week.
God dammit.
I swear it was working earlier today.
What might be going on is that the algorithm is right for some of the cipher lengths but not others, and it's random what cipher length you get.
Fuck.
When people ask me, "Do you still hack?" and I answer, "Not really, I just do irritating bullshit in self-defense," this is exactly what I mean.
My friends get a similar response from me.
And lately, I've sort of come to hate that I know how to write code, because half the problems I "fix" are problems no non-hacker I know seems to have. When you don't know things could be better, you don't labor to fix them.
I confirm it was working yesterday.
Well my point was if the user did infact download the video then at some point you have to assume they were interested in the video.( and not just the comments, or ralted videos)
(and about your points on what parts of the video was watched...your assuming waay to much about the end-user, what if they dont have js enabled? what if they are using some pos software like ie5 or lower a simple link to a pure .mp4 or .flv is the most cross-platform solution that works for everyone what if they dont even have flash installed?)
also the watermark would be done as the video was uploaded server-side so that wouldnt take no more then a few minutes to setup
my math isnt supposed to add up,the whole point of my comments was basically to say this "the way they are doing it now is not only wrong its retarded and just points out that they dont give a shit about us"
however you want to reason it...adding a simple api to download a video would just make things just so much easier for us devs and those end-users because the way things are now it feels like we have to create these complex scripts and programs to circumvent a shitty system, thats whole point is to prevent users from viewing the videos on thier own time
I'm sure you're smarter than all of Google/YouTube's employees. There's no way they can have considerations you haven't thought of, or business/legal/technical barriers you're not already familiar with.
It's not about you, it's about control of the masses through FUD.
If Google regularly add little pointless roadbumps between the page URL and video URL, there will always be some point of unreliability with anyone-but-Google's tools. Because Google is deliberately fucking with them.
Thus, the typical person who wants an easy life and is scared by error messages will stick with going to youtube.com. The world is a shitty place full of people trying to make it as easy as possible for you to give them money, and as hard as possible for you to avoid giving them money.
Do you remember how Microsoft deliberately broke Windows if it detected it was running on DR-DOS instead of MS-DOS, but if you took out the few bytes of obfuscated DR-DOS detection code, Windows ran perfectly on DR-DOS? It's like that. Corporate bullshit to fuck with competitors, and you're in the crossfire.
Sadly, the only way to stop this is to fight back by being computer literate enough to get around the bullshit.
Speaking of annoying "security" tactics:
% wget http://www.jwz.org/hacks/youtubedown
--2013-06-27 09:51:48-- http://www.jwz.org/hacks/youtubedown
Resolving http://www.jwz.org (www.jwz.org)... 199.48.144.22
Connecting to http://www.jwz.org (www.jwz.org)|199.48.144.22|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2013-06-27 09:51:48 ERROR 403: Forbidden.
If you're the kind of hotshot who feels the need to use wget instead of Save Link As, and you don't immediately know what to do about this, then I guess it's too much 'puter for you.
I had to block bots in self-defense because of the periodic Linux fanboys and their broken scripts downloading my entire web site in a tight loop for days at a time.
You must be this tall to ride wget.
Sometimes a "roadblock" is really just an "entrance exam".
Oh, dear jwz, where are your "favorite comment" buttons, so I may espouse my amusement of your orneriness? Or are you not that social?
(But I will be in San Fran soon, and do plan on stopping at DNA Pizza. I'd even be willing to buy you a slice or whatever. And on a serious note, thanks for all that you've done and writing about it, Mr. Mozilla-the-name and Mr. Mozilla-the-organization.)
ShitTube