A badge for the software industry's failures:

Let's say you want to automate a post to Twitter. This used to be an HTTPS POST with a user name and password, and essentially it still is, except that now it's a token with a very convoluted generation-and-signing process. Still, after that token has been generated once, a post comes down to nothing more than connect(), write(), read(). The whole process fits in like 6 packets.

So then you do "cpan Net::Twitter".

The dependency hell that follows results in 49 tar files downloaded, 91 packages installed, and 10,403 lines of compiler and installer output.

For 6 packets.

(Oh, and then it doesn't work anyway, because something called "Moose" fails to install with an "unknown error" -- which apparently really means "unsatisfied dependencies"! So it is a mystery how many packages it would have actually installed.)

Tags: , ,

61 Responses:

  1. chort says:

    The awesome part is that none of the Twitter libraries I've seen bother to validate the certificate, so it's all susceptible to MitM any way.

    Aren't you glad you did all that work for absolutely no gain in security?

  2. Winfield says:

    Is your beef with software or just the crufty state of Perl modules?

    If you're looking for concision or innovation, Perl is the wrong language and community.

    • oZ says:

      Except Net::Twitter and Moose aren't crufty. Sounds like the above Perl install is crufty, if this is the first sighting of Moose.

      • Paul Boldra says:

        Agreed. If your local perl doesn't already have Moose, it doesn't sound like you're using it very often.

        • Boxcat says:

          Try this simple test. Pick any vanilla Linux / Unix install you like, which to make life easy for you, already has the client libraries for your chosen database already present on the box.

          Now, using the CPAN shell, install the database drivers and any self-bootstrapping modules that an out-of-the-box Perl needs.

          Time how long this install takes. Check whether it actually leaves you with a usable install which is actually even fucking capable of connecting to a DB of the specified type at the end. (Free clue: If the DB you're trying to get at is Oracle, the answer is a resounding "No!"). For extra credit notice how much manual interaction is required to get the install to run, and the difficulty of scripting this install in anything like a reliable and portable way.

          This is utter, unmitigated fucking bullshit. This isn't some esoteric use case, or a corner of the language where there are competing standards. This is fucking bread-and-butter. This is the shit where Perl is supposed to win. Not this "I've fallen and I can't get up" bullshit.

          I'm not even going to get into how this compares to other languages. Fuck, I can probably figure out a language I don't know's packaging system and get it done in less fucking time than it takes CPAN.pm to do its thing.

          This was the final straw for me. I was a professional Perl programmer for >5 years, but this state of affairs was what finally did it for me. Perl is now a cross-platform sed/awk/glue replacement language and nothing more.

        • jwz says:

          I use Perl all the time, for all kinds of things, and until Net::Twitter I don't think any package I was using had pulled in Moose. But that probably just means that you would consider the packages that I find useful to be written in a tastelessly procedural style or something. Oh well. I assume the fact that I'd never even heard of Moose before I posted this will make you think further bad things about me. Oh well.

          This was, in fact, one of the first packages I had installed after a fresh OS install, but so what? That doesn't change the fact that there was a truly gargantuan list of dependencies, and the dependency-chasing was broken.

          I probably wouldn't have noticed or cared that the deps were long if the deps were not also broken, but the latter is a frequent and likely consequence of the former.

        • J Greely says:

          Twenty years of working with Perl every day, and the first I ever heard of Moose was when I (made a long lunch and) installed the MongoDB module, which uses another module that hides behind a wrapper and pretends to be Moose. Fortunately most of the modules I need are still written in Perl, instead.

          When I do reverse dependency lookups on Moose and Mouse, about the only thing I see that I might find a use for is Net::Amazon::Route53, and everyone recommends Boto anyway, so probably not.

          -j

  3. fantasygoat says:

    I feel your pain. I was recently asked to add a module to PHP which had dependencies on two separate, specific versions of libraries that are not available from any of the standard repositories, and when I did eventually hunt them down, I could not then get them to compile.

    After a few hours of trying to get it to work, I asked why they needed this module and it was used once, in one script, to do an international date conversion. That you could have done in like 3 lines of code.

    It just seems like LAZY THINKING to me.

  4. Maybe Net::Twitter::Lite is lighter?

    http://search.cpan.org/dist/Net-Twitter-Lite/lib/Net/Twitter/Lite.pm says

    This module provides a perl interface to the Twitter APIs. It uses the same API definitions as Net::Twitter, but without the extra bells and whistles and without the additional dependencies. Same great taste, less filling.

    This module is related to, but is not part of the Net::Twitter distribution. It's API methods and API method documentation are generated from Net::Twitter's internals. It exists for those who cannot, or prefer not to install Moose and its dependencies.

  5. Tony Finch says:

    I wrote a post about "simple shell scripting for Twitter" using Jef Poskanzer's little http and oauth tools. In it I said "I didn't fancy dealing with the dependency hell of mainstream scripting languages" so I hope it's up your street. See http://fanf.livejournal.com/108436.html

  6. The Perl community has finally turned its back on KISS. The prize they're after is being taken seriously. They're tired of getting beat up about their low tech (but insanely flexible) object system. They're trying to placate their detractors -- something that never works and often results in turning your back on your core constituency in the process.

  7. Mark Beeson says:

    Twitter needs to just hire about five guys to do nothing but write sample libraries in perl, Java (sorry, Scala), Javascript, and PHP (sorry, Ruby) and post the results on dev.twitter.com. Instead they've got this crappy half-wiki with nifty "consistent user experience" icons to reuse.

    • DFB says:

      Hear, hear! But do the unparenthesized parts first, and add Objective C and C for that matter. Why not Python too?

      The more they offer in such libraries, the fewer headaches they would get from third party clients. The idea that you can publish an API and expect people to do exactly what you want without intentional or accidental abuse is absurd. If you have in-house people doing libraries it will help you debug the API docs, too, which is not an insubstantial problem for the Twitosphere.

  8. Thomas Lord says:

    The entire web is made of such stuff. Don't act so surprised.

  9. dan ǝɹooɯ says:

    As someone else has pointed out, try Net::Twitter::Lite. There are good reasons for Net::Twitter being the way it is, and also a reason for there being Net::Twitter::Lite.

  10. This might be a good time to rethink your stance on the delicious, delicious sewer rat that is Python. 'easy_install python-twitter' generated 60 lines of output and pulled in three additional dependent libraries -- admittedly, only one of which offered serious added value beyond the standard library. And the whole whitespace thing actually does taste like pumpkin pie.

    Ruby would probably be a better fit for your Perl background, but I tried 'gem install twitter' and the only output I got from it after waiting two minutes was a 20 degree CPU temperature increase.

  11. I wrote and maintained Net::Twitter 1.x and 2.x. They were both traditional perl and did not use Moose. When I handed it off to Marc he switched to Moose where I had always refused. Though I wouldn't do it myself, I understand his reasoning.

    Disclaimer, I've had absolutely zero to do with Twitter for the two years since I handed off the module. I can tell you nothing about what the API team is like today, but I've kept vague contact with Marc and have heard stories, and I'm fairly certain it hasn't changed much since 2009.

    Let's lay out facts.

    http://deps.cpantesters.org/?module=Net%3A%3ATwitter&perl=any+version&os=any+OS

    Net::Twitter has 122 dependencies. 54 of those are DateTime. 33 of the DateTime dependencies are test modules. Don't agree with that, fine. But it's not Marc's nor my fault that 44% of the depenencies come from perl's date parsing swiss army knife. You want to parse dates, you use DateTime.

    12 of the deps, 10%, are OAuth. Blame twitter, they dictated it as the only way to authenticate.

    18 of the deps are Moose.

    Your argument that "this is easy, it's six packets" is standing on a very simple fallacy, that those six packets are a fixed target that makes sense, that they are designed by a team able to even grasp fundamentals of how such an API should be written.

    None of that is true. I handed off the module in 2009 because I had finally run out of "give a shit" after dealing with the Twitter API team since it's inception. The API endpoints were a swarm of moving targets, not just with definitions and arguments, but with seemingly random decisions about HTTP return codes, specifically where a method in one part of the API would behave an entirely different way than a method in another.

    Three separate times I had the API team claim my code was broken because I was following the HTTP Authentication RFC and they had invented something from whole cloth that had no functional relationship to "This is how HTTP works". And these auth changes weren't anything planned or announced, someone simply came back from lunch and threw some new code up.

    The way certain API methods would react changed seemingly every few days. I'd usually find out when I'd start getting tickets that "your module is broken" and I'd have to spend a few hours going through dumps to figure "Oh, now if you aren't logged in they'll return a 404 on this method, where it's entirely different everywhere else."

    It ended up being that it required excruciating amounts of code to dance around the inconsistencies and I finally gave up. Marc used Moose because Moose makes writing exactly this sort of quick changing code far more manageable.

    Don't like it? That's fine, don't use it. But spare us your snark that it shouldn't take that much code to make "just six packets".

    I speak for myself alone, but I know Marc well enough to vouch for him. Not a god damned byte worth of code has ever been in Net::Twitter because it was cool or trendy, every last line is there to solve a problem with pharmaceutical grade fail whale.

    • Ian says:

      This sounds right and it reminds me of a developer of a web browser who, in response to people trying to rewrite it from scratch, said something like, yes, the ftp code looked messy but that's because it had to be to work - what he had sweated blood over was dealing with an inconsistent mess of servers disagreeing with each other as to what the standard way of doing it was.

      What was his name...?

      Thanks for putting up with Twtter API hell for as long as you did. I want to send messages from a micro-controller and it should be much, much easier and simpler than it is.

      • jwz says:

        Har har, but there's a big difference between trying to support a legacy protocol that has had literally hundreds of implementations over the preceding two decades; and supporting a protocol that was invented basically last week and has exactly one server-side implementation.

        The difference being: for the former to be a big hairy mess is an understandable drag. For the latter to already be that much of a mess is absurd.

        • pavel_lishin says:

          > invented basically last week
          > has exactly one server-side implementation.

          And going by Chris's post, re-invented again and again every time someone came back from a trip to BurgerLord.

          So, yes, absurd, but Twitter's fault, not Chris's or Perl's or whoever's.

          • pavel_lishin says:

            wish I could edit these; just now saw the comment right below this one where you explicitly unlift the blame from Chris's shoulders.

        • Ian says:

          Almost everything about Twitter is absurd. Clearly someone thinks that should include the API.

          Or more probably, no-one ever did think about it, they just did it without bothering to check what everyone else was ''just doing" or had done or whether it made sense.

          In a just world, this would be reflected in their valuation, but see the first sentence.

    • jwz says:

      Your story is tragic and unsurprising, but that doesn't mean the status quo isn't bullshit, regardless of who you point the bone at.

      Please note that the subject of this post says "the software industry", not "Chris Thompson".

    • Lloyd says:

      Twitter is complex enough to have an API? And an API team?

      Twitter is not even complex enough to figure out how it might shorten urls itself. Figuring out a possible redirection for a url and then discarding it if the entry for the tweet is not made can't be that hard. In Javascript.

      Apparently Twitter has started advertising for engineers, so things may improve.

    • The reason I wrote CPANdeps in the first place was to make visible the huge number of dependencies that some CPAN distributions have, and to encourage authors to think carefully before pulling in a module to do something simple.

      Now, I'm not saying that dependencies are always bad, and you seem to have good reasons for them, but dependencies *are* a big problem in perl. Managing them is incredibly painful in large installations.

    • Matt Sergeant says:

      As far as I can tell in the code, the use of dates is simply parsing using strptime and greater-than comparison.

      Seems like overkill to pull in DateTime when perl core has a module for that (admittedly only as of 5.12, but it's on CPAN too).

  12. Paul Crowley says:

    I'm sorry that software developers like you have to put up with this sort of thing; thanks for writing this and thanks for writing Net::Twitter.

  13. Moose is kind of the metacomment on how fucked it all is, though. Now that Perl 6 is competing with Duke Nukem for non-delivery times, classic Perl folks have given up and done their own OO gunk. Which is why I gave up and started doing stuff in Python, despite my long-standing aversion.

    • Duke Nukem has shipped alphas/developer/OMG-The-Blood releases monthly for a year? Why hasn't anybody said anything?

      For all of the things people might say about Perl 6, they are shipping code. They are even shipping code that people find useful and usable. They have been doing so for at least a year now, and trying to recruit people to help them figure out where the finish line is.

      I'm sorry of course that Moose was so awful that you had to move to Python despite your long-standing aversions. It did take a lot of effort to force people to use it, the psychological warfare experiments were very expensive. Would you mind terribly filling out a questionnaire so that we can better target them next time and fewer people will have to suffer your fate?

    • Another way to look at it: the project called "Perl 6" effectively ceased to be a project to ship a new language several years ago. Their main role, instead, became acting as an ongoing skunkworks for Perl. (Technically "Perl 5", but I call it "Perl".) And in this light, it's been rather brilliant.

      The Perl 6 hackers keep inventing crazy shit, and every so often the practical Perl community says "Hey, that's nifty" and ports it into Perl, either as a CPAN module (a la Moose) or as a core language feature (which is helping to feed the regular release schedule that stable-Perl's been on for a few years.)

      The result of this is Perl's own inexorable transformation into... if not so much a new language than a new set of community-driven idiom, enforced to some degree by popular CPAN modules having fat new dependencies, as JWZ discovers. I think the end result is really cool; I actually love working with Moose and some of the major Moose-powered projects (like Catalyst and DBIC), which I use daily for both client work and personal projects. But I can totally see the shift not being for everybody, just the same.

  14. John Dalton says:

    You might be interested in TTYtter which is a single perl script with no dependencies (other than cURL). It's primarily an interactive client, but with support for CLI scripting and custom extensions. I didn't write it, I'm just a happy user.

  15. ank says:

    1. I want simple software that works.
    2. I use Perl.

    These two are mutually exclusive (lwall and his minions are insane: proof and satire).

    • zby says:

      What a wonderful satire - I guess you must really be adult.

      • ank says:

        Me? I'm very, very immature, thankfully. If it bothers you, I must be doing something right.
        Perl 6 has a joke of a logo and a joke of an intelligentsia (people like lwall or Shane Warden aka chromatic.) It's not my fault you are in denial. In fact, I was too - for the longest of times.

  16. osfameron says:

    Try cpanminus instead?

    $ cpanm Net::Twitter

    produced, in my case (on my non-dev machine) 42 lines of output, to install 5 dependencies.

  17. jesse says:

    Yes, that depchain is sucks. The reasons for its craziness have been explained so I don't need to go there. Discoverability for CPAN modules is ok, not great. As several commenters have pointed out, there are lighterweight alternatives, but none of them are named Net::Twitter, which means that users who aren't involved in the community may not find the "right" thing by default.
    CPAN.pm's verbosity sucks. It's been getting a lot better, though. Newer releases cut down the verbosity. And cpanminus is nearly silent.

    Now, about that Moose failure. I know it wasn't the point of the rant, but would you mind (if you still have the 10403 lines of output) posting (or mailing jesse@perl.org) the failure. I'd like to make sure that it doesn't happen for anyone else.

    • jesse says:

      As a meaningless statistic, on what I'll release as Perl 5.14.0 RC1 in a couple hours, "cpan Net::Twitter" produced a still-unacceptable 4529 lines out output, but at least it all _works_.

      We know things are fucked. Perl 5 is not exactly young software. Unwinding the fuckage takes work.

      We're getting better. Not as fast as we'd like, but we're making progress.

      -jesse

  18. Last i checked "dependency hell" meant that either there were no means to resolve and install dependencies automatically or deps were fundamentally broken so that automated systems were rendered unable to resolve them.

    More importantly though: Please make those 10k+ lines available, since it seems you managed to find an edge case that isn't already tested for.

    • jwz says:

      So what's your point? Dependencies were not properly resolved. I pushed the shiny button and did not receive runnable software.

      I mailed my log to Jesse as requested.

      • My point being that you're implying a much worse situation than there is. It means: "Either there actually isn't any automation software; or that software is blowing up thanks to circular or conflicting dependencies, that mean not even upstream patches to deps can resolve the situation, so you're completely stuck."

        In your case you have a perfectly servicable automation mechanism that requires no human intervention in most situations and was able to to resolve the dependency chain, but merely failed to install one piece of it because there was a rare bug occurring with your system configuration. Reporting it upstream (best in #Moose) will give you a bugfix quickly, after which CPAN can pick up on it and things'll work fine. (And if you're impatient they'll happily tell you how to work around things quickly.)

        You're never actually stuck or forced to install every single individual piece of the dep chain manually, but as a widely-read blogger you're now making big parts of the internet think that's what perl gets you. I think that's unfair.

        Thanks for sending in the log though! :)

        • jwz says:

          Regardless of what you think I was implying, I described what happened.

          Heaven forbid that such facts should besmirch the good name of Perl.

          Apparently you have some very specific definition of what the words "dependency hell" means. Whatever, dude. The dependency list is ridiculously deep, and cpan failed to resolve it. I consider both of these to be hellish states.

          And wow, I think it's been ten years since someone told me "instead of complaining about it, go beg for help on IRC." Is it's the 90s again? Cool, I'm gonna order something from Kozmo!

          • Yeah, now it's my turn to say wow. :(

            "Beg"? I only suggested it because it can give you ridiculously quick turnaround times and because i thought it would be useful to know that there is such a direct line available. Of course you're free to also mail the rt queue or complain on your blog or do anything else that strikes your fancy.

            I'm sad that you'd take such offense at what was meant to be helpful information.

            As for dependency hell, those programmers whose opinion on the matter i know use the same definition, so i didn't regard it as particularly specific. Apparently we disagree however, so i think it's pointless to discuss that more.

            • Don says:

              "I'm sad that you'd take such offense at what was meant to be helpful information."

              you're new here, huh?

  19. Sean Conner says:

    If you want a simple way of posting to Twitter (now that they've changed everything):

    1. Get oauth_sign and compile it. You only need the standalone executable from this.

    2. Get http_post and compile it. You only need the standalone executable from this.

    3. Get your CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN and ACCESS_SECRET from Twitter. I don't have notes about doing this (sorry), but they're now required.

    4. Write a simple shell script that looks like:

    #!/bin/bash
    CONSUMER_KEY="..."
    CONSUMER_SECRET="..."
    ACCESS_TOKEN="..."
    ACCESS_SECRET="..."
    URL="https://api.twitter.com/statuses/update.json"
    http_post -v -h Authorization "$(oauth_sign $CONSUMER_KEY $CONSUMER_SECRET $ACCESS_TOKEN $ACCESS_SECRET POST "$URL" status="I can now post to twitter")" "$URL" status="I can now post to twitter"

    It works for me, and it's two small executables instead of a gig of perl code dependencies.

  20. substitute says:

    I agree. I'm a sysadmin and a perl fan, and I know some of the perl developers personally, and I can't stand this crap. It's one of those nerd disasters that makes sense only to obsessive developers: test suites suitable for naval invasions, grand frameworks of everything, modules to support module modularity within the module framework.

    I mean, Rube Goldberg's shit at least WORKED.

  21. Lloyd says:

    You think it’s an accident Apple went with Twitter for iOS 5 single sign-on? Please. -- MG Siegler, Facebook PR: Tonight We Dine In Hell!, 17 June 2011.

    Twitter's infrastructure is the shakiest and flakiest I've seen. It took years for them to start handling their own shortened urls. And now they're setting single sign-on standards?

    Still, at least it's not Microsoft Passport, right?