routing around RSS damage

I can't sleep, so pardon me while I punditate.

The latest trend over in syn_promo seems to be, "gosh, when we add RSS feeds of sites, we'd better make sure the author is ok with that first!" This because some doofus had a hissy fit when he found that -- gasp! -- someone was actually making use of the RSS feed that he himself provided, in exactly the way that RSS was intended to be used. He later retracted his hissy fit, but now everyone is all paranoid about checking first before adding feeds.

I think this is completely silly, but I can also see that this isn't going to go away: as time goes on, and as RSS gets more popular, more people are going to be bitchy about who aggregates it for whom, and are going to bend over backwards to break things (like the oh-so-many sites that already provide crippled RSS feeds that include only the first few words of the entry, making you click through to actually get any coherent content.)

My prediction: LiveJournal in particular, and RSS-aggregation sites in general, will cease to be useful as a tool for combining the updates from many different sites. It's the late-90s "portal" game all over again, but in reverse: where it used to be that people like Netscape and Yahoo and Alta Vista were trying to prevent you from ever visiting anyone else's server, now all the little guys are trying to get you to hit their server instead of viewing a cached copy elsewhere (which is mystifying, in these days of zero banner-ad revenue.)

And yet, it's really convenient to have one place to go to read all of your subscriptions.

So when that happens, people like me (those with the time and skill to do so) will stop using publicly-available web sites like LJ to aggregate things, and will end up setting our home pages to be local files instead. When it comes to it -- when the only way to get a decent RSS feed of a site is to roll your own and keep it secret -- those of us who are able will use tools like Cheesegrater and Portalizer (or their hypothetical less-halfassed descendants) to scrape other sites and combine them locally, forging user-agent strings as necessary so that our cron jobs don't look like robots, but just look like eager humans wastefully hitting reload. Then we'll have our aggregation and convenience. We'll also have the aggravation of chasing the tail-lights of HTML changes, but we'll do it anyway, because the short-sighted will have left us no better choice.

Shame for everyone else, though: the folks who don't have the time or skill to do this kind of thing.

Like Gilmore said, "The Internet treats censorship as damage, and routes around it."

(Yeah, censorship isn't really the right word, whatever. Perhaps a more apt quote would be the Jeff Goldblum character in Jurassic Park: "life finds a way." You'll often hear cypherpunk weenies with poorly-thought-out philosophies trot out "information wants to be free" as some kind of pseudo-socialist Utopian vision, but the point is, information "wants" to be free in the same way nature "abhors" a vacuum: it's not some moral view, it's just the natural state of affairs. It's the path of least resistance. It is "the sound of inevitability.")

Tags: , , , ,

33 Responses:

  1. naturalborn says:

    I think you're way too pessimistic - there will be sites which make it hard for you to read them, but they'll be in a small minority, and people won't read them.

    Unfortunately people who sit around thinking about RSS development need something to get their panties in a bunch about. There isn't enough to it to be interesting to spend all day working on, and they certainly haven't given any thoughts to the limitations of the static file mindset.

    • jwz says:

      Well, so far, of those sites that provide RSS feeds, the ones that go out of their way to make it hard to read them seem to be, if not the majority, then pretty close. I'd guess that slightly under half of the sites I've been interested in that do have RSS feeds have had useless ones.

      Of course most sites still don't have feeds at all.

  2. jes5199 says:

    it's odd that hackish statements about information
    are always met with skepticism

    when's the last time a reporter said to a chemist or a physicist
    "i don't think you're right about this whole 'Entropy' thing, I expect perpetual motion machines to appear any day now"
    ?

    is it that we havent made a rigorous scientific proof?
    or that it's too obscure to understand?
    or that it just hasn't been demonstrated enough?

    • jwz says:

      It's easy: people understand "property" in terms of physical objects, and now they've been told that information can also be a subject of property, so they expect it to work the same way, but it doesn't, at all. The notion of "intellectual property" has poisoned the way people think about it. They look at events as property transactions which are more rightly characterized as, "you and I will agree to keep a secret."

      There is a lot of money to be made in the business of secrets, of course. But it works differently than posession of objects, since information doesn't have laws of conservation.

  3. icis_machine says:

    so you're saying pointcast will never make a comeback? *sob*

  4. deus_x says:

    Centrally-hosted aggregation sites are already all but useless. I read all my RSS subscriptions, and my LJ friends list (via an RSS template) with AmphetaDesk, which is a little desktop HTTP server whose only app is to present an aggregation of your feeds in a browser, and periodically refresh all the feeds. There's also Radio UserLand, which includes aggregation and blogging features on the desktop, as well as practically all the facilities of an OS with a scripting language under the hood. Kinda emacs-ian in that respect.

    As for mangled RSS feeds, well, I don't mind feeds with just excerpts, but I predict that sites not providing decent RSS feeds will see a much lower growth in traffic as personal aggregators become more popular

  5. ex_cortana says:

    RSS is the source of almost all my news these days, InstaPundit, Doc Searls, Wired News, C|Net, all sorts of places are using RSS to push out news and give extracts which will allow you to choose what content is valuable.

    When is LJ going to learn that full featured RSS feeds are the way to go instead of just headlines?

    • voidstar says:

      I've been asking for RSS feeds from livejournal that include at least an abstract of the post and preferably all of it for about 13 months now. Don't hold your breath.
      http://www.livejournal.com/talkread.bml?itemid=14311193
      http://www.livejournal.com/talkread.bml?itemid=14192455
      http://www.livejournal.com/talkread.bml?itemid=15964954

      • insomnia says:

        So have I, Julian. So have I.

        It's one of those issues that generally doesn't get addressed until a bunch of people make a stink about it and someone codes it. Generally, that someone is Brad, often because he makes it difficult for others to code patches for LJ that aren't rejected (or more likely, ignored...) for aesthetic reasons.

        That's why <lj user="deus_x"> and <lj user="marcpasc"> created styles within LiveJournal that at least allow paid users to generate RSS feeds for their entire journal. You can create a feed of any paid livejournal user (or paid community) that contains the whole post using the following format:
        http://www.livejournal.com/customview.cgi?user=username&styleid=32679

        I know this doesn't solve the issue of description tags, or for that matter many other things that it would be nice to support in LJ feeds... I wish I knew the answer there. Perhaps if people were to submit patches, mentioning them loudly to those also interested in syndication, and then doing a bit of brow/ shillelagh beating, both with Brad and in public, asking repeatedly for any changes that need to be made before the patch is approved, then maybe things would get done. Still, it's like pushing a rope, isn't it?!

        Sometimes you have to wonder whether the real answer is that Brad would rather not give people good feeds, at least unless they pay $$. If so, that hobbles all the other LJ Server sites.

        LiveJournal isn't just a business entity. It should be an open source application that creates interconnectivity between various LJ server sites by design. I don't believe in balkanization, but I have to admit that LiveJournal is far from open. If it were up to me, the decision making process for what enters the codebase and what doesn't would be far more democratic, public, and transparent. Unfortunately, it is none of those things.

        (This is where I throw my hands up into the air and walk away. Don't take it personally...)

        • jwz says:

          I don't think you can really blame Brad for this until someone has coded a patch to do it that he has rejected. Sure, it'd be nice if Brad would implement it for us -- but apparently he's not interested. Ok, someone who is should do it instead.

          • insomnia says:

            I don't blame Brad for not coding things that he never said that he would code... I blame him for not doing the things that he said he would do, despite repeated requests. This isn't a unique situation, either.

            When you allow patches to fall into the proverbial black hole, it creates a real disincentive for anyone to contribute.

            And yeah, apparently he's not all that interested. If so, he should let other people handle the process of approving new patches more transparently and get out of the way, rather than just implementing the things that interest him when he feels motivated to do so.

            • insomnia says:

              In other words, LJ has become a lot bigger than Brad. People would benefit greatly if the open source project was seperated from LiveJournal.com and run in a more open manner, because the interest of the business and the open source project don't necessarily coincide.

              • brad says:

                So to what side do I step after this imaginary split?

                I hate the business stuff, but I'm not going to hand over the company to some random person. But I know the codebase best. *shrug*

                uJournal talked about forking recently .... how'd that effort go?

                • insomnia says:

                  If there was a split, who says you'd have to choose between being developing or running a business? It's kind of a moot point that you hate the business stuff -- you own the business. All you can do really is give it away, sell it, shut it down, or have others run it for you.

                  I have no idea how uJournal's talk of forking went -- I presume that not a lot happened, because they don't have a good structure to point people to in order to encourage development from interested parties. Still, it does indicate some degree of frustration with the current system when numerous people are expressing an interest in forking the development.

                  Personally, I'd rather not see the development fork. I'd rather see it move to a more neutral, open environment ( perhaps livejournal.org), and see decisions made by a committee that includes people from Blurty, uJournal, DeadJournal, etc. Although you might not always feel the decisions made were the right decisions for the codebase, you'd still have control over the code you ran on your site, and it would encourage more code, which is the real goal anyway.

                  Such an infrastructure would also encourage more interconnectivity between LJ Server sites, making it a *LOT* easier for people on other LJ Server sites to participate in development.

            • brad says:

              That patch (from <lj user=mart>) was terrible for two reasons:

              a) it duplicated an offensive amount of code, instead of abstracting things.

              b) it didn't work, because LiveJournal at the time didn't support character encodings. we treated everything as bytes, and blindly threw them about. the whole point of LJ's UTF-8 migration was so we could do all-things-XML properly finally.

              Since we've gone to UTF-8, nobody's sent me a patch for better RSS, and I'm too busy with other things. It's really pretty easy. I can't be the only programmer here.

  6. andrewducker says:

    Seeing as I currently get all of my news via Livejournal's syndication, this seems unlikely to me.

    Of course, I'm in favour of the 'headlines here, story elsewhere' approach, as it means I can go to just the places I want on the other sites, but they can make money off of my banner views (or however these sites claim to make money).

    For people that actually value being read over anything else, straight RSS feeds of the complete text is the way to go.

  7. westyx says:

    I think that what will happen is that those sites that provide unfettered rss feeds will prosper, while those that require effort won't. People will eventually give up on battling with revision 31 of some site's news page, trying to get their scrapers to work, and just grab the unfettered feed of another site that does.

  8. mswagner says:

    I'm the guy JWZ is talking about here, the one he describes as a "doofus."

    I'm not sure when I started publishing my weblog in RSS, but it has been at least six months now. I have been using various RSS aggregators for about the same time, and I have come to find them indespensable. Yes, the LiveJournal usage that was objectionable to me at first now appears, upon further thought, to be proper usage of RSS. Yes, I was mistaken.

    My initial outrage was prompted by the assumption that here was a commercial service that was providing my text as to its customers, charging those customers to read my words -- and not giving me any of the money! I later discovered that my assumption was not entirely true. (There was some truth to it -- LJ does take money from some of its users -- but I'm choosing to ignore that, at least for now.)

    Information wants to be free, but the supermarket, the bank that holds the mortgage on my house, and the doctors all want to get paid.

    JWZ, your post here

    http://www.livejournal.com/talkread.bml?journal=jwz&itemid=126859&thread=555659#t555659

    about intellectual property links IP with secrets, when that is not the case. On the contrary, the New York Times is hardly a secret, it is available to anyone with a bit of pocket change to pay for the newspaper, or for free on the Web. Nonetheless, the entirety of the Times is intellectual property.

    As far as I can see, ALL property is intellectual property. This computer that I'm working at had very little value when it was just a bunch of ore and stuff sitting in the ground, it gained its value when people applied their intellectual property to turn it into a computer.

    As to the matter of my being a "doofus": I always love it when I am patronized and lectured by a self-important rich pissant who calls me a "doofus" for being concerned about the value of my work. YOU no longer have to worry about money, so you appear to consider the entire concept of "intellectual property" to be somehow flawed. This is baffling to me, because from what little I know about your career, you made your money as a programmer at Netscape, where you ... where you ... why, where you CREATED INTELLECTUAL PROPERTY. Either that, or you inherited your wealth.

    JWZ, this situation came to a happy ending despite you, not because of you. Some other users of LJ, and one or two prominent weblooggers, to the time to discuss the issues with me and persuade me to their point of view.

    • jwz says:

      the New York Times is hardly a secret, it is available to anyone with a bit of pocket change to pay for the newspaper, or for free on the Web. Nonetheless, the entirety of the Times is intellectual property.

      The New York times is not selling ideas, they are selling access to ideas. The only way they are capable of doing that is because there is a contract (via copyright law) to give them exclusive duplication rights to the particular expression of those ideas. The whole point of copyright law is to get people to not do what is natural -- allow information to flow unhindered -- in order to accomplish something which has been judged by society's lawmakers to be a greater good: paying people for thinking. The nature of this process is the management of secrecy: compelling third parties to not share directly with each other: to keep secrets.

      Opinions vary on whether and to what extent this is a good idea, but that's how it works. I happen to think that the way copyright works is mostly pretty good and effective (unlike, say, patents), but I don't believe it's property. It's a contract.

      As to the matter of my being a "doofus": I always love it when I am patronized and lectured by a self-important rich pissant who calls me a "doofus" for being concerned about the value of my work. YOU no longer have to worry about money, so you appear to consider the entire concept of "intellectual property" to be somehow flawed. This is baffling to me, because from what little I know about your career, you made your money as a programmer at Netscape, where you ... where you ... why, where you CREATED INTELLECTUAL PROPERTY. Either that, or you inherited your wealth.

      Actually, I wasn't lecturing you, I was pointing and laughing.

      I've believed "intellectual property" to be an oxymoron since long, long before I happened to make a huge pile of cash on it. Life is full of those little ironies, isn't it?

      But actually, I didn't make my money because people bought intellectual property: I made money because people gambled via stock market speculation. Netscape made almost all of its actual cash by selling services and support, not by selling secrets in the form of copying-control of software.

      But even if that weren't the case, I'd still have done it in a heartbeat. Just because I believe the system is flawed doesn't mean I wouldn't take advantage of it when the stakes are high. I'm pragmatic like that.

      I do always love it, though, when people try and imply that my opinions must be wrong because I made a bunch of money.

      JWZ, this situation came to a happy ending despite you, not because of you. Some other users of LJ, and one or two prominent weblooggers, to the time to discuss the issues with me and persuade me to their point of view.

      Good for them, and good for you. I didn't bother because I had never read your blog, and so didn't care enough to go out of my way to try and convince you of the error of your ways. I read a lot of web sites, and I don't consider it my duty to correct every stupid and misinformed opinion I run across. I'm (honestly) glad that someone who did care took the time.

    • cryllius says:

      My initial outrage was prompted by the assumption that here was a commercial service that was providing my text as to its customers, charging those customers to read my words -- and not giving me any of the money! I later discovered that my assumption was not entirely true. (There was some truth to it -- LJ does take money from some of its users -- but I'm choosing to ignore that, at least for now.)

      I just love this argument. You know, my ISP "charged me to read [your] words" -- should I tell them to cut you a check? Maybe the phone company should get a percentage of all telemarketing sales?

      That's ridiculous. LiveJournal provides an infrastructure just like anyone else, and its users are responsible for what content they put on it or channel through it. To think that the providers of this generic service were instead specifically pandering your words is borderline delusions of grandeur.

    • zapevaj says:

      Hi.

      I don't know you, and have no personal stake in this conversation, but if you're so dumb that you can't figure out that LJ is a non-commerical service (what, did the numerous incidences of "it's free" on the home page not clue you in?), then I'm astounded you can use a computer at all. Not only that, but you threatened to rain down hellfire and legalese on an entity (LJ, that is) without even figuring out what type of organization it is. Nice research skills.

    • ronbar says:

      I have no money; am I more worthwhile than a jwz, in your opinion?

      By reading this, you have agreed to the ever-changing license terms I form in my brain, which state (in part):

      After reading the ingenius and highly valuable musings of Ron Denton Jr. Esq. (henceforth RON), you must immediately and without hesitation expunge them from your brain.  Any and all copies of said musings remaining in your brain after reading are unauthorized and must be destroyed.  A suggested method of destruction involves pouring battery acid into your left ear, waiting until it reaches your brain, and shaking your head vigorously.  This action will prevent the valuable intellectual property of RON from leaking out of your mouth or spewing forth from your fingers in typed or handritten form.  Additionally, you must videotape the brainmelting and send RON a copy.  He's a sick, sick bastard.

  9. bryant says:

    Hm. Jamie's neglected to mention that Mitch's feed is licensed under a Creative Commons license. The specific license Mitch used was the by-nc license, which means it's OK to copy, display, distribute, and perform his work as long as it's for non-commercial use.

    Just putting something in an RSS feed doesn't magically make it owned by the world, any more than putting a machine on the Net gives me the right to hack into it. If Mitch puts a license on his RSS feed, we gotta abide by it. I may be down on intellectual property (and in fact I am), but I also recognize the existing laws.

    RSS is a technology for communicating information. It is not a policy statement. Confusing technology and policy is bad.

    (Parenthetically, I think Mitch should put the licensing information in his RSS feed; it's easy to do and it lessens the opportunity for confusion.)

    Anyhow. I think the point he raises regarding commercial use is interesting, but misguided. Since access to his RSS feed via LiveJournal is not a for-fee service, LJ's use of the feed is not commercial. There's really no ambiguity here. If you couldn't get to his feed except by paying money, that would be a different matter -- but that's not the case.

    He's also wrong in saying that it's only OK because of the intermingling. His Creative Commons license doesn't say anything about that. It says it's OK for me to reproduce his stuff as long as it's for non-commercial purposes.

    • jwz says:

      Just putting something in an RSS feed doesn't magically make it owned by the world, any more than putting a machine on the Net gives me the right to hack into it.

      Putting something in an RSS feed is clearly an invitation to do with it RSS-like things, just like posting something to USENET is an invitation for hosts to store, forward, and archive your message hither and yon -- it's implicit in the medium you chose. Any other interpretation is, in a word, nonsense.

      But that's all beside the point (the point that I care about, anyway.) I couldn't care less about what rules Mitch wants people to abide by to read his blog: it just doesn't matter.

      My point was, when you put something on the web for people to read, they will read it in whatever way they find most convenient. If they don't want to load your banner ads, they won't. If they don't want to obey your Expire: header, they won't. If they want to read it by screen-scraping your HTML and reformatting it locally, they will.

      If authors get huffy over RSS aggregator sites like LJ, then readers will just move to other more hidden mechanisms. It will be more work for the readers, but the authors still won't get what they want.

      • bryant says:

        I'd generally agree with your point. Heck, I'd even remove the words "for people to read." If you put streaming audio on the Web, people will turn it into static files. If you put pictures on the Web, people will cut and dice them. Etc.

        You're still gonna have to live with the legalities of copyright until the copyright laws are repealed, though. (Not sarcasm. It'll happen eventually. I'll be glad when it does.) I'm not gonna try and convince you, cause you're an idealist, which is good. I'm just pointing out that the "RSS implies permission to syndicate" argument won't hold up in court, in case anyone confuses the idealism with case law.

    • joshc says:

      Since access to his RSS feed via LiveJournal is not a for-fee service, LJ's use of the feed is not commercial. There's really no ambiguity here. If you couldn't get to his feed except by paying money, that would be a different matter -- but that's not the case.

      This is sort-of true: anyone can see (http://www.livejournal.com/users/tomorrowfeed); only paid accounts can add it to a friends list (http://www.livejournal.com/friends/add.bml?user=tomorrowfeed)

    • insomnia says:

      I believe that he was wrong about wanting his feed pulled, but some things have to be made clear.

      LiveJournal is a business, and one of their paid features is the ability to add RSS feeds to their friends list. Still, one can say the same thing about a lot of other RSS aggregators, which are either supported by fees or by banner ads.

      I'm a big supporter of Creative Commons, but I question whether the Creative Commons license in question is appropriate if it gives the right for RSS newsreaders to make a digital copy of Mitch's weblog if they are freeware, but doesn't give that right if they aren't. I'll have to contact a friend at CC about this for clarification, I think.

      • bryant says:

        Yeah. It's an interesting and difficult question. I think the problem really highlights /why/ intellectual property rights need to be rethought.

        I suspect that aggregators, paid or not, are somewhat like Web browsers: pieces of software that enable people to read RSS feeds/Web pages. I'm not sure whether or not a court would uphold that, though. Ah, fun fun fun.

      • insomnia says:

        And yes, Jamie is correct when he says that anyone who creates an RSS feed should expect it to be used, often for commercial purposes. While you're at it, Mitch, you might be interested to know that this big company called Google has cached copies of your weblog on its servers.

        This is, incidentally, why people who love the Internet hate those people who whine about "unauthorized linking", or hate the Church of Scientology when they intimidate search engines, threatening legal action in order to take down content. These kinds of selfish acts make the the Internet unusable for the many, and turn the technically savvy few into criminals.

        If an article from your weblog gets published without permission on page 23 of the N.Y. Times, by all means, sue. I'll recommend a good lawyer, even... but if you do something that puts a chill into the room and leaves ordinary people wandering if they can legally use the technology they used yesterday in the same manner today without being a "criminal", you're part of the problem, not the solution.