RSS of protected LJ friends?

Dear Lazyweb (and brad),

How do I get an RSS feed of my LJ friends that includes friends-only posts?

This method does not include friends-only posts, even when you add &checkcookies=1. The &filter parameter doesn't appear to do anything, either.

There doesn't appear to be any way in the LJ protocol to do this either. I'm about to resort to screen-scraping the HTML.


Update: alierak has the goods: First do this thing, then run his script to generate the feed locally. Yes, this is stupidly complicated and kludgy, but it's working for me.

Tags: , ,

38 Responses:

  1. duskwuff says:

    My solution was to write a custom S2 theme that outputs XML (s2id=17648572), and XSLT that into RSS locally. (Sadly, you can't write S2 that generates RSS directly, because the output filter lowercases all HTML XML tags for some reason.)

    • jwz says:

      So how do you use that? And since when is RSS required to be upper case?

      • duskwuff says:

        So how do you use that?

        Suck down http://jwz.livejournal.com/friends?auth=digest&s2id=17648572, then transforming it using this XSLT. The S2 layer source is public if you want to customize it.

        And since when is RSS required to be upper case?

        <pubDate>. At least one client I've used at one point (don't remember which) refused to recognize the tag in lowercase.

  2. a_0001 says:

    I don't know about the friends page, but you can add each journal from which you want to see friends-only posts to your RSS reader individually using URLs like:

    http://a-0001.livejournal.com/data/atom?auth=digest

    This causes your RSS reader to do HTTP digest authentication, optionally with a saved password.

    If you're using NetNewsWire, however, doing this is likely to infuriate you; the authentication seems to fail frequently, so you'll be prompted to re-enter your password often. I switched to Vienna, which seems to handle this somewhat more gracefully.

    • jwz says:

      Actually it will annoy me because now I have to maintain two lists, one of the journals I have friended and a parallel one of the journals I read.

      My other concern is that I want hard failures: if it's not auth'ed, I want to notice it, rather than discovering, some time 4 months from now, that I had been seeing public posts only.

      (LJ likes to pull that kind of shit all the time too, but usually then I notice it the first time I try to comment.)

      • houdini_cs says:

        This is what a bunch of people were saying when they said that NNW didn't auth to LJ correctly. I've got the same problem.

      • bitwise says:

        I once followed a high-volume feed here, but removed from my default view because I couldn't follow all the posts. Now, if I ever see that feed appearing in /friends I know I've been logged out.

    • node says:

      I don't know about the friends page, but you can add each journal from which you want to see friends-only posts to your RSS reader individually using URLs like:

      It's far easier to massage the OPML links (add ?auth=digest to each link, maybe use atom instead of rss).

      Will you have to type your password in for every auth'd feed? Sadly, yes. It's either that, or some custom style/html scraping hack.

      • foxmagic says:

        Having to type in my LJ username/password separately for every authenticated feed is something I've been complaining about for a long time. I wish LJ would treat each feed as coming from "www.livejournal.com/username" as it used to, but I was told there's some reason they changed this to "username.livejournal.com". I've asked the developers of the Mac clients NetNewsWire and Vienna to change their software to handle LiveJournal authentication more nicely, but they've declined.

        I'd love to read my friends' journals as separate feeds in an RSS reader, but I'm not going to sit here typing in my username/password a hundred times to start with and then another hundred times each time I change my password.

        I was just thinking about it again today when I searched to see if there was a solution yet, and that's how I found this post.

  3. thorfinn says:

    You need a feed reader which supports your login cookie... which pretty much means a feed reader built into your web browser.

    The only other combo that works is Safari + AppleMail (because AppleMail will snarfle your login cookie from Safari).

    There's something about S2 style layer hacks as another option:

    http://community.livejournal.com/lj_nifty/119447.html

    I actually used that for my Dreamwidth account because they don't support an authenticated RSS feed of your friends yet.

    • jwz says:

      No.

      If I follow the URL-construction instructions in that post I linked to, and load that URL in this very browser window I still don't see friends-only posts in the returned RSS. It is absolutely getting my login cookie, and it is absolutely not working.

    • There are several other readers that do this, I pointed out one called Akregator (which seems to use Konqueror, rather than being built into Konqueror), but then you'd need KDE, and that's an entirely different annoying issue...

  4. jayp39 says:

    FAQ 219 says you no can has:

    "While LiveJournal does not offer an RSS feed of your Friends page, you can subscribe to the RSS feeds of your Friends' journals, using an RSS reader."

    • pavel_lishin says:

      I'm pretty sure that still doesn't show their private posts.

      • phil_g says:

        It does if you put '?auth=digest" at the end of the feed URL and you set your feed reader to use HTTP authentication and you give your feed reader your LJ username and password. Apparently, NetNewsWire has problems with remembering your password in this scenario.

  5. alierak says:

    I use "this method", but wrapped with a perl script that logs in. My NNW is subscribed to the perl script, not the LJ view:


    #!/usr/bin/perl

    my $user="...";
    my $pass="...";
    my $login_url = 'http://www.livejournal.com/login.bml';
    my $feed_url = "http://$user.livejournal.com/data/customview?styleid=670763&checkcookies=1";

    use LWP::UserAgent;
    use HTTP::Cookies;
    use HTML::Form;
    use Digest::MD5 qw(md5_hex);

    my $browser = LWP::UserAgent->new();
    my $cookies = HTTP::Cookies->new();
    $browser->cookie_jar( $cookies );

    my $response = $browser->get( $login_url );
    exit 1 unless $response->is_success;

    my @forms = HTML::Form->parse($response->content, $response->base);
    @forms = grep { defined($_->action) && $_->action eq $login_url } @forms;
    exit 1 unless @forms;

    my $login_form = shift @forms;

    $login_form->value('user', $user);
    $login_form->value('response', md5_hex($login_form->value('chal'),
    md5_hex($pass)));
    $response = $browser->request($login_form->click);
    exit 1 if $response->is_error;

    $response = $browser->get( $feed_url );
    $response = $browser->get( $feed_url ) unless $response->is_success;

    print $response->content;
  6. seanb says:

    There's a third-party site that, given your lj credentials (!) will create RSS feeds for each of your friends and communities, including locked posts. It even provides an OPML file aggregating all feeds for import into your RSS reader of choice.

    http://lj.abnib.co.uk/login

    I do love this item from his FAQ:
    Is it safe to share my username and password with you?

    That's up to you to decide. I can promise that I won't do anything nasty, and I hope that my word is worth something. And I've got a long history in computer security, so you can rest assured that I won't be letting any Tom, Dick or Harry have access to the database in which your passwords are stored, but seriously: it's up to you.

    • bifrosty2k says:

      Now that is lulz... Anyone who falls for that is a real genius.

      • cmm says:

        the thing (actually, an earlier incarnation thereof which went by the name "livejournal feed fetcher") was/is available as source, so if you have your own domain you can just install your own copy.  I did, and it works well enough.  until the hosting company decides to sell my LJ password to evil people, that is.

        than again, that Perl snippet above is morally equivalent, so who cares now anyway.

  7. davesmith42 says:

    Livejournal does publish a page that lists the URLs of all your friends' RSS feeds, at
    http://www.livejournal.com/misc/fdata.bml?user=exampleusername

    You can take that, parse out the URLs for every friend, append ?auth=digest to them, grab each of those feeds with something like cURL, parse the XML to find all the individual items, then recombine them all into one crazy-big RSS feed.

    There are boatloads of ugly inefficiencies here, since LJ doesn't (as far as I know) give me a way to specify "only send me the most recent X items" or "only send items after a given timestamp," like Twitter's interface does; you always get the most recent 25 entries for each friend/community you follow, regardless. For me, this leads to a 2MB file with 600+ items that takes about thirty seconds to assemble (due to bandwidth more than CPU). It works, sorta, and only took a couple hours to put together.

    Anyone that thinks it'd be useful, let me know and when I clean up the source a bit more (it's a quickie bit of PHP) I'd be glad to share.

  8. frogworth says:

    Sorry for commenting on an old entry - only just saw it.
    I guess your feed reader doesn't accept authentication within the URL? With mine, I just subscribe to my LJ friends' feeds with username:password@ appended. i.e.

    http://username:password@jwz.livejournal.com/data/rss?auth=digest

    One could do the same for the all-friends feed I suppose.
    But I know a lot of feed readers won't like that. Mine is the stale old Gregarius project, which hasn't been updated in over a year, but works alright for me.