more syndication nonsense

A followup to my previous post, since you're all probably like me and don't click on "previous twenty links" very often. (Yo brad, when's that notification system going to actually exist?)
  • URLwatcher: pretty straightforward, but it just emails you a list of changed URLs; I want the new content to show up on my friends list, or perhaps in my inbox.
  • I can't even figure out how to make it do anything useful.
  • I think I figured out how it works, and it's just abyssmally bad.
  • newsmonster: apparently a Mozilla plugin of some sort. *plonk*
  • stapler: this looked promising at first, but apparently it's a plugin to some standalone commerical program called Radio UserLand (which is, incidentally, one of the stupidest names I've ever heard ever.)

So syn_promo is the community that exists because Brad still, for some inexplicable reason, won't add the ability to get a list of all the syndicated feeds? Nice.

I've started writing a scraper, and it's actually going fairly well, since nobody but me actually writes HTML by hand any more, so the sites I want to scrape are pretty easily parsable. It's not done yet, but if you want to play with it, it's here: cheesegrater. This thing will generate .rss files from a list of web sites, and then I suppose I could periodically push them out to and lj-befriend them from there. But, I only have like 1.8 syndication points left, so that leaves me with the options of:

  • Talk Brad into giving me like 30 more points; or
  • Hack up a way to merge all the sites I want to watch into a single RSS feed (which will be tricky, because they don't have, like, timestamps or anything to sort by. Maybe there's a tool that does this already?); or
  • Give up on the idea of using my friends list for this, and use some standalone program or something instead. That would suck. Though I guess I could then scrape my lj friends list into whatever other thing I end up using instead. Hmmmm...
Tags: , ,

13 Responses:

  1. alanj says:

    I'd be surprised if your syn point limits weren't higher within, say, three weeks. The current limits need changing, the only question is how to change them.

  2. rahaeli says:

    I write HTML by hand :)

    And we've been kicking around a few proposals for somehow buying more syndication points, but nothing's gotten past half-baked yet, and certainly not to the level of "present it to brad and get it approved".

    • alanj says:

      Aside from the "buying higher limits" possibilities, I want to see syn point limits for permanent users go way up. Maybe raise it for paid users as well, but definitely for permanent users. (Not that it's my call, but that's what I'll be pushing.)

  3. retrodiva1 says:

    It was good to see you last night and I must say I was really impressed that you recognized me in my green camoflage on Halloween.

    I think you'll find that there are still alot of hand coders out there. Here's an excerpt from one of my posts the other day:

    "On Friday I was directed to another internal website that my co-worker wanted me to see so that I could try implement some of the stuff in it on our site. When I viewed the source code I could practically feel my eyes welling up with tears of joy (no it's wasn't the antibiotic drops). The code contained beneath that humble veil was beautiful, astounding almost magical, well maybe not magical. It was however the best, crisp, clean, nicely laid out hand code (with comments even) that I had ever seen. It was easy to read, easy to follow, generated an amazingly pretty set of dynamic pulldown menus and the beauty of it all was it's simplicity. EVERYTHING I aspire for my websites to be.

    I came to work this morning, found out who the person responsible was and called him to tell him what a magnificent job he did. I felt like a giddy groupie and if he hadn't been up in our Oregon office I would have walked into his office with a copy of the code and congratulated him in person. Sigh..."

    Hand coding done well is a truly beautiful thing.

    • bdu says:

      indeed, I'd rather build a site that lacks some flare but is simple to maintain than the opposite. And there's a level of control that hand-coding pages gives you that there is no substitute for.

  4. insomnia says:

    Like I said... I've yet to be impressed by any of the scraping tools out there. I'm glad you wrote a scraper, frankly.

    I'm especially upset about myrss, which has the potential to be really powerful. All it does is greatly simplify the ability for anyone to horridly unusable feeds.

    "So syn_promo is the community that exists because Brad still, for some inexplicable reason, won't add the ability to get a list of all the syndicated feeds? Nice.

    Indeed. It's also where people figure out kludges to work around LiveJournal's existing RSS support, and where people maximize their pittance of points by hopefully attracting others to sign up for the same feeds they do.

    As far as a short term fix to your struggling syndication points, you're missing a few alternatives:

    - Post a list of your communities that others might not know about to syn_promo. (Most effective method, though it feels entirely too much like a pyramid scheme...)
    - Mention the new feeds you create in your journal.
    - If you have more than one paid account, subscribe the same feeds to your other account too, thereby splitting the point cost across multiple accounts. You can also subscribe one account to all your most expensive feeds, then use this trick to create an RSS feed out of your friends list, which you can subscribe to with your main account. (Yeah, this is *really* pushing it.)

  5. baconmonkey says:

    so maybe this is just too obvious, but why bother trying to force it all through LJ? square peg, round hole.
    why not just have some part on that shows you all the updates? like a module that periodically polls your favorite sites for updates and then spits out nice chunks, then annother module that gathers up all the fresh chunks and displays them.

    I mean, wouldn't having a mere 2 sites to check be vastly better than having to check 2 dozens sites?

    hell, you could just have the display module iframe your friends list.

    • jwz says:

      why not just have some part on that shows you all the updates?

      Only because I'm trying to avoid having to write the code to implement that.

  6. naturalborn says:

    newsmonster also works in konqueror and opera, and I believe its scraping code is separate and can be re-used.

    It's written using XUL, which is an interesting choice, but it's good to see someone at least trying to use the supposedly general-purpose GUI engine which delayed mozilla development by several years.

  7. deus_x says:

    While I'm tickled to see LJ closing the circle with pulling in syndicated content, I use AmphetaDesk as my desktop content aggregator, along with an outliner-based template hack I made for it, AmphetaOutlines.

    As for my friends list here on LJ, I made a custom hackish template here to render my friends list as RSS:

    (Do a s/deus_x/jwz/ to play, of course.)

    I like having the aggregator here, personally. Use my own resources for it, hack on it at my whim, don't have to worry about syndication credits, etc. But I still want to get into hacking LJ's syndication works for people who don't want to run their own aggregator.

  8. brad says:

    dude, it's not like anybody's sent me a patch to add listing feeds. why do I have to do everything? I don't even want it. I want to spend my free time hacking on my fun projects. :)

    But I upped your syn quota and I can continue to do so.

  9. andrewducker says:

    I don't know if you're kidding about Radio Userland, but if not it's a fantastic piece of kit.

    Download the demo and give it a go.