everything's all busticated

I'm deeply dissatisfied with how the internet works these days.

Don't get me wrong, I think things are better now than they were before: every day, there more interesting stuff out there than the day before. There are more voices, and by Sturgeon's Law 90% of them are crap, but that other 10% continues to increase in raw numbers, and that's great.

But it sure was a lot easier to manage stuff back when it was all just email and USENET. There are two pretty basic ideas that have all but vanished in the interfaces we use to acquire information, and those are "subscription" and "mark read."

<lj-cut text="--More--(10%) . . . . once I start, I just can't stop.">

There are dozens of web sites and alternate blog systems out there that I would read if I didn't have to poll them: if I could just sit down in front of my Appliance (mail reader, news reader, web reader, whatever you want to call it) and be presented with all the things that are new since last time.

The way it works today is, I have to click on each of my bookmarks in order, and look: nope, nothing new there. Nope, nothing new there. Livejournal is like that too, to some extent, and it's especially infuriating when there's an interesting thread going on in the bowels of a comment on someone else's journal: if I want to see updates, I have to keep clicking back there to see if anything has been added, and scan through the entire thread while hoping to spot any new entries that happened to appear in the middle of the page. Slashdot has the same problem, but at least there it's easy to momentarily turn off threading and sort by Most Recent First.

There are a bunch of sites out there that implement "URL watchers" that will notify you when the modification-date on a page has changed, but that results in a pretty lame interface. And it doesn't do anything sensible with the message-oriented sites. It's impossible to do anything about it, because every web site invents their own navigation system from first principles: there was no convention for them to follow. It's as if instead of having SMTP, we just have one big directory full of text files, with every author making up their own naming convention to encode topics and recipients.

I'm tempted to write my own URL watcher, but I know that it will end up sucking as much as all the others. And my experiences with webcollage have shown me the futility of doing "screen scraping" and trying to parse HTML generated by other people: the effort of chasing that moving target far outweighs the payoff.

There are lots of folks out there who have journals that I enjoy reading when A: they have updated them and B: I remember to check, but I don't read them very often because it's not easy (which, today, means "they aren't on LiveJournal.") But clearly "everyone should use LiveJournal" is not the right solution. Cool though LiveJournal is, that doesn't scale. What we need are applications that are able to present a common front on all the disparate sources out there.

For example, I enjoy Brunching Shuttlecocks, and always read the new items -- but that's only because it has a Slashbox. If it didn't, I'm sure it would be on the long list of sites that I like but rarely read.

Some people will tell you that RSS/RDF/XML/whatever-they're-calling-it-today will save the day here, someday. Maybe it will, but from what I've seen of it, I can't imagine how, since there still doesn't seem to be any convention to distinguish between headline feeds, like the DNA Lounge Slashbox, and full-content feeds, like this one of Boing Boing. From what I've seen of it, RSS is really good for headline-lists like Slashboxes, My Netscape, etc., but not for much else. There's still no notion of individual messages and threads: there's still no way to have it omit things you've already seen, and show you the things you haven't yet seen, even if their hierarchical and chronological orderings are totally different.

Headline lists are inherently about publishing rather than conversations, which is what makes them simpler. The one editor, many readers model is easier to manage and present than the many-participants model.

I still can't decide whether, when I write DNA updates, I should just link to them from here, or paste the HTML and duplicate it in both places. Neither is really the right thing. brad mentioned that they're working on a way of making it so that someone on LJ could subscribe to an RSS version of another site and have it presented as a regular journal entry, so maybe that's a better way to go than either posting a link, or pasting the text. I think that to make this work requires coming up with a new RSS-based convention for all participating websites to conform to. Maybe there is one already and I just don't know about it.

It's especially frustrating as I spent so many years working on mail/news readers, thinking about these problems and fine-tuning interfaces to make it easier to deal with vast quantities of information... and now the most interesting information out there is in a form that mail/news readers can't touch at all! All that thought and effort by all of those smart people is suddenly inapplicable. On the one hand, we have a class of applications with decades of effort put in to making it easy to manage large numbers of messages in them. And on the other hand, we've got ten zillion web sites full of messages, all in a format that no message-handling-program can come anywhere near.

I think I'd be happier if every single comment on every single journal in the world got emailed directly to me; then it would sit in a folder somewhere, threaded, until I chose to read it or delete it. (Though USENET is obviously a better model for making this scale than email: the model I'm thinking of is one where every message-oriented web site could be viewed as a newsgroup, or something analagous.)

brad also mentioned that the LJ crew are working on a notification system that will let you, e.g., set things up so that you get sent mail when someone makes a new post to a thread you found interesting, which will help, but it's still rather halfassed: it will send you a message to let you know that there are new messages. Presumably it will even be able to append those messages: but still, you end up with a mail folder full of messages from lj_dontreply@livejournal.com, instead of the messages themselves. Your good message-handler informs you that there are updates in this other halfassed message-handler over there.

And don't even get me started on the travesty the world has become when people think that Yahoo Mail is a mail reader, and Google Groups is a news reader. That's like comparing a Vespa to the Space Shuttle.

How can the content get continuously better, while the tools to manage it get continuously worse?

It's all quite mad, I tell you.

Tags: , , , , , , , ,

16 Responses:

  1. otterley says:

    What's your idea of a good newsreader? Netscape Mail/News? slrn? tin? nn? GNUS? Outlook Express? Forte Agent?

    My opinion on the matter: As they said in Fight Club, "Use SOAP."

    • jwz says:

      I don't read Usenet any more. I have a bookmarked Google Groups search that I periodically use to scan for recent posts about my software or the club, but that's it. I guess it seems to me like all the interesting stuff has moved to web-based forums.

      I still read my mail with Netscape 3.02. If it was pried from my cold, dead fingers, I guess I'd go back to something Emacs-based (VM or GNUS, whichever seemed best at the time.)

      But really, for the purposes of this debate, my idea of a "good mail or news reader" is one that has threads, folders, and knows which messages I've already read. So that's all of them.

      • brad says:

        Does Netscape 3.02 support UTF-8? (I'd think not.)

        Do you use the LiveJournal setting to transcode all your comment emails into ISO-8859-1?

  2. ex_meta says:

    Having built a little RSS aggregator, yes, duplicates and working out what's been read before are the major problems.

    There are lots of interesting discussion forums that I just refuse to take part in because they only have a web interface. I told someone the other week, make your private discussion forum available as e-mail or NNTP news, and I'll join. There's no hope of my adding yet another web site to the list of sites I go to each day; I'm working hard to reduce that list, not make it longer.

  3. dormando says:

    For LJ's specific notification system, it should be possible to have it presentable in several different ways. Although it can't be quite as useful as the "syncitems" protocol mode (which you basically feed a date, and it'll tell you everything that's changed since then. I wonder if this could just be extended). I've noticed http://freshmeat.net/ does something similar, which is handy.

    For LJ, events could pend until you poll them with a specialized client (or web interface). Then fetch full information as needed.

    Of course, this all still sucks, since I don't have a "weblog" client on my home machine that can read a standardized protocol for leeching weblog events, or web "events" in general. HTML (along with just about everything else people throw across HTTP) is a client/server pagemaker, not something useful for organizing a feed of information.

    ... It's all very messy indeed.

  4. rahaeli says:

    I miss Usenet. I miss usenet from back when it was readable. It's possible to have a good discussion on usenet these days, but the sheer effort of wading through/killfiling/etc all that crap makes it exhausting to try. And a lot of the smart people, a lot of the people I'd want to listen to, have either never heard of Usenet, or think that it's dead.

    (My girlfriend had never even heard of Usenet before I went off on one of my rants about The Way Things Used To Be.)

    For bookmark management, I use iCab as a browser, which has a Check Hotlist function, but that doesn't change the fact that you have to manually go and read things that it flags, and nine times out of ten they're not actual changes at all. And I have the same problem with people who have non-LiveJournal journals or blogs; remembering to check them.

    I don't think that RSS is ever going to be the solution; for one thing, as you've pointed out, it's good for headlines and bad for full content, and for another: it requires the site owner to make an effort, to produce the RSS feed. Most of the non-blogs I read are run by decidedly non-technical people. Until the popular blogging software (Blogger, MoveableType, Greymatter, etc) makes providing RSS feeds as easy as clicking an option in the control panels, that just won't happen. The solution would have to be something that the reader controlled, not the author.

    No ideas here, really; I'm just musing out loud...

    There've been discussions about syndication systems over in <lj comm="lj_dev"> on and off, and you used to be able to get an RSS feed of your journal by going to, for instance, http://www.livejournal.com/users/jwz/rss -- but that broke a while ago, and it hasn't been satisfactorially fixed the last time I looked at it. I seem to remember a discussion about an NNTP LiveJournal gateway, too, which would be wonderful, but it's suffered the usual fate of volunteer projects: death due to lack of developer interest.

    (it's bad enough that people think of Yahoo Mail as a mailreader, but when they start thinking of Yahoo Groups as listserv software? ...not like half of them would know what a listserv is... I occasionally wish I could roll the internet clock back to about 1997 or so.)

    • jwz says:

      I don't miss usenet at all. It collapsed under its own weight, plus the weight of spam and lack of media-richness, but thousands of other things sprang up to fill the gap: all those web communities are usenet. Just with worse interfaces. So I don't miss usenet per se: I miss the better interfaces.

      Netscape used to have one of those "check for updates" thingies in bookmarks too (maybe still does), but I didn't find it terribly useful.

      • rahaeli says:

        Just about any system is never going to scale to support the kind of traffic that Usenet was pushing at it height, though. (I don't know what a full feed is up to these days, but I do know that full usenet minus the binary groups used to be several terabytes. I can't imagine it's gotten smaller.)

        I think that's a problem that we're going to face more and more as the Internet keeps getting bigger -- Sturgeon's Law and all that. There are places like Advogato that use complicated trust metrics to parse out what each person thinks is worth reading, but those don't scale, either. I'm working on a version of that system that's designed to help fan fiction readers find stories that they might be interested in reading, based on what they've found good before, but the problem I'm running into there is the problem of database integrity; you have to strike a balance between allowing for differing tastes, and guarding against database pollution and the general lowering of standards.

        The web's got that problem too, in addition to the interface problems. Google makes it easier to deal with, to some extent; their heuristics make finding what you're looking for a lot easier. But even they are starting to break down under the volume, especially since people are learning how to fool Google.

        I sometimes wonder if an 802.11b access point in the back of my skull would let me filter information more efficiently.

      • usenet remains the superior venue for free online porn aquisition.

        you just can't argue that.

  5. brad says:

    I've mused over an NNTP and/or IMAP gateway to LiveJournal before, but haven't had time to really get into it.

    Would you be interested in helping out at all with that? I don't need help with implementation as much as somebody to bug when I'm not sure the right way to do something. Since you're the mail & news expert, I'd feel much better working on a gateway if I had somebody like you to bug with questions.

    • otterley says:

      You don't want that. Really.

      What you really want is a SOAP server that's capable of supporting all the methods the Web server and other clients support. Hell, make them all SOAP clients. Publish the interface, ta-da, anyone can build components that are LJ-aware.

      • brad says:

        To do ANYTHING interesting, you need to call so many functions that the latency in SOAP alone makes any 3rd party components slow as fuck.

        Not to mention the line between exposing enough of the real internal API such that it's useful for people (see "slow as fuck" above) yet restricted enough that people can't do malicious/annoying things.

        We do support XML-RPC [1], but only to do high-level things.

        If we wanted to make an IMAP gateway (which would be neat...), it's not something a 3rd party could efficiently implement.

        [1] http://www.livejournal.com/doc/server/

    • jwz says:

      Sure, if you're hacking on NNTP stuff, I'd be happy to talk about it (I can't promise how much time/enthusiasm I'll be able to muster up for actual hacking, but talking, sure.)

      The kind of thing I'm thinking of is something like, what if there were "livejournal", "slashdot", "advogato", etc groups, with arbitrary numbers of subgroups, all marked as moderated. They could share feeds with each other, but posts would all get sent back to the primary host for processing (due to the moderation.) There wouldn't have to be any one site that took a full feed of everything (maybe ISPs would want to do that, but maybe not.) One could just point one's news reader at news://news.livejournal.com/ to read LJ groups, and maybe, if the LJ folks were in a sharing kind of mood, that server would happen to have Slashdot groups on it as well.

      But this is probably doomed to fail, because the NNTP protocol is not expressed in XML and other buzzword compliance, and so the modern breed of propellorheads will reject it out of hand.

      It's also kind of predicated on everyone having news readers that can display HTML inline well, which is probably still a distinct minority of them. (And there's the second reason this will fail: the web folks will hate NNTP because it's not smurfy enough, and the people in charge of the news readers will hate it because the idea of even a privately-operated newsgroup that allows HTML will make their knees jerk so hard they'll be in danger of breaking bones.)

      So maybe it really would be better to invent a different protocol that looks on the surface like it has something to do with RSS. For much the same reason that Java, which is to a large extent a dialect of Lisp, has an Algol-like syntax: to sneak one past the chimps.

  6. baconmonkey says:

    when feeling nostailgic, play with this