My current theory as to why the memory usage got so much worse between 10.5 and 10.6 is that the 10.6 garbage collector sucks in the following way:
It only does a collection when a threshold of outstanding collectible allocations has been surpassed. However, CoreGraphics creates lots of small collectible allocations that contain pointers to very large non-collectible allocations: a small CG object that's collectible referencing large malloc'd allocations (non-collectible) containing bitmap data. So the large allocation doesn't get freed until GC collects the small allocation, which triggers its finalizer to run which frees the large allocation. So GC is deciding that it doesn't really need to run, even though the process has gotten enormous. GC eventually runs once page-outs have happened, but by then it's too late, and the machine's resident set has been sodomized.
So I fixed this by forcing an exhaustive garbage collection in the ScreenSaverEngine process approximately every 5 seconds whether the system thinks it needs one or not.
There still seem to be some leaks happening, but I think this improved matters a lot. (The X11 version doesn't leak; these are Cocoa-specific problems.)
Good times.
