SF's Market Street Subway Is Running on Reagan-Era Technology:
"Our train control system in the Market Street subway is loaded off of five-and-a-quarter inch floppy drives," SFMTA director Jeffrey Tumlin told KQED's Priya David Clemens this week. "We have to employ programmers who are experts in the programming languages of the nineties in order to keep running our current system. So we have a technical debt that stretches back many decades."
Ah yes, the withered programmers from the Nineteen-Hundreds. I'm glad they included these historical fun facts about floppy disks!
For those who've never had to use one -- and keep it far away from any magnets -- floppy disks are still used as the "save" icon on computers worldwide, but they've been obsolete for 30 years or more. Originally a full eight inches across when introduced in the 1970s, floppies later shrunk to five-and-one-quarter inches, and, eventually, a mere three. Early generations were indeed bendable, although the final generation was made of a harder, thicker plastic, with no flop at all. Eventually, CD-ROM technology superseded them before becoming nearly extinct, too.
Here's an article from 2015 about some upgrades, but this sounds like an even older, pre-floppy system:
Muni train control system gets biggest upgrade since the '90s:
In a dank walkway tucked away inside the Van Ness Avenue Muni station is the mainframe that controls Muni trains underground.
Relay racks extend down a cramped hallway nearly 15 feet deep, where dozens of 4-inch copper filaments clatter up and down like teeth. The speedy "click, clack" sound signifies a smooth running Muni light rail automatic control system [...] The clapping beat signifies the proper alignment of the railway's 83 axle (controls) and numerous other track switches. [...]
When this happens, or a track switch is damaged, a green relay cable running the length of the underground track is tripped. This signals the maintenance crew, which then checks each ticking copper relay for those with an off beat, or ones that have stopped altogether. [...]
"Last year, we had water up to here" Haley said, gesturing with his hand to his waist. The relay system is under a city drain, he said. A yellow tarp is strung at the roof of the relay room, while sandbags, a bucket and more tarps are tucked away in the corners. [...]
"These things are hard to replace," Kelly said. "You can't just go to Relays R Us." [...]
The crew that maintains and inspects the relays every night said it is hopeful for the new system -- but it also respects the old one. "I trust it to run another 20 years," said Hoa Huynh, a 20-year maintenance crew member. But, he said, "It's slow. Very slow."
I love the idea that part of the diagnostic process is to stand in this room and listen to it -- "that's not how the tunnel song is supposed to go." An interface that draws big red boxes using a supercomputer is not necessarily easier or more expressive than one that communicates ambiently through the system's inherent musicality.
Previously, previously, previously, previously, previously, previously, previously.
I'm curious where they find the replacement drives and disks. Those things will have hit their MTTF at some point. Probably a decade ago?
Previously! https://www.jwz.org/blog/2022/09/last-man-standing-in-the-floppy-disk-business/
that’s deeply terrifying. I wonder how long before we see embedded computers as floppy disk replacement units.
Programming languages that were popular in the 90s! Heavens!
Did we, uh, they have to use magnets on the drum?
All this talk of new technology. I programmed with languages popular in the 1960s, in the 1960s.
Might be more reliable than a rewrite of the codebase, though.
100% Just ask IRS how their nth attempted reimplementation is going. I would however consider using a floppy emulator. It should be possible to find one that is more reliable than the original physical product.
Yeah, that was the only thing I found unusual there.
I know this is unfair and unfounded of me, as it is surely possible to build incredibly reliable systems in 2023, but I feel more comfortable with the idea of important infrastructure running software from the nineties than I would if it ran something made in the 2020s.
Older is sometimes better, probably because it forces the system designer to minimize complexity. The control system for the Panama canal might be a good example. It was built in ~1913, and it was completely mechanical. The user interface was a scale model of the canal, with blocks of blue marble that were raised and lowered to show the water level in each section of the lock. It used complex mechanical interlocks to prevent anyone from operating the locks incorrectly; you can raise or lower a ship through the locks, opening the gates one at a time, but you cannot open all the gates or valves at once. Apparently it’s never failed; the canal has been closed for other reasons, but never because the control system wasn’t working.
But it’s ultimately still quite simple, because all it allows you to do is turn first this handle, then wait until the water rises, then turn the next, etc. It’s a very simple state machine, and it never has to change. Even though they added new sets of locks to the canal, each flight of locks has it’s own control system so the existing ones didn’t have to change. Compare that with a train system where each block of track might have a train in it at any time, where there are multiple routes through the system, where trains get faster every decade, routes are added and removed at a politician’s whim, etc.
Absolute Block, which appears to be what you're describing for trains, isn't really any different than the canal situation. Neither routes nor train speed need to matter to Absolute Block. Like the canal, the railways did have mechanical interlocking for Absolute Block, that's why a classic mechanical signal box cabin is up a flight of steps - underneath it is a tremendous amount of complicated gears and levers which implement the interlocking, in a space big enough for maintenance engineers to actually work on it.
Eventually the mechanical interlocking was replaced by transistors, and eventually the transistors were replaced by software but it's the same outcome. A signaller who screwed up and would once have found that all their strength won't move the lever (because doing so would create a dangerous situation) in the box, now finds that clicking doesn't do anything because if it did that would create a dangerous situation. The modern UI might do better at pointing out what the problem is though, which can be helpful.
For example a real incident took place in which a mech signaller arrived after "his" box had been subject to a lot of maintenance work, the first train which arrived at his outer signal that night got Danger (ie a red signal), the driver called the signaller after sitting at this signal for a while, signaller tried to release it and it wouldn't budge. He assumed the maintenance engineers fucked up, but procedure is procedure so he tells the driver the usual, "Pass this signal at caution, obey all further signals". Next signal, same problem, same speech, still cursing the engineers. Signaller gets another call from the driver. Same problem? Nope. The driver is now face-to-face with an oncoming train. At caution speed there was enough time to brake and so it's scary but no damage done.
The Danger signals were real, the signal wouldn't release because release is conditional on Points-Locked-Straight for the points (maybe US term is "switch?") which choose whether your train crosses onto the opposite track, and those points had failed in indeterminate state. If the Signaller had checked why the release wasn't possible he'd have discovered one of his dozens of dials and gauges said "Indeterminate" instead of "Points Locked", but he was blaming maintenance engineers instead. A modern computerised system could have said "No, there's a points failure" and shown where, ensuring the signaller's focus was on the actual problem earlier.
As a bonus that old hardware is probably a lot less vulnerable to malicious hacking. The sensors and actuators are probably directly driven by an air gapped control computer. Though air gapping doesn’t always prevent hacker penetration (see Stuxnet), the ancient controller and direct drive would make design of the attack an exercise in bespoke historic reverse engineering.
I recall network admins back in the day would sometimes pipe netcat to /dev/audio 🙂
Still true? There was a big overhaul of Muni metro control systems in the late '90s. Did not go smoothly to say the least! But, at least, I would expect that was the end of the original floppy based programming.
At Big #Software Company™️, we bend over backwards to support customers that simply can't, for reasons either political or organizational, upgrade to meet modern standards. For example we kept supporting our stuff on (pre-open-standards) Solaris until like 6 months ago.
Until you've had to do it you'll never know: some companies are so paralyzed by their own bullshit that asking them to upgrade to anything remotely modern is literally impossible for them. Talking about a company I won't name but it these with Beneral Pholectric.
this is literally the only kind of conversation I see from customers about Solaris at Oracle. I wish people wanted new features from Solaris, but they really just want the
thing they bought in the 1990s to not break. (But I’ve also seen our folks bend over backwards to keep a customer running, from SPARC emulation to bug compatibility with 90s gcc.)
"Don't fix what ain't broke" is a very good policy.
A significant exception is "the machine I am using to connect to the internet that hosts my personal secrets". Yes, install your patches and stop using Windows.
But MUNI using clicky relays? I am 100% down with that forever.
To be scrupulously fair, the customers of Big Software Company™️ that are Microsoft shops are less -- much less! -- of a problem than unix (no I did not spell that wrong) shops that are clinging to "something that worked in 1982 so why would we change it".
But also, yes, click-and-bang relays were so beautifully elegant I would feel bad about telling any customer they had to abandon them 🥺
Just yesterday I had a piece of my critical infrastructure malfunction as a result of a power failure and my only diagnostic sidechannel is the clicking of relays. One burst of the motor at startup. Two bursts of the motor after a DHCP response has been received.
It's not great. But it's also not running fsck or slashing its own throat because it can't verify a certificate chain.
I'm old enough to miss the clean universal logic of the Bell system, but I have to be real and tell customers, look, I'm so glad that Solaris or AIX or HP-UX has worked well for you for 25 years but even Big Software Company™️ can't find anyone to develop on a platform that went end of life in the Clinton administration, BECAUSE THEY'RE ALL RETIRED OR DEAD, so,
Don't even get me started on DB2
Thank you for reminding me of another of the times when a single line of code got a CEO to personally yell at me and that line was:
# define rename hpux_sucks_wet_farts_from_dead_pigeons
https://www.jwz.org/doc/censorzilla.html
I don't see anything wrong with that
I'm in a place where:
HR: Now hiring, people who can program the flight control on the Space Shuttle computers
Me: Uh, you can't, that isn't, there aren't any, uh, this doesn't exis--
HR: THIS IS WHAT THE CUSTOMER NEEDS
Me: I am very sure that the customer is doing it wrong, because there hasn't been one of those in production sinc--
HR: ARE YOU NOT A TEAM PLAYER?
Me: the only surviving one of these is hanging from the ceiling of the Air and Space museum. But you go ahead and put that in the job description. See how many applications you get. (spoiler alert, it's zero)
I feel like "I am hanging from the ceiling of the air and space museum" only counts as "surviving" in a reputational sense. My mess is still on the floor beneath the noose.
I fully admit that NOTHING that I write deserves to be in a museum unless the Smithsonian opens a "bad code that duct taped over a big problem so nobody would notice" branch.
I think "Museum of Duct Tape That Saved Humanity" would be an incredibly worthy museum. Let's name it after Stanislav Petrov.
love it.
I did have a lot of fun though when a customer called me in 2016 looking for an expert on the Sony CellBE.
There are a lot of gamedevs with nostalgic love for SPU programming that would happily take an overpaid gig like that.
I remember in the mid-1990s reading your pages about mcom.bad-attitude and its SGI predecessor (and subsequent really-bad-attitude list) and in my naïveté trying to create a similar newsgroup at my much-more-straightlaced Fortune 500 gig at the time. It, uh, did not happen. Probably something to do with the subpoena risk y’all ran into the reality of. 😔
/cc @andthisismrspeacock @14mission
I should clarify that I read them *in* the mid-90s while they were still recent. This is the world’s longest-delayed fanboying.
I just hope that the current generation has the attitude of "I am selling the fuck out, fuck you pay me" and are not buying into the bullshit of "we are all in this together, this is a joyous journey into the shimmering arc" nonsense that we naive fuckers in the nineteen hundreds got suckered into.
^^^ this
I think the current generation has ANDed them into “We are all in this together, so you fuckers better pay us or we strike”
From your lips to Loki's ears
And, as one of those "open source people", obviously the "we are all in this together" thing has some appeal, but it is absolutely the case that the only time food went into my mouth was as the result of, let's say, something other than collective community action. It was billionaire groundscore.
Yes, but open source can definitely sow the ground first. I got approached for interviews with two separate companies last week thanks to my darn *Perl* work (a few small packages but mostly blogging, speaking, and general social gadflying). Both went on to second interviews with their respective CTOs. Things can move fast if you’re a known quantity.
The current generation, and pretty much the one before it, expects all their stuff to be free, and also gets angry when the people who made the thing they love/rely on/generate income from do anything to either make money from it or make it valuable for to a buyer. Makes it hard for the creators when their cheerleaders last year then harass them because they're now in a position to capitalise somewhat on the energy and money they risked/invested.
indeed. You’ll risk to farpothsket it. 🙂
https://www.yiddishslangdictionary.com/word/198700/farpotshket
If you're in Seattle on a Sunday, stop by the Telecom Museum. They have walls of old electro mechanical phone switches.
So. Much. Clicky.
The place is a feast for the senses. You can hear the whole room thinking. Maintenance processes send little storms of activity around the room. There's a heartbeat from the wall of spinning clock generators. There's a distinctive smell.
I love that place.
I had an acquaintance involved in that project.
His story was that when it came to the end they were too uncertain about the new software to implement it so procrastinated with indefinitely extended "testing" until a small seismic event disrupted something in the old infrastructure and forced their hand.
Faced with trusting the new software or halting all Bay Area mass transit… well, that's the point the floppies were retired.
Sounds like the old phone system— people would listen to the multi-relays clicking around. But where’s the paper tape?
This gives me flashbacks to discovering in the mid 00s that our security system at the big San Francisco Borders still ran on OS/2.
Experts in 1990s programming languages? Like Python, Ruby, Java, or JavaScript?
Gonna go ahead and guess that there ain't no Unix kernel driver behind those MUNI floppy drives
To clarify: it's almost certainly win31 or something even more baleful and arcane
I dunno, there were plenty of x86 SVR4 options back then until NT stomped in
I'm sure a modern monstrosity to replace a fully functional control system would solve all of muni's problems
pretty sure there are at least 3 or 4 non functioning packages published in npm @jwz
"Floppy disks...obsolete for 30 years or more". Ah yes, I remember going to school in 1995 with my 8GB USB drive when we got rid of the floppies.
The article does say it uses 5.25 disks. Which died several years before 3.5 ones. 30 years seems about right.
I remember scouring the campus in 1993 looking for a machine that had a 5.25” drive so I could get data off some old disks.
You had a better school than me then, we were still using 5.25" drives in classrooms at that time.
That’s awesome. Must be extremely efficient code.
https://cdn.southampton.ac.uk/assets/imported/transforms/content-block/CB_RImg/FEC3F56CF1B3403490965D54E7D07F62/thumbnail_VINDTA%20-%20figure%201-web.jpg_SIA_JPG_background_image.jpg
The music of machinery is a wonderful thing. This is the current state of the art for measuring carbon and alkalinity in deep sea water samples. After a week or so of 12 hours shifts you don't need to monitor it at all - your brain has memorised the exact sequence of valve clicks and pump noises, and will alert you if the sequence is off by more than a few seconds.
I'm also convinced that it was invented by a frustrated Mad Scientist who couldn't afford a castle in Transylvania.
Dammit, I failed to use a simple embed feature.
In 2019, the London Underground retired a (mostly mechanical) signalling box from 1926. There's some pictures of it here.
(Probably retired anyway, it was planned to be replaced in 2013 and 2016, and I've yet to find a story saying that it actually has been retired)
Years ago I remember hearing that BART was run off of an old Mac Plus, and this went on for many years. Not sure if it was true but it sounded plausible at the time.
https://www.floppydisk.com/ stopped offering 5.25" floppies, I think. MUNI needs a truckload of 3.5" drives!
Look, they 3d-printed the Save icon!
This concept is not entirely dead. I've heard of people setting their programs up to make a tiny noise each time garbage collection kicks in (I think different noises for different generations) and then being able to quickly gain some information about how their program is behaving from the pattern. In particular programs with different phases apparently have different rhythms for each.
When I was a kid I had a sound chip in my computer that leaked noise from the rest of the components into my headphones. It made squeaky noises. They sounded different depending on what the computer was doing at the time.
Back in the 1900s I watched over a "ring" network of Apollo workstations. "Ring" as in, just like cheap Christmas lights, a single failure anywhere kills the whole network. Some Apollo models has a switching power supply that made an audible scratching noise when the power load shifted. Any time you heard a rhythmic scritch... scritch... scritch.. it meant the network card was forcing retries and it was time to start tracking down a broken net connection.
When we were mucking about with a zx81 in the early 80s, my mate would play the cassettes to find the right programs.
I remember playing with Peep back in the early 2000s, trying to tune it so the ambient sounds it produced from routine network and server events were both pleasant and useful. I see Net::Peep is even still in CPAN.
I like how it suggests floppies haven't been used in 30 years when computers still routinely came with a floppy drive until about 15 years ago.
They are talking specifically about 5 1/4" floppies, and the 30 years number is pretty accurate for that. Maybe 35.
I had a 5.25" drive from 1992, and around 2010 or so I installed it in a PC chassis and was slightly surprised to see that it still worked. I used it to make "dd" images of the handful of floppies I still had. A few years later, one of my associates needed to rescue data from an old industrial control system, but by then the drive wouldn't read past the first few sectors.
They don't even need to be using floppy drives or disks. There are reliable, cheap floppy emulators, such as those from GoTek, that will replace nearly any drive (there were only actually a handful of manufacturers and models), in any encoding format, in everything from MIDI keyboards to retro PCs to industrial equipment.