SF's Market Street Subway Is Running on Reagan-Era Technology:
"Our train control system in the Market Street subway is loaded off of five-and-a-quarter inch floppy drives," SFMTA director Jeffrey Tumlin told KQED's Priya David Clemens this week. "We have to employ programmers who are experts in the programming languages of the nineties in order to keep running our current system. So we have a technical debt that stretches back many decades."
Ah yes, the withered programmers from the Nineteen-Hundreds. I'm glad they included these historical fun facts about floppy disks!
For those who've never had to use one -- and keep it far away from any magnets -- floppy disks are still used as the "save" icon on computers worldwide, but they've been obsolete for 30 years or more. Originally a full eight inches across when introduced in the 1970s, floppies later shrunk to five-and-one-quarter inches, and, eventually, a mere three. Early generations were indeed bendable, although the final generation was made of a harder, thicker plastic, with no flop at all. Eventually, CD-ROM technology superseded them before becoming nearly extinct, too.
Here's an article from 2015 about some upgrades, but this sounds like an even older, pre-floppy system:
Muni train control system gets biggest upgrade since the '90s:
In a dank walkway tucked away inside the Van Ness Avenue Muni station is the mainframe that controls Muni trains underground.
Relay racks extend down a cramped hallway nearly 15 feet deep, where dozens of 4-inch copper filaments clatter up and down like teeth. The speedy "click, clack" sound signifies a smooth running Muni light rail automatic control system [...] The clapping beat signifies the proper alignment of the railway's 83 axle (controls) and numerous other track switches. [...]
When this happens, or a track switch is damaged, a green relay cable running the length of the underground track is tripped. This signals the maintenance crew, which then checks each ticking copper relay for those with an off beat, or ones that have stopped altogether. [...]
"Last year, we had water up to here" Haley said, gesturing with his hand to his waist. The relay system is under a city drain, he said. A yellow tarp is strung at the roof of the relay room, while sandbags, a bucket and more tarps are tucked away in the corners. [...]
"These things are hard to replace," Kelly said. "You can't just go to Relays R Us." [...]
The crew that maintains and inspects the relays every night said it is hopeful for the new system -- but it also respects the old one. "I trust it to run another 20 years," said Hoa Huynh, a 20-year maintenance crew member. But, he said, "It's slow. Very slow."
I love the idea that part of the diagnostic process is to stand in this room and listen to it -- "that's not how the tunnel song is supposed to go." An interface that draws big red boxes using a supercomputer is not necessarily easier or more expressive than one that communicates ambiently through the system's inherent musicality.
Previously, previously, previously, previously, previously, previously, previously.
I know this is unfair and unfounded of me, as it is surely possible to build incredibly reliable systems in 2023, but I feel more comfortable with the idea of important infrastructure running software from the nineties than I would if it ran something made in the 2020s.
Older is sometimes better, probably because it forces the system designer to minimize complexity. The control system for the Panama canal might be a good example. It was built in ~1913, and it was completely mechanical. The user interface was a scale model of the canal, with blocks of blue marble that were raised and lowered to show the water level in each section of the lock. It used complex mechanical interlocks to prevent anyone from operating the locks incorrectly; you can raise or lower a ship through the locks, opening the gates one at a time, but you cannot open all the gates or valves at once. Apparently it’s never failed; the canal has been closed for other reasons, but never because the control system wasn’t working.
But it’s ultimately still quite simple, because all it allows you to do is turn first this handle, then wait until the water rises, then turn the next, etc. It’s a very simple state machine, and it never has to change. Even though they added new sets of locks to the canal, each flight of locks has it’s own control system so the existing ones didn’t have to change. Compare that with a train system where each block of track might have a train in it at any time, where there are multiple routes through the system, where trains get faster every decade, routes are added and removed at a politician’s whim, etc.
Absolute Block, which appears to be what you're describing for trains, isn't really any different than the canal situation. Neither routes nor train speed need to matter to Absolute Block. Like the canal, the railways did have mechanical interlocking for Absolute Block, that's why a classic mechanical signal box cabin is up a flight of steps - underneath it is a tremendous amount of complicated gears and levers which implement the interlocking, in a space big enough for maintenance engineers to actually work on it.
Eventually the mechanical interlocking was replaced by transistors, and eventually the transistors were replaced by software but it's the same outcome. A signaller who screwed up and would once have found that all their strength won't move the lever (because doing so would create a dangerous situation) in the box, now finds that clicking doesn't do anything because if it did that would create a dangerous situation. The modern UI might do better at pointing out what the problem is though, which can be helpful.
For example a real incident took place in which a mech signaller arrived after "his" box had been subject to a lot of maintenance work, the first train which arrived at his outer signal that night got Danger (ie a red signal), the driver called the signaller after sitting at this signal for a while, signaller tried to release it and it wouldn't budge. He assumed the maintenance engineers fucked up, but procedure is procedure so he tells the driver the usual, "Pass this signal at caution, obey all further signals". Next signal, same problem, same speech, still cursing the engineers. Signaller gets another call from the driver. Same problem? Nope. The driver is now face-to-face with an oncoming train. At caution speed there was enough time to brake and so it's scary but no damage done.
The Danger signals were real, the signal wouldn't release because release is conditional on Points-Locked-Straight for the points (maybe US term is "switch?") which choose whether your train crosses onto the opposite track, and those points had failed in indeterminate state. If the Signaller had checked why the release wasn't possible he'd have discovered one of his dozens of dials and gauges said "Indeterminate" instead of "Points Locked", but he was blaming maintenance engineers instead. A modern computerised system could have said "No, there's a points failure" and shown where, ensuring the signaller's focus was on the actual problem earlier.
As a bonus that old hardware is probably a lot less vulnerable to malicious hacking. The sensors and actuators are probably directly driven by an air gapped control computer. Though air gapping doesn’t always prevent hacker penetration (see Stuxnet), the ancient controller and direct drive would make design of the attack an exercise in bespoke historic reverse engineering.
I recall network admins back in the day would sometimes pipe netcat to /dev/audio 🙂
Sounds like the old phone system— people would listen to the multi-relays clicking around. But where’s the paper tape?
"Floppy disks...obsolete for 30 years or more". Ah yes, I remember going to school in 1995 with my 8GB USB drive when we got rid of the floppies.
The article does say it uses 5.25 disks. Which died several years before 3.5 ones. 30 years seems about right.
I remember scouring the campus in 1993 looking for a machine that had a 5.25” drive so I could get data off some old disks.
You had a better school than me then, we were still using 5.25" drives in classrooms at that time.
That’s awesome. Must be extremely efficient code.
https://cdn.southampton.ac.uk/assets/imported/transforms/content-block/CB_RImg/FEC3F56CF1B3403490965D54E7D07F62/thumbnail_VINDTA%20-%20figure%201-web.jpg_SIA_JPG_background_image.jpg
The music of machinery is a wonderful thing. This is the current state of the art for measuring carbon and alkalinity in deep sea water samples. After a week or so of 12 hours shifts you don't need to monitor it at all - your brain has memorised the exact sequence of valve clicks and pump noises, and will alert you if the sequence is off by more than a few seconds.
I'm also convinced that it was invented by a frustrated Mad Scientist who couldn't afford a castle in Transylvania.
Dammit, I failed to use a simple embed feature.
In 2019, the London Underground retired a (mostly mechanical) signalling box from 1926. There's some pictures of it here.
(Probably retired anyway, it was planned to be replaced in 2013 and 2016, and I've yet to find a story saying that it actually has been retired)
https://www.floppydisk.com/ stopped offering 5.25" floppies, I think. MUNI needs a truckload of 3.5" drives!
Look, they 3d-printed the Save icon!
This concept is not entirely dead. I've heard of people setting their programs up to make a tiny noise each time garbage collection kicks in (I think different noises for different generations) and then being able to quickly gain some information about how their program is behaving from the pattern. In particular programs with different phases apparently have different rhythms for each.
When I was a kid I had a sound chip in my computer that leaked noise from the rest of the components into my headphones. It made squeaky noises. They sounded different depending on what the computer was doing at the time.
Back in the 1900s I watched over a "ring" network of Apollo workstations. "Ring" as in, just like cheap Christmas lights, a single failure anywhere kills the whole network. Some Apollo models has a switching power supply that made an audible scratching noise when the power load shifted. Any time you heard a rhythmic scritch... scritch... scritch.. it meant the network card was forcing retries and it was time to start tracking down a broken net connection.
When we were mucking about with a zx81 in the early 80s, my mate would play the cassettes to find the right programs.
I remember playing with Peep back in the early 2000s, trying to tune it so the ambient sounds it produced from routine network and server events were both pleasant and useful. I see Net::Peep is even still in CPAN.
I like how it suggests floppies haven't been used in 30 years when computers still routinely came with a floppy drive until about 15 years ago.
They are talking specifically about 5 1/4" floppies, and the 30 years number is pretty accurate for that. Maybe 35.
I had a 5.25" drive from 1992, and around 2010 or so I installed it in a PC chassis and was slightly surprised to see that it still worked. I used it to make "dd" images of the handful of floppies I still had. A few years later, one of my associates needed to rescue data from an old industrial control system, but by then the drive wouldn't read past the first few sectors.
They don't even need to be using floppy drives or disks. There are reliable, cheap floppy emulators, such as those from GoTek, that will replace nearly any drive (there were only actually a handful of manufacturers and models), in any encoding format, in everything from MIDI keyboards to retro PCs to industrial equipment.