The reason the Tuesday Noon sirens haven't returned:

A new system was installed in 2005, which was then hacked in 2018, and fixing that exploit apparently requires replacing the entire communications infrastructure. SFDEM has been downplaying this and referring to this security firedrill as simply "upgrades".

tl;dr version --

I keep seeing articles asking what happened to the sirens, and then answering themselves that they "are antiquated" and "need repairs", which sounds like they're rusty or something. But what really happened was, in 2018 the siren network was hacked because it had no encryption.

The vendor claimed to have immediately rolled out a fix, and then in 2019, San Francisco shut the entire system down for what they believed at the time would be two years. For "upgrades". So, upgrading this system, which had been going off weekly since 1945 necessitated shutting the whole thing down immediately. Not, like, acquiring the budget and the equipment; testing it; staging it; and then shutting down the old system, no. Something was so badly wrong with it that they decided to completely scrap this piece of security infrastructure. Keeping it running at all was judged to be more dangerous than not having it at all.

That sounds like an active exploit in the wild, to me. That sounds like "the only way to prevent this attack is to replace the entire system". My guess is that the fix they came up with is to go with a new vendor entirely. Why is it so expensive? One guess would be that the new vendor uses a different communication system that requires replacing the radios and antennae on all of the horns.

But since SFDEM has been completely silent about what's involved in this "upgrade" (E.g., what is being replaced? Why? Who are the new vendors?) we have no way of knowing.

Here's a timeline that I was able to scrape together:

1942: Sirens installed. This page went online in 2015 and hasn't been updated since, but describes the 2005 system:

Each device is capable of playing up to seven different tones. The most common one is a "wail".

Voice messaging can either be: 1) pre-recorded on a chip installed in each device; 2) broadcast from the Department of Emergency Management through a recorded message or a live message; or 3) broadcast through the use of a mobile transmitter. [...]

Public safety mobile and portable radios can be remotely programmed to patch into the siren devices to allow the operator to make emergency announcements. [...]

Siren devices can be pre-programmed into a variety of groups for specific announcements. One such group is the Tsunami Warning group for sirens located in the inundation areas of the City.

I haven't found any technical details on how that original system worked, or what kinds of upgrades (if any) were made to the signalling network between 1942 and 2005. That probably means that the answer is "none". It's unlikely that the WWII-vintage system was hard-wired, so it's fair to assume that the old analog system was also trivially exploited by anyone who knew the frequencies and signaling protocol.

Oct 1995: Emergency Sirens Fail to Wail:

Nine of San Francisco's 49 emergency sirens, including one at the Ferry Building, failed to go off as scheduled during Tuesday morning's test, officials disclosed yesterday.

"These sirens were built in 1942, and many of them need repairs," said Frank Schober, coordinator of the Mayor's Office of Emergency Planning.

Schober hopes to replace all 49 of the 500-pound electromechanical devices with lighter electronic sirens. The cost would be about $125,000 a year with the job spread over five years.

Nov 2004: It's kaput for those old air-raid sirens:

The old air-raid sirens that have been sounding in San Francisco every Tuesday at noon since World War II are being replaced with a state-of-the-art emergency warning system that can be used to alert the public in the case of earthquakes, tsunamis, bioterror attacks or other disasters, Mayor Gavin Newsom said Tuesday. [...]

San Francisco's old system has fallen into disrepair over the years, with only about a dozen of the original 50 sirens in working order. Officials are replacing the old mechanical devices with a digital system that will be both siren and public address system. They will be located in 65 locations in the city.

The federal government provided a $2.1 million Homeland Security grant to pay for the upgraded system. The new devices are expected to be fully up and running in January.

By 2005, the siren system was being described as "new", so 2004 or 2005 is when the WWII-vintage analog system was replaced with a digital radio network. Sorry, I meant to say a "state of the art" digital radio network. So how did that work out? Let's check in...

Nov 2005: Hearings urged on faulty siren system:

Mayor Gavin Newsom and Board of Supervisors President Aaron Peskin called separately Tuesday for public hearings to educate residents about flaws found with the city's new emergency siren system.

City officials say the sirens, an early warning system for disasters, aren't loud enough and can be heard in only 50 to 60 percent of the city rather than the 90 percent called for in the contract with Acoustic Technology Inc. The city attorney sent a letter to the contractor Friday claiming breach of contract and demanding that the problems be resolved by the end of the year.

After that, I don't see any press about the sirens for a few years, until a couple incidents where they mysteriously went off at unplanned times. And then... womp womp...

Aug 2012: Emergency siren accidentally activated:

San Francisco emergency officials activated a warning siren Sunday afternoon, triggering some confusion among residents. The siren, which sounded around 3:45 p.m., was activated accidentally, and there was no emergency, according to the San Francisco Department of Emergency Management.

Nov 2014: Officials investigate after outdoor sirens triggered at odd hours:

Outdoor emergency sirens in San Francisco were accidentally triggered late Saturday and early Sunday morning, according to the San Francisco Department of Emergency Management. The sirens were temporarily out of service on Sunday afternoon as city crews conducted testing to determine the cause.

Alarms went off around 11 p.m. Saturday in the Bernal Heights, Noe Valley and Hunters Point neighborhoods, the Bayview District, City Hall, and other areas, but there is currently no known emergency that would have triggered the alarms, department spokesman Francis Zamora said.

Alarms around the city went off again around 5 a.m., he said.

Apr 2018: SF's emergency sirens had a security bug -- it's fixed now:

San Francisco officials have been quietly scrambling since early February to patch a security vulnerability in the city's outdoor alert system that, if left unaddressed, could have allowed hackers to seize control of the city's network of 114 emergency sirens.

On Thursday, the Department of Technology announced that the problem had been fixed. [...] The technology department declined to share the specifics of the vulnerability, other than to say that it had to do with how electronic signals were being encrypted as they were being relayed across the alert system.

"It's fixed now", huh?

Apr 2018: This Radio Hacker Could Hijack Citywide Emergency Sirens to Play Any Sound:

Now, after two-and-a-half years of patiently recording and reverse-engineering those weekly radio communications, Seeber has indeed found that he or anyone with a laptop and a $35 radio could not only trigger those sirens, as unknown hackers did in Dallas last year. They could also make them play any audio they choose: false warnings of incoming tsunamis or missile strikes, dangerous or mass-panic-inducing instructions, 3 am serenades of death metal or Tony Bennett. And he has found the same hackable siren systems not only in San Francisco but in two other cities. [...]

When WIRED reached out to ATI Systems, the company responded that "the vulnerability is largely theoretical and has not yet been seen in the field." It also argued that Bastille had broken the law with its research by violating FCC regulations against intercepting and even merely divulging the existence of government radio signals without authorization. But in a statement it sent to Bastille after the researchers warned ATI about its security flaws, ATI wrote that Bastille's findings are "likely true" and that it's testing a software update it plans to roll out soon.

Apr 2018: SirenJack White Paper (PDF), and CVE-2018-8862:

No no no -- thank you!
The SirenJack vulnerability is distinct from the replay attack that struck the Federal Signal-manufactured Dallas tornado warning system on April 7th, 2017. The older Dallas system used Dual Tone Multi Frequency (DTMF) tones to activate the system over an analog radio link. It is trivial to record the audio of those tones (e.g. on a laptop or tape recorder), and then replay them on the same frequency while transmitting. The activation 'code' usually is fixed, and therefore can be accepted multiple times. [...]

The proprietary digital radio protocol used by ATI to control the San Francisco OPWS was found to have no encryption. As messages were sent in the clear, the patterns of changing elements became easy to interpret. These patterns could be extrapolated to craft malicious messages that conform to the protocol's format and therefore look legitimate, such as activation commands to trigger false alarms. In a deployment where regular testing takes place, knowledge gained by passive observation of test activation commands can be used to trigger the siren system in that deployment at will. [...]

The protocol does not draw on any truly secure practices to prevent analysis of the relevant fields, and thwart potential interference with the system. It is therefore vulnerable due to its reliance on security through obscurity. [...]

A Proof-of-Concept was demonstrated on an ATI siren node with a single horn at a low volume at an isolated location. A modulator and transmitter were created using GNU Radio and a USRP B200mini SDR. Knowledge of the protocol gained by passive observation of two active deployments (San Francisco, CA and Sedgwick County, KS) provided sufficient information to enable the crafting of legitimate activation commands for this node, the configuration for which was unknown. [...]

ATI has stated they have worked on increasing the level of security of their radio protocol, and this fix has now been reported to be rolled out across San Francisco's OPWS. During the weeks leading up the public disclosure, the OPWS frequency in San Francisco was active with an increasing number of packets that displayed higher entropy (appeared random), and activation commands in San Francisco have no longer been seen in the clear since public disclosure. No cryptanalysis has been performed to determine the efficacy of the fix. Details of remediation steps have not been made available publicly.

Oh, so the fix has been rolled out in San Francisco, huh? Let's see how that's going....

Dec 2019: Upgrades will silence sirens for two years:

The last scheduled siren test is planned for Dec. 10 before a hardware and software overhaul expected to cost up to $2.5 million takes them offline.

The upgrades -- the first since 2005 -- are intended to make the sirens more reliable and secure from outside tampering, the city's Department of Emergency Management said in a statement.

The two-year outage is necessary so that the city can test new specialized equipment before upgrading all 119 sirens.

Securing the sirens has been an issue for the city recently. Last year, the Department of Technology, which maintains the sirens, disclosed that it spent months trying to patch a security vulnerability that, if left unaddressed, could have allowed hackers to seize control of the sirens.

Dec 2021: Siren system stays silent after original upgrade deadline:

The Outdoor Public Warning System, which dates back to World War II, was silenced in December 2019 due to security concerns.

Upgrades were originally expected to take two years, but the city isn't any closer to finishing the project now. Zamora said it's because the COVID-19 pandemic response altered spending priorities.

Jan 2022: Tsunami advisory wouldn't have triggered SF's emergency sirens, but why do they remain silent?

"Right now the sirens are offline and they are offline due to the fact that there were some significant security issues related to the technology," said Mary Ellen Carroll, Director of San Francisco's Department of Emergency Management. "So, we had to take them offline about two years ago."

The city's Department of Emergency Management says this tsunami advisory would not have triggered an outdoor alert even if it were up and working because of the low risk to the area. Director Carroll says the department relied on first responders securing the beach and existing wireless technology to push alerts to the mobile devices of those who have opted into AlertSF and if necessary even to those who have not. "We would not have sounded the sirens for this alert, and we did use AlertSF, out texting alerts to let people know what was going on," said Carroll.

During the 2018-2022 period, we also got a lot of journalistic malpractice like this article on Curbed, which is what happens when so-called journalists just publish press releases without asking any real questions:

Why is it being repaired? It's antiquated. San Francisco will invest between $2,000,000 to $2,500,000 in upgrades to the bring the OPWS up to snuff. Upgrades will include new hardware that will improve the reliability system.

But we can always rely on the @SFSiren twitter account to tell the truth:

Nov 9, 2021: It's my #Twitterversary! I have been on Twitter for 12 years, since 10 Nov 2009

Nov 9, 2020: It's my #Twitterversary! I have been on Twitter for 11 years, since 10 Nov 2009

Mar 16, 2020: @SFSiren Retweeted @mjg59: San Francisco, noon tomorrow: the entire population leaning out of their windows and making the emergency siren noise

This is a test. This is a test of the outdoor warning system. This is only a test.

This is a test. This is a test of the outdoor warning system. This is only a test.

This is a test. This is a test of the outdoor warning system. This is only a test.

This is a test. This is a test of the outdoor warning system. This is only a test.

Here are some questions that I still have. If you are a journalist with enough clout that SFDEM will take your calls, how about you try and get these answers?

  1. What actually happened in 2012 and 2014 when the sirens were going off unscheduled? You're probably going to need to FOIA the incident reports to get a straight answer about this.

  2. What happened in 2018 when "officials" were "scrambling" to fix the security problem? What was their understanding of the exploit? What specific actions were taken?

  3. Was the exploit considered to have been mitigated? If not, why was the system left operational between Apr 2018 and Dec 2019?

  4. Why was the system completely shut down in Dec 2019? Was it because of the exploit discovered in 2018? Please note, "we needed to test new specialized equipment" does not answer the question of why the existing system was taken completely offline.

  5. What are the details of the plan for bringing the system back online? What hardware will be replaced? What vendors and what products are involved? What security analysis has been performed on the new products?

But those are just the questions that I would be asking, if I was a journalist. What do I know.

Previously, previously, previously, previously, previously, previously, previously, previously.

Tags: , , , , , ,

18 Responses:

  1. Will Watson says:

    This seems like something journalists should have followed up with.

    As it is, I doubt you'll get anything without making the Public Records request yourself.

    • Elusis says:

      Yeah, this sounds like a perfect excuse to file a stack of FOIA queries.

      (I currently have a pending FOIA request to the Department of Education just to get my own fucking student loan payment history, since no one seems to have any records of even who my lender was before 2011, never mind whether I actually made qualifying payments for PSLF, which in my mind is something of an... issue.)

    • Rusty Hodge says:

      6. How much money has been spent to date of the original 2.5 Million anticipated cost?

  2. Jay says:

    Sounds like exactly the type of thing Bay Curious would dive into

  3. Rusty Hodge says:

    The system in the 90s and early 2000s was really simple. It was just an analog VHF FM channel that sent a 3 or 4 digit code to turn the sirens on, and another to turn them off. I used to keep it programmed in my scanner and would hear it for the tests. Talk about an easy to hijack system.

    This old, old frequency list for SF (circa '94) was adapted from some postings on Usenet.

    Air Raid siren control primary Input 155.745 Output 158.760 PL 173.8
    Air Raid siren control backup Simplex 155.385 PL 156.7

    I'm not sure what the post-2005 system used, other than it was an RF-based system and possible still VHF based on the antennas on the siren sites.

    Now that the city has switched to the p25 digital system (with encryption) they should be able to just tie that in with the that. But I'm sure Motorola (or whoever is the vendor for the p25 system) and ATI (or whoever the new siren contractor is) aren't talking.

    But it's also possible that the delay in getting the sirens back running was caused by the delay in getting cut over to the new p25 digital system which just happened in the last month.

    • Sireeen says:

      If the sirens were activated over RF it's guaranteed they used DTMF.

      • Adolf Osborne says:

        Not necessarily. There's lots of stuff in RF-world, including AES256-encrypted traffic on P25 trunking systems.

        But even today, the simplest siren activation styles aren't DTMF. They often use two-tone or long-tone signalling, like an old voice pager would have done.

        And by two-tone, I don't mean Dual-Tone, Multi-Frequency. I mean 1 second of a particular audio frequency (from a set of pre-defined frequencies), followed by 3 seconds of another audio frequency (from the same set), and the siren activates.

        The tones are pre-defined, because they were originally sold and configured as physical, resonating plug-in modules for the completely analog receivers that used them.

        Long-tone is even simpler: Play one predetermined tone for 8 seconds, and the siren activates.

        These methods predate security, and probably predate DTMF, and certainly predate the convenience of modern Flash EEPROM, but they're still used a lot for outdoor warning sirens in my neck of the woods (which is rather far from the SF Bay area).

  4. Rusty Hodge says:

    A couple more links:

    Black Hat 2018 presentation of an exploit of the sirens:

    ATI getting all defensive about that presentation

    tl;dr- in 2005 they switched to a 202A-style FSK but over the (presumably same) voice VHF frequencies again. There wasn't any encryption on that FSK data stream. It was security by obscurity. The FSK data won't work on top of the P25 enctypted voice channels.

  5. Patrick says:

    Wait, what? Your emergency alert sirens are some sort of speakers?
    How is that supposed to be powered in a, well, emergency?

    Here the sirens are driven by stored compressed air. So the only electrical backup needed is for the control electronics. They just need to open/close a valve to make really loud sounds.

    • Philip Guenther says:

      Effective uses of emergency sirens occur before anything has failed. They exist to warn people to prepare and take shelter, not to tell them that shit has already hit the fan. I lived in Minnesota for a decade and heard the tornado sirens both in regular drills and when St Peter was hit by a tornado and they came before any power outage.

      (Running them past the disaster is more difficult and often screwed up, yes, but depending on the nature of the disasters of concern, not typically a problem. It merely really annoying when the college scheduled summer power work and discovered they hadn't been testing their telecom power backup system and the diesel generator filled the basement room directly under my office with smoke from burnt oil.)

      • Patrick says:

        This system was originally built to be used for air raid warnings in wartime, so I suppose working without power was a higher priority.

      • Adolf Osborne says:

        Adding to this: All of the siren systems I've been involved with have a fairly substantial battery backup system. The most common build-out around here uses four M24MF marine deep-cycle batteries, per siren, with heaters.

        We use them for tornado warnings. It is important that the sirens still work after a tornado that is 20 miles away knocks out power to the city that it is headed toward, but before it reaches that city.

        The activation duties are normally handled by law-enforcement agency dispatchers, which usually have layers of backup battery systems on their communications systems and a standby genset. There are normally 2 or 3 other ways to activate the sirens that are physically well-removed from the primary point of activation.

        And all of this is tested periodically, including load-testing the batteries that are scattered around on siren poles all over the place.

  6. anon guy says:

    Weekly tests? Ours are reliable enough that annual testing is good enough.

  7. I'm reminded of an April Fools prank I performed while a student at Swarthmore College some two decades ago now, inspired by your, JWZ, assertions (via IRC #dnalounge) about Postscript's Turing-completeness and the complete lack of security around redefining functions in them. (Documentation of that and its fallout for me.)

    Somehow it's a lot less funny when it's "civic safety instrument" rather than "corporate or college campus printers", though.

  • Previously