A new system was installed in 2005, which was then hacked in 2018, and fixing that exploit apparently requires replacing the entire communications infrastructure. SFDEM has been downplaying this and referring to this security firedrill as simply "upgrades".
tl;dr version --
I keep seeing articles asking what happened to the sirens, and then answering themselves that they "are antiquated" and "need repairs", which sounds like they're rusty or something. But what really happened was, in 2018 the siren network was hacked because it had no encryption.
The vendor claimed to have immediately rolled out a fix, and then in 2019, San Francisco shut the entire system down for what they believed at the time would be two years. For "upgrades". So, upgrading this system, which had been going off weekly since 1945 necessitated shutting the whole thing down immediately. Not, like, acquiring the budget and the equipment; testing it; staging it; and then shutting down the old system, no. Something was so badly wrong with it that they decided to completely scrap this piece of security infrastructure. Keeping it running at all was judged to be more dangerous than not having it at all.
That sounds like an active exploit in the wild, to me. That sounds like "the only way to prevent this attack is to replace the entire system". My guess is that the fix they came up with is to go with a new vendor entirely. Why is it so expensive? One guess would be that the new vendor uses a different communication system that requires replacing the radios and antennae on all of the horns.
But since SFDEM has been completely silent about what's involved in this "upgrade" (E.g., what is being replaced? Why? Who are the new vendors?) we have no way of knowing.
Here's a timeline that I was able to scrape together:
1942: Sirens installed. This page went online in 2015 and hasn't been updated since, but describes the 2005 system:
Each device is capable of playing up to seven different tones. The most common one is a "wail".
Voice messaging can either be: 1) pre-recorded on a chip installed in each device; 2) broadcast from the Department of Emergency Management through a recorded message or a live message; or 3) broadcast through the use of a mobile transmitter. [...]
Public safety mobile and portable radios can be remotely programmed to patch into the siren devices to allow the operator to make emergency announcements. [...]
Siren devices can be pre-programmed into a variety of groups for specific announcements. One such group is the Tsunami Warning group for sirens located in the inundation areas of the City.
I haven't found any technical details on how that original system worked, or what kinds of upgrades (if any) were made to the signalling network between 1942 and 2005. That probably means that the answer is "none". It's unlikely that the WWII-vintage system was hard-wired, so it's fair to assume that the old analog system was also trivially exploited by anyone who knew the frequencies and signaling protocol.
Oct 1995: Emergency Sirens Fail to Wail:
Nine of San Francisco's 49 emergency sirens, including one at the Ferry Building, failed to go off as scheduled during Tuesday morning's test, officials disclosed yesterday.
"These sirens were built in 1942, and many of them need repairs," said Frank Schober, coordinator of the Mayor's Office of Emergency Planning.
Schober hopes to replace all 49 of the 500-pound electromechanical devices with lighter electronic sirens. The cost would be about $125,000 a year with the job spread over five years.
Nov 2004: It's kaput for those old air-raid sirens:
The old air-raid sirens that have been sounding in San Francisco every Tuesday at noon since World War II are being replaced with a state-of-the-art emergency warning system that can be used to alert the public in the case of earthquakes, tsunamis, bioterror attacks or other disasters, Mayor Gavin Newsom said Tuesday. [...]
San Francisco's old system has fallen into disrepair over the years, with only about a dozen of the original 50 sirens in working order. Officials are replacing the old mechanical devices with a digital system that will be both siren and public address system. They will be located in 65 locations in the city.
The federal government provided a $2.1 million Homeland Security grant to pay for the upgraded system. The new devices are expected to be fully up and running in January.
By 2005, the siren system was being described as "new", so 2004 or 2005 is when the WWII-vintage analog system was replaced with a digital radio network. Sorry, I meant to say a "state of the art" digital radio network. So how did that work out? Let's check in...
Nov 2005: Hearings urged on faulty siren system:
Mayor Gavin Newsom and Board of Supervisors President Aaron Peskin called separately Tuesday for public hearings to educate residents about flaws found with the city's new emergency siren system.
City officials say the sirens, an early warning system for disasters, aren't loud enough and can be heard in only 50 to 60 percent of the city rather than the 90 percent called for in the contract with Acoustic Technology Inc. The city attorney sent a letter to the contractor Friday claiming breach of contract and demanding that the problems be resolved by the end of the year.
After that, I don't see any press about the sirens for a few years, until a couple incidents where they mysteriously went off at unplanned times. And then... womp womp...
Aug 2012: Emergency siren accidentally activated:
San Francisco emergency officials activated a warning siren Sunday afternoon, triggering some confusion among residents. The siren, which sounded around 3:45 p.m., was activated accidentally, and there was no emergency, according to the San Francisco Department of Emergency Management.
Nov 2014: Officials investigate after outdoor sirens triggered at odd hours:
Outdoor emergency sirens in San Francisco were accidentally triggered late Saturday and early Sunday morning, according to the San Francisco Department of Emergency Management. The sirens were temporarily out of service on Sunday afternoon as city crews conducted testing to determine the cause.
Alarms went off around 11 p.m. Saturday in the Bernal Heights, Noe Valley and Hunters Point neighborhoods, the Bayview District, City Hall, and other areas, but there is currently no known emergency that would have triggered the alarms, department spokesman Francis Zamora said.
Alarms around the city went off again around 5 a.m., he said.
Apr 2018: SF's emergency sirens had a security bug -- it's fixed now:
San Francisco officials have been quietly scrambling since early February to patch a security vulnerability in the city's outdoor alert system that, if left unaddressed, could have allowed hackers to seize control of the city's network of 114 emergency sirens.
On Thursday, the Department of Technology announced that the problem had been fixed. [...] The technology department declined to share the specifics of the vulnerability, other than to say that it had to do with how electronic signals were being encrypted as they were being relayed across the alert system.
"It's fixed now", huh?
Apr 2018: This Radio Hacker Could Hijack Citywide Emergency Sirens to Play Any Sound:
Now, after two-and-a-half years of patiently recording and reverse-engineering those weekly radio communications, Seeber has indeed found that he or anyone with a laptop and a $35 radio could not only trigger those sirens, as unknown hackers did in Dallas last year. They could also make them play any audio they choose: false warnings of incoming tsunamis or missile strikes, dangerous or mass-panic-inducing instructions, 3 am serenades of death metal or Tony Bennett. And he has found the same hackable siren systems not only in San Francisco but in two other cities. [...]
When WIRED reached out to ATI Systems, the company responded that "the vulnerability is largely theoretical and has not yet been seen in the field." It also argued that Bastille had broken the law with its research by violating FCC regulations against intercepting and even merely divulging the existence of government radio signals without authorization. But in a statement it sent to Bastille after the researchers warned ATI about its security flaws, ATI wrote that Bastille's findings are "likely true" and that it's testing a software update it plans to roll out soon.
Apr 2018: SirenJack White Paper (PDF), and CVE-2018-8862:
The SirenJack vulnerability is distinct from the replay attack that struck the Federal Signal-manufactured Dallas tornado warning system on April 7th, 2017. The older Dallas system used Dual Tone Multi Frequency (DTMF) tones to activate the system over an analog radio link. It is trivial to record the audio of those tones (e.g. on a laptop or tape recorder), and then replay them on the same frequency while transmitting. The activation 'code' usually is fixed, and therefore can be accepted multiple times. [...]The proprietary digital radio protocol used by ATI to control the San Francisco OPWS was found to have no encryption. As messages were sent in the clear, the patterns of changing elements became easy to interpret. These patterns could be extrapolated to craft malicious messages that conform to the protocol's format and therefore look legitimate, such as activation commands to trigger false alarms. In a deployment where regular testing takes place, knowledge gained by passive observation of test activation commands can be used to trigger the siren system in that deployment at will. [...]
The protocol does not draw on any truly secure practices to prevent analysis of the relevant fields, and thwart potential interference with the system. It is therefore vulnerable due to its reliance on security through obscurity. [...]
A Proof-of-Concept was demonstrated on an ATI siren node with a single horn at a low volume at an isolated location. A modulator and transmitter were created using GNU Radio and a USRP B200mini SDR. Knowledge of the protocol gained by passive observation of two active deployments (San Francisco, CA and Sedgwick County, KS) provided sufficient information to enable the crafting of legitimate activation commands for this node, the configuration for which was unknown. [...]
ATI has stated they have worked on increasing the level of security of their radio protocol, and this fix has now been reported to be rolled out across San Francisco's OPWS. During the weeks leading up the public disclosure, the OPWS frequency in San Francisco was active with an increasing number of packets that displayed higher entropy (appeared random), and activation commands in San Francisco have no longer been seen in the clear since public disclosure. No cryptanalysis has been performed to determine the efficacy of the fix. Details of remediation steps have not been made available publicly.
Oh, so the fix has been rolled out in San Francisco, huh? Let's see how that's going....
Dec 2019: Upgrades will silence sirens for two years:
The last scheduled siren test is planned for Dec. 10 before a hardware and software overhaul expected to cost up to $2.5 million takes them offline.
The upgrades -- the first since 2005 -- are intended to make the sirens more reliable and secure from outside tampering, the city's Department of Emergency Management said in a statement.
The two-year outage is necessary so that the city can test new specialized equipment before upgrading all 119 sirens.
Securing the sirens has been an issue for the city recently. Last year, the Department of Technology, which maintains the sirens, disclosed that it spent months trying to patch a security vulnerability that, if left unaddressed, could have allowed hackers to seize control of the sirens.
Dec 2021: Siren system stays silent after original upgrade deadline:
The Outdoor Public Warning System, which dates back to World War II, was silenced in December 2019 due to security concerns.
Upgrades were originally expected to take two years, but the city isn't any closer to finishing the project now. Zamora said it's because the COVID-19 pandemic response altered spending priorities.
Jan 2022: Tsunami advisory wouldn't have triggered SF's emergency sirens, but why do they remain silent?
"Right now the sirens are offline and they are offline due to the fact that there were some significant security issues related to the technology," said Mary Ellen Carroll, Director of San Francisco's Department of Emergency Management. "So, we had to take them offline about two years ago."
The city's Department of Emergency Management says this tsunami advisory would not have triggered an outdoor alert even if it were up and working because of the low risk to the area. Director Carroll says the department relied on first responders securing the beach and existing wireless technology to push alerts to the mobile devices of those who have opted into AlertSF and if necessary even to those who have not. "We would not have sounded the sirens for this alert, and we did use AlertSF, out texting alerts to let people know what was going on," said Carroll.
During the 2018-2022 period, we also got a lot of journalistic malpractice like this article on Curbed, which is what happens when so-called journalists just publish press releases without asking any real questions:
Why is it being repaired? It's antiquated. San Francisco will invest between $2,000,000 to $2,500,000 in upgrades to the bring the OPWS up to snuff. Upgrades will include new hardware that will improve the reliability system.
But we can always rely on the @SFSiren twitter account to tell the truth:
Nov 9, 2021: It's my #Twitterversary! I have been on Twitter for 12 years, since 10 Nov 2009
Nov 9, 2020: It's my #Twitterversary! I have been on Twitter for 11 years, since 10 Nov 2009
Mar 16, 2020: @SFSiren Retweeted @mjg59: San Francisco, noon tomorrow: the entire population leaning out of their windows and making the emergency siren noise
Dec 10, 2019: WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
This is a test. This is a test of the outdoor warning system. This is only a test.
Dec 3, 2019: WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
This is a test. This is a test of the outdoor warning system. This is only a test.
Nov 26, 2019: WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
This is a test. This is a test of the outdoor warning system. This is only a test.
Nov 19, 2019: WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
This is a test. This is a test of the outdoor warning system. This is only a test.
Here are some questions that I still have. If you are a journalist with enough clout that SFDEM will take your calls, how about you try and get these answers?
What actually happened in 2012 and 2014 when the sirens were going off unscheduled? You're probably going to need to FOIA the incident reports to get a straight answer about this.
What happened in 2018 when "officials" were "scrambling" to fix the security problem? What was their understanding of the exploit? What specific actions were taken?
Was the exploit considered to have been mitigated? If not, why was the system left operational between Apr 2018 and Dec 2019?
Why was the system completely shut down in Dec 2019? Was it because of the exploit discovered in 2018? Please note, "we needed to test new specialized equipment" does not answer the question of why the existing system was taken completely offline.
What are the details of the plan for bringing the system back online? What hardware will be replaced? What vendors and what products are involved? What security analysis has been performed on the new products?
But those are just the questions that I would be asking, if I was a journalist. What do I know.
Previously, previously, previously, previously, previously, previously, previously, previously.
Buzzer is a Russian military station currently haunted by radio pirates:
mhoye:
Violet Blue:
On the twitter hack:
