Y2K bug, meet 60K Feet Bug.

Spy Plane Fries Air Traffic Control Computers, Shuts Down LAX

A relic from the Cold War appears to have triggered a software glitch at a major air traffic control center in California Wednesday that led to delays and cancellations of hundreds of flights across the country.

On Wednesday, a U-2 spy plane passed through the airspace monitored by the L.A. Air Route Traffic Control Center in Palmdale, Calif. The L.A. Center handles landings and departures at the region's major airports, including LAX, San Diego and Las Vegas.

The computers at the L.A. Center are programmed to keep commercial airliners and other aircraft from colliding with each other. The U-2 was flying at 60,000 feet, but the computers were attempting to keep it from colliding with planes that were actually miles beneath it.

Though the exact technical causes are not known, the spy plane's altitude and route apparently overloaded a computer system called ERAM, which generates display data for air-traffic controllers. Back-up computer systems also failed.

As a result, the FAA had to stop accepting flights into airspace managed by the L.A. Center, issuing a nationwide ground stop that lasted for about an hour and affected thousands of passengers.

Previously, previously, previously, previously, previously.

Tags: , , ,

15 Responses:

  1. Richard Jones says:

    Please don't say they're storing height in 16 bit integers.

  2. MattF says:

    16-bit unsigned integers, I"d guess. Some smart person figured that you wouldn't see a lot of negative altitudes.

    • Pavel Lishin says:

      I hope they didn't sell that software to Atyrau Airport.

      It is the lowest commercial airport in the world at 22 m (72 ft) below sea level.

    • Maththew Ernest says:

      Flight levels are recorded with a 100ft LSB but tracked to only 500ft increments. You would have to be packing down to a 7-bit field to get an overflow in the right range: one bit for the hundreds leaves 6 bits for the thousands so overflow at FL640.

      Since only 500ft increments are used, Atyrau Airport would round to FL000

  3. alex4pt says:

    That's the letter U and the numeral 2.

  4. Jon Konrath says:

    ERAM is a black hole that the FAA keeps pushing money down to try and replace a 40-year-old outdated system. They went through the same thing in the 90s with a program called AAS, where they burned through almost three billion dollars before saying "fuck it" and going back to a system running on vacuum tubes.

    ERAM is something like four years behind schedule, and they shovel twenty or thirty million dollars a month into it, only to have shit like this happen.

    More fun reading:

    http://www.baselinemag.com/networking/The-Ugly-History-of-Tool-Development-at-the-FAA/

    http://www.ainonline.com/aviation-news/aviation-international-news/2013-10-02/eram-development-reminiscent-failed-aas-program

  5. Darby says:

    Bug... what bug... like y2k bug....

  6. Alex says:

    Probably worth reading this William Langewiesche piece as a side-order. http://www.theatlantic.com/magazine/print/1997/10/slam-and-jam/305134/

    (PS, oh, it used to be so nice that your OpenID implementation worked. now it doesn't seem to provide a callback to the referring post, so just leaves you at the OpenID server.)

  7. I admit that feet > 65535 was the first thing I thought of too. But yeah, since altitudes are stored /100 that's not plausible. The real explanation turned out to be something complicated involving 'VFR above clouds'. I guess the interesting question is what was different this time, why had it not happened on any previous U-2 flight.

    Oh wait, I know: it was the first time there were clouds in Southern California.

  8. William says:

    comp.risks

    Interpretation of press comments:

    As aircraft flew through the region, the $2.4 billion system made by Lockheed Martin Corp, cycled off and on trying to fix the error, triggered by a lack of altitude information in the U-2's flight plan, according to the sources, who were not authorized to speak publicly about the incident.

    Before a controller entered the usual altitude for a U-2 plane - about 60,000 feet - the system began to consider all altitudes between ground level and infinity.

    FAA spokeswoman Laura Brown said the computer had to examine a large number of air routes to "de-conflict the aircraft with lower-altitude flights". She said that process "used a large amount of available memory and interrupted the computer's other flight-processing functions".

    The conflicts generated error messages and caused the system to begin cycling through restarts.

    Lesson learned: Don't put flight plans without altitude information in to ERAM. That will solve _everything_.