GPIO noise

The payphone regularly gets GPIO events that are not real. I have four switches attached to GPIO pins, as well as a MAX7219 LED 128x32 display, and I get ghost open/closed events on the switches several times a day. There is no chance that the switches themselves are triggering. And these aren't intermittent bounces: it will register a switch closing, and then it opens again ~20 seconds later. The wires to the various switches are under a foot long.

Also, there's a USB keyboard attached (the keypad) and I regularly read impossible characters from it when it's not even being touched: NULLs and low-numbered control characters. Note, this is not a serial port, these are making it all the way up the USB stack.

So I'm guessing there's some systemic grounding shit going wrong, or something. Any suggestions?

In other payphone news, it only took a month for one of the savages we call customers to completely destroy the rebuilt payphone handset. So that's great. I guess I have to re-build that from scratch now, since it's impossible to get the end caps off once you've screwed them on. Or, just toss the whole phone in the dumpster, because you people don't deserve to have nice things.


Tags: , , , , , , ,

25 Responses:

  1. Tod Kurt says:

    I solved mains noise injecting glitches by using an AC line conditioner. It was from TrippLite. Maybe that could help?

    If it was just GPIO glitches, I’d suggest stronger pull-up/pull-down resistors.

  2. Brian Enigma says:

    Is the connection just directly from the switch to the GPIO pin, or are pull-up/pull-down resisters involved? You can definitely get some weird floating-logic behavior without a resister in there to help nudge things to a solid zero or one. (Please disregard if you're already using a resistor. If that's the case, I'm not sure I have a good theory, other than poltergeists. Which would be my same theory for the USB, since that's a little more out of my realm. But GPIOs I know.)

    • jwz says:

      Pis have built in pull-up/pull-down resistors and I am using them:

      bcm2835_gpio_fsel (pin, BCM2835_GPIO_FSEL_INPT);
      bcm2835_gpio_set_pud (pin, BCM2835_GPIO_PUD_DOWN);

      • Lady Errant says:

        Built in pull up resistors are typically not very strong, I'd try adding maybe 4.7k resistors and turn the onboard ones off.

        • tfb says:

          I was going to say exactly this. If it says it has built-in programmable pull-up or pull-downs it probably effectively only has them in some nominal sense which is definitely not going to work in a noisy environment. I'd work out what current whatever is driving the pins is up to and have real physical resistors.

          However this is based on pretty distant experience now.

      • relaxing says:

        Which pins are you using? The first 8 as well as 34-36 can only be pullups.

        • jwz says:

          What the fuck? I mean what the ABSOLUTE FUCK? This is what I hate most about all of these microcontrollers. Every god damned pin is a unique fucking snowflake.

          I'm using 7, 11, 12 and 13 for switches (powered from 4). The LED thing uses 2, 6, 19, 24 and 23.

          • relaxing says:

            Connector 7 seems suspect. I’d also be wary of 12, since it’s shared with the PCM peripheral and someone else could be driving it.

            I see the LED driver is using SPI, so no need to touch those pins.

            Other thoughts:
            Failing power supply could be a culprit?
            Add ferrite chokes on usb cables?

          • Zygo says:

            The pinmux chapter in the manual for these chips is typically hundreds to thousands of pages long, and it's all tables of "pin #573 supports these 3 modes, pin #574 supports these 7, pin #575 has only 2." If you're lucky, they'll even tell you what the snowflake option fields mean, as opposed to tables listing only the numeric values accepted by the hardware.

            The Pi's SoC is designed to live in smartphone-sized devices, where the longest wire off the chip is maybe 2.5 inches, and the whole thing lives inside a foil box for EMI isolation. They switch at low voltages and drive tiny currents to save power, which makes them awful for GPIO on mechanical switches at payphone-sized scales--they'll report a switch press if someone flips a light switch in the same room. You'll need some kind of external hardware interface to filter out assorted noises, though an external pull resistor or cheap buffer chip might suffice for that.

            I have a jar of random 74xx logic gates left over from projects I built in the 1990's. I build a lot of projects that look like "Raspberry PI GPIO -> XOR gate with one input tied to ground -> device" because most of the 74LSxx chips have lovely signal filtering features and brute output force that the Pi does not.

          • rozzin says:

            More importantly: which GPIO-numbering scheme are we even using for this discussion? Because Raspberry Pi has at least 3 different "standard" schemes, none of which have any relation to each other:

            * "BCM" numbering: the pin-numbers according to the Broadcom SoC
            * "P1" numbering the order in which the pins are connected to the Raspberry Pi breakout header
            * "wiringPi" numbering: the numbers arbitrarily assigned to GPIOs by the guy who wrote the "gpio" commandline utility
            * if you're using a Compute Module, then you also have the SODIMM pin numbers where the GPIOs are brought out

      • David Konerding says:

        Yeah, Pi's pullup resistors are not so great. For me, however, the only thing that fixed my switch noise (I tried various external pullups, RC filters, etc) was to put a Schmitt trigger on the switch using one of these
        after which all my switch problems went away.

        However, from your description of the problem (happens sporadically throughout the day), I am unsure whether the schmitt would make a difference. If I had to debug this, I'd probably lug out the oscilloscope with a trigger and look at the noise history, before the switch was invoked. Is it just a little switch noise? Or a huge DC change that lasts 20 seconds? You're in such a crazy EMF environment that pretty much anything random could happen.

        Nonetheless, it's worth trying; the component is cheap, trivial to wire (one wire in, one wire out, one Vcc, one ground), and you just sort of sit around waiting a couple days to see if the problem goes away :(

  3. AMS says:

    For something in that sort of environment you're probably better off with stronger pullups/pulldowns (10k or less) and RC filters for EMC protection (100ohm-10nF). AVRs are pretty robust but even they can only take so much ESD on their unprotected pins.

  4. Tux2000 says:

    Some generic advice:

    Make sure you have a stable supply. Add small (10..100 nF) and big (10..100 µF) caps to the supply line, preferably as close to the processor as possible.

    Add a common mode choke to the supply lines.

    Add small caps (100 nF .. 1 µF) in parallel to all push buttons to suppress HF noise.

    Keep all wires as short as possible.

    Avoid suppling inductive loads from processor supply.

    Make sure that you have flyback diodes on inductive loads.

  5. thielges says:

    How did the handset break? The plastic housing for commercial units is very robust, designed to tolerate harsh locations crawling with vandals. They can be broken though it requires pretty strong whacks against the steel phone case to break the plastic.

    I’m tempted to suggest including an accelerometer triggered taser zap to the rebuilt unit, but that might just encourage the cretins.

    • jwz says:

      Yeah, no shit. They managed to yank on it hard enough to pull wires out of the cable connection to the handset, and I don't even know how that's possible.

      • Elusis says:

        Maybe you should re-consider admitting gorillas to the club after all...

      • kwk says:

        Bunch of savages in this town.

      • thielges says:

        OK, that matches the most common failure mode I've seen in the wild. I guess the cable connection is the weak link. One design trick to partially thwart this kind of vandalism is to design that connection to detach at an electrical connector when forced with over 40 pounds or so. Put an nondetachable steel cable inside the housing with about an extra 4 inches of slack to retain the assembly. That way the vandal gets their "Woot Woot! I destroyed something!" rush to quell their insecurity and leaves the equipment mostly intact for a tech to plug it back together later.

        Yanking a handset cable can be done artfully as Peter Gabriel performed Come Talk to Me. I still get chills when seeing how he and the band extracted so much emotion out of a simple conveyor belt prop. If you don't have time for the whole six minute video, advance to 2:45 when the cord yankin begins.

  6. Dan Steingart says:

    Did you try powering the uC from a battery to isolate? If that works something like this will help.

  7. Lloyd says:

    Life goals: HACK THE HANDSET!

    That Hackers movie was way cooler than this.

  8. Carlos says:

    In addition to the external pull up/down resistor changes, which I'd also recommend, there's the following...

    The cast-metal case/chassis of the phone should make an excellent Faraday cage. I presume you've already got a nice solid electrical ground connection to the case?


    • K J says:

      Yes / no. The mounting plate is typically a big block of plastic, so depending on the design of the phone it might be open to RF from the back.

  9. Pronoiac says:

    I saw that solipsist on tiktok was having wtf fun with payphones, and thought you should see it.

  • Previously