More cert fuckery

We have several ancient iPads that we use at DNA Lounge for managing the guest list and so on; and since Thursday's Let's Encrypt firedrill, they can't load our web site.

Presumably updating "ca-certificates" nuked some cert that these old devices require. Perhaps I can put it back and make them work again, but I have no idea how to figure out which one it is. Any suggestions?

The ones that stopped working are: iPad mini Gen 1, 2012, iOS 9.3.5, MFL432LL/A or MD531LL/A.

(Also, we can always give a good home to old iPads. Maybe not quite this old, though.)


Tags: , , , , ,

25 Responses:

  1. Kazriko says:

    No, those old root CA certificates expired, you can't put them back without changing the clock back on every device that might connect to them.

  2. ducksauz says:

    Yup, this is more fallout from the DST Root CA X3 cert expiring. iOS versions less than 10 doesn't include ISRG Root X1 in the trusted root CA.

  3. Ben says:

    It looks like the Root certificate for the Let's Encrypt SSL cert on (and, and my own website) is the "ISRG Root X1". That certificate isn't it the list of trusted root certificates for iOS 9.

    The ISRG Root X1 cert is SHA-256. Many older root ca certificates were SHA-1, which as you may know has some problems. The good news is that there are other SHA-256 root certs listed on that Apple list. Some older operating systems can't even deal with anything newer than an SHA-1 cert. (This is incidentally the way that my own company got out of having to support windows XP/IE 8 end users.)

    You should be able to find some examples online of how to load a root CA onto an iOS device, this is common for accessing something like citrix or vmware vcenter that have self-signed root ca certificates. You might want to load the intermediate certificate on there as well. Here's the Let's Encrypt certificate download page.

    • Tari says:

      Here's a video that provides a worked example of doing so. Download the PEM file of the ISRG Root X1 certificate, then opening it will open a dialog from which the certificate can be installed and marked as trusted.

      • Nick Lamb says:

        Note that, perhaps due to an oversight when adding the highlight after making the video, that video circles the wrong certificate. The voice over is right, you probably want to trust ISRG Root X1, the root authority created by ISRG (the charity behind Let's Encrypt) years ago which is the same thing a modern OS trusts. But the video circles a PEM for the much newer ISRG Root X2.

        Now, in principle ISRG Root X2 seems trustworthy but that probably isn't what you want. It's for elliptic curve certificates only, even if your device is happy doing elliptic curve crypto (most iDevices should be) the random smaller web sites you visit trusted via Let's Encrypt don't rely on ECC certificates, so that wouldn't help you on those sites. ISRG Root X1 covers both types of certificate.

  4. This may be too far into the realm of unhelpfulness that annoys jwz but I wanted to mention you can repurpose the displays using cheap aliexpress driver boards. It’s not gonna help this problem, but I wanted to let you know.

  5. Nick Lamb says:

    Presumably updating "ca-certificates" nuked some cert that these old devices require.

    No. The simple passage of time means those devices no longer trust DST Root CA X3. They see that the self-signed certificate for DST Root CA X3 in their local trust store expired before their current time, and they cease trusting it. You may be able to teach them to trust ISRG Root X1, the way anything that actually got OS updates in the last few years would, you could have done this last week, or last month, or last year, and it would have been a little easier, but you should also be able to do it today. This assumes their trust decision software isn't stupid and can actually figure out that trusting ISRG Root X1 is enough (which it should be).

    The problem you had previously was software (likely based on OpenSSL) that's also dumb enough that it's looking for reasons not to trust certificates, and finding that DST Root CA X3 is expired is enough reason to also disregard trust paths to ISRG Root X1. So their workaround was to have your local machine forget all about DST Root CA X3 in its trust store (ca-certificates) so that the software will focus on ISRG Root X1 instead. Updating to modern software would also have fixed that.

    The reason most old Android devices don't care is that they figure if nobody is updating the trust store then I guess these expiry dates don't matter. So, from their point of view the expired DST Root CA X3 remains trustworthy, or at least, no less trustworthy than the rest of the world for a device that hasn't seen any OS update since the ra*ist was elected president. Whether this is better or worse than the no-updates Apple behaviour or the OpenSSL problem I leave to you to decide, but it is more common and so Let's Encrypt consciously prioritised this case.

  6. g051051 says:

    I have an unused iPad First Gen 64Gb wifi (MB294LL). Too old?

    This is the device that soured me on went out of support far quicker than any other model (30 months), and has been suffering from progressive bitrot ever since. Most games and apps don't even launch anymore.

    • jwz says:

      If it can currently load this blog, I'll take it! But I suspect the answer is "no, it cannot."

      • g051051 says:

        Surprisingly, it looks fine. The only oddity I can see is that the pictures on the sidebar look distorted, like they've been squeezed horizontally and/or stretched vertically. It did initially complain about the cert, but once I said to continue, it was fine.

        • jwz says:

          That's actually kind of funny -- that version of the OS must be old enough that the "accept self-signed or expired certs" was still available. Newer versions give you no option to load anyway.

          • g051051 says:

            iOS 5.1.1 from May of 2012.

          • Mr E says:

            I think you're right about the iOS age—I am able to visit the site on an iPad Mini model MD530LL/A, iOS 9.3.5 (13G36). Safari warns me that it "Cannot Verify Server Identity", but allows me to Continue.

            If you think you can use this iPad Mini, I'm happy to hand it over. It's pretty sluggish, though.

            • jwz says:

              We can probably use it, thanks! It does not take a lot of horsepower to check someone off the guest list.

              (I mean, it's utterly offensive that a computer that is merely ten years old might have trouble loading any web page short of a client-side Playstation emulator, but that horse has left the barn, died, been beaten, and left to rot.)

              • Mr E says:

                We can probably use it, thanks!

                Cool, I'll get it all cleaned up and bring it by. I should be able to drop it off at DNA Pizza tomorrow evening, but feel free to email me if you'd prefer I deliver it another way.

              • g051051 says:

                If you still want mine as well, let me know where to send it. I still have the original packaging!

                • jwz says:

                  That would be great, thank you very much! If you're in town you can just drop it off at DNA Pizza any day after 4pm, or you can mail it to me at the club (375 11th St., SF CA 94103).

                  • g051051 says:

                    OK, I'll get it reset and packed up and get it in the mail to you (I'm on the east coast).

                  • g051051 says:

                    It's in the mail. I can give you the tracking number. Does this count as club business, or personal?

                  • jwz says:

                    Thank you! And I don't even know the difference any more.

  7. Steve Allen says:

    This is probably appropriate background for upcoming posts in the topic

  8. Pronoiac says:

    Something that might work: set up an HTTP proxy, possibly checking docs for Charles Proxy. Maybe add a VPN.

  9. I see that cygwin just did an update adding a ca-certificates-letsencrypt package, so you might want to check for a new ca-certificates update on your particular OS/distro:

  10. Pronoiac says:

    There are alternative providers that offer the same API now. I looked into it because the Let's Encrypt root change bit me too; using ZeroSSL fixed my problem, and it looks like the iPad is compatible.

  • Previously