More Apple Pay certificate fuckery

Fucking Apple.
Dear Lazyweb, can anyone explain this to me?

Starting a few months ago, I now get this email from Apple about once a week:

Your website domain that uses Apple Pay has an SSL certificate that expires on Nov 6, 2021. After automatically trying to reverify your domain, we found that this SSL certificate has not been updated. Your domain is automatically checked 30 days, 15 days, and 7 days before this expiration date.

If you have an updated SSL certificate and the domain hasn't been successfully verified 7 days before expiration, please revalidate it manually by Nov 6, 2021 in Certificates, Identifiers & Profiles to ensure uninterrupted use of Apple Pay on your website.

And yet, when I follow the incomprehensible trail of breadcrumbs to:

  • Certificates, Identifiers & Profiles
  • Select "Identifiers"
  • Select "Merchant IDs" from menu on the upper right
  • Select "merchant.com.dnalounge.store"
It says

Merchant Domains

Domain: www.dnalounge.com
Status: Verified                                 [ Remove ] [ Verify ]
Verification Expires: Nov 6, 2021

The last time this happened, I emailed Apple support twice, was ignored, and when the expiration date passed, sure enough, Apple Pay stopped working. (And the expired whatever-it-is is not listed on the top-level-ish page that shows your expired things. It's in the basement, behind the sign saying "Beware of leopard"). At that point I had to re-add the domain, and download another big binary blob to replace the big binary blob that was already in "/.well-known/apple-developer-merchantid-domain-association.txt".

I would like to live in a world where I don't have to hand-edit that file every god damned month. What is going on? Is this some new Let's Encrypt-related fuckery?

Previously, previously, previously.

Tags: , , , , , ,

20 Responses:

  1. Ben says:

    This sounds very much like apple is set up for organizations that purchase SSL certs at yearly intervals and install them a month in advance and not Let's Encrypt certs that expire every quarter and that the certbot replaces with like 20 minutes to go or whatever.

    My only suggestion is to make sure that certbot or whatever you're using to install new Let's Encrypt certs installs a new one more than 7 days before the old one expires so Apple's cert verifier sees a new one at the 7 day interval and you don't have to do the manual /.well-known/blahblah file upload bullshit again.

    • jwz says:

      But I have been using Apple Pay for like a year, and have been using Let's Encrypt since forever, and this shit just started happening recently. I don't even understand what Apple is checking.

      • frymastter says:

        It's just checking the website SSL cert, surely? Certainly when I go to http://www.dnalounge.com it says the 6th of November is when the SSL cert expires

        • jwz says:

          But as I said, the last time this happened, I did nothing, and when the date passed, Apple Pay stopped working. Despite the fact that Let's Encrypt had absolutely updated my web site cert with a new one in the meantime.

    • jwz says:

      Like, maybe Apple is saying, "every time your web site's SSL cert is renewed, you need new shit in .well-known, and no there's no way to automate that", but I can't even tell if that's what they're saying.

      • Jered says:

        It's accurate that your current cert expires on Nov 6. If you force-renew with certbot now, what happens when you click that mysterious "verify" button?

      • Ben says:

        I think what they mean is "If we can't automatically verify at 30, 15, or 7 days before your current SSL cert expires that you have replaced it with a new SSL cert that expires more than a month in the future, you'll lose your verification and have to do it again manually".

        This error is not one I have yet to experience; however as the SSL point person at a small dev shop that does online / mobile banking work for community financial institutions I have run into a lot of similar nonsense from big 3rd parties around payments, especially around the short-term-ness of Let's Encrypt certs.

        "Did you know this certificate expires in 60 days— why haven't you loaded a new one yet?! Is this a class 3 certificate?" etc.

        • jwz says:

          What's the default amount of time before expiration that Let's Encrypt renews things? And is there a way to tell it to do it earlier?

          • Lee says:

            Let's Encrypt certificates last for 90 days, but are recommended to be automatically renewed after 60 days. i.e if your cert is set to expire in under 30 days, it's an indication the renewal process has not occurred.

            https://letsencrypt.org/docs/faq/

            • Lee says:

              Obviously set-up dependent, but if you have a cron job running "certbot renew" every 12 hours, and an httpd restart script in /etc/letsencrypt/renewal-hooks/deploy/ it should auto renew and apply live before the 30 day expiry.

          • CJ says:

            I think it depends on what you're using to do the renewals -- I believe that certbot's default is 30 days prior to expiry. Some digging indicates that they use an internal `renew_before_expiry` var to control that, whose default is "30 days" (yes, an English-language string). I'm not sure where exactly you can override that yourself, though it seems like you can maybe do it on a per-domain basis inside /etc/letsencrypt/renewal/<certname>.conf

          • Ben says:

            If you're using certbot, there should be a configuration file for this, or at least a cronjob that runs at an interval to process the renewal. The latter is how our infra is set up, but it looks like the newer versions of certbot have significantly changed according to these docs (certbot-auto doesn't exist anymore even) so YMMV.

            Here's how my stuff works: we have a monitoring script that checks the expiration on all of the sites' certs, and if it's a let's encrypt cert and there's 32 days left on it, certbot is invoked to renew it. That way all the "your certificate expires in 30 days" hair on fire emails don't get sent.

            • jwz says:

              Well, I'm running "certbot renew --renew-hook ..." every day at 5am. Nov 6 is 29 days from now. Maybe it should have renewed last night, but I ran it by hand just now and it renewed, and now says Jan 5.

              I changed "30 days" to "40 days" in /etc/letsencrypt/renewal/*.conf, so let's see what that does next time...

  2. CJ says:

    I've got, like, negative amounts of experience dealing with Apple, but it sounds like they're literally just checking the expiration date of the SSL cert you've got at https://www.dnalounge.org/. That cert does, indeed, expire at Sat, 06 Nov 2021 11:00:41 GMT, though of course certbot/whatever should get it renewed at some point before that point. I'd expect the re-verifications to continue to show the Nov 6 date until the cert's been updated.

    I suppose the real proof would be if you forced a cert renewal via Let's Encrypt (I assume such a thing is possible?), doublechecked that the site starts using the newer cert, and then see if that warning goes away. I suspect that the reason you've never seen it before could just be that maybe Apple's proactively alerting merchants due to presumably-largeish numbers of them not getting their SSL certs updated prior to expiry.

  3. Krisjohn says:

    We use Let's Encrypt for lots of things, but one specific use is an internal Wi-Fi. It "expired" and stopped working before the certificate expired this month. It appears to be something about a Let's Encrypt root certificate. Just try renewing early.

    • Jred says:

      That was the DST Root X3 expiration fuckage, which was totally epic and wide-reaching, as jwz hints at above. This is almost certainly unrelated.

      As an aside, Let's Encrypt got to choose between "support Android<7" and "support old LibreSSL" for the default chain and .... chose poorly.

      • Krisjohn says:

        Still, if your 90 day certificate can be renewed after 30 days and Apple complains at 60 days, just renew it at 45 days. No reason to wait until the last week out of spite.

      • Nick Lamb says:

        I think that prioritising the vastly larger population of relying parties who are unlikely to be able to understand what's wrong with their aging Android phone let alone fix it, over the software developers who apparently can't be bothered to get something right once, and then won't push out critical fixes for their dumb errors is exactly the right choice. But I appreciate that fruit cultists don't see that.

        It's certainly telling that this bug is in OpenSSL and inherited in LibreSSL and they didn't say, fix it in their first months of frenzied changes, suggesting that in fact OpenBSD's team isn't too hot on this stuff and kind of assumed the way OpenSSL did it must be correct. Same bug was also in GnuTLS.

        • Jered says:

          It ends up being more nuanced than that. Yes, all of Apple's built-in unix-like tools (fucking cURL even) are built with broken libreSSL. But I had the joy of discovering that the third-party email client that Motorola ships on their _brand new_ Android 11 phones also is fucked in this way.

          So, I argue that the right answer would have been "ditch Android<7" because those devices are probably outrageously full of security holes due to Google's utter lack of a support ecosystem. Maybe having certs fail will finally get them into the trash heap where they belong.

        • グレェ「grey」 says:

          " telling that this bug is in OpenSSL and inherited in LibreSSL and they didn't say, fix it in their first months of frenzied changes, suggesting that in fact OpenBSD's team isn't too hot on this stuff and kind of assumed the way OpenSSL did it must be correct. Same bug was also in GnuTLS."

          Except, is it really telling? LibreSSL started circa 2014, and this bug wasn't rectified in OpenSSL until 2018, so it wasn't exactly under the magnifying lens in the first months of frenzied changes (I'm inferring that you mean LibreSSL had frenzied changes, since well, it did and ripped out something like 40KLoC in the process during earlier development). Still, it's a shame that LibreSSL devs failed to notice it being addressed four years later and only got around to fixing it with 3.3.5 this month. it.sh happens.

          Of course, if you get to know some of the devs personally, you'll know they've got a lot of other things going on in their lives, and at least one of them was contending with COVID-19 last year, before a vaccine was even available, so I don't tend to lay a ton of harsh vibes at their feet. On the whole, they tend to do exemplary work, with little in the way of financial incentives which seems more than commendable in an industry rife with cronyism.

Leave a Reply

Your email address will not be published. But if you provide a fake email address, I will likely assume that you are a troll, and not publish your comment.

You may use these HTML tags and attributes: <a href="" title=""> <b> <blockquote cite=""> <code> <em> <i> <s> <strike> <strong> <img src="" width="" height="" style=""> <iframe src="" class=""> <video src="" class="" controls="" loop="" muted="" autoplay="" playsinline=""> <div class=""> <blink> <tt> <u>, or *italics*.

  • Previously