Fucking certbot

My Apple Pay / Let's Encrypt fuckery is ongoing.

The dnalounge.com cert expires in 16 days and /etc/letsencrypt/renewal/dnalounge.com.conf says "renew_before_expiry = 40 days". So why is "certbot --dry-run renew" saying "Cert not due for renewal, but simulating renewal for dry run"?

Adding "-vvvv" spat out a bunch of XML that did not answer my question.

Previously, previously.

Tags: , , , , , ,

14 Responses:

  1. freiheit says:

    Where are you seeing the 16 days?

    When I look at the cert on https://www.dnalounge.com it shows a cert generated October 7th that expires Wednesday, January 5, 2022

    • jwz says:

      Apple sent me one of their annoying fucking emails today that says "Your website domain that uses Apple Pay has an SSL certificate that expires on Nov 6, 2021."

      • Nick Lamb says:

        That would have been helpful information for the body of the original post, which, at time of writing just complained about the certificate, not about this email from Apple.

        Now that we know the actual symptom, there are two possibilities that are interesting here. One of which you might be able to do something about if that's where the problem is, the other I would guess not.

        Because we know Apple don't care, there probably isn't more information, but I'll ask just in case: Do they by any chance tell you where and how they found this certificate that's expiring on November 6th?

        1. Maybe somewhere there is a service that Apple are talking to which believes it is dnalounge.com and didn't know about the renewed certificate. Backup web server. SMTP relay you forgot about. Anything. You being an unfrozen caveman I'd guess this is likely to be an actual computer somewhere or at least a virtual machine with some stray server program that oh yeah, only notices new certificates when it's restarted. I would go and look for that service so I can fix it, but clearly you and I have very different approaches to problem solving, so feel free to just reboot everything if that seems easier.

        If that's the problem, you need to modify your Certbot renewal hook so that it triggers all the necessary changes on renewal, not just whatever it's doing now. Most modern server software can be signalled to do soft reloads, but for sure if you don't know it can't hurt to restart the whole program.

        2. Or maybe Apple just doesn't bother actually checking as often as they claim. After all, they have all the money in the world, so unless they annoy somebody important like the Orange Buffoon or Zuckerberg they needn't sweat it when they blow up yet another small business.

        • Al Iverson says:

          Yeah, what Nick said. I was getting a similar Let's Encrypt renewal notice for two of my domains and was confused at first until I remembered that I moved those two of my domains to new hosting not too long ago and had set up certbot to provide a new LE SSL successfully on the new host. The certbot warning was for the cert on the OLD host, and there wasn't really a darn thing in the email itself to make that clear to me.

          • jwz says:

            I only have one host and I haven't changed anything recently, except for the Let's Encrypt fire drill a few weeks ago.

            I don't know what SSL cert Apple might be referring to if it's not the one attached to the dnalounge.com web site, as that's the only thing remotely related to Apple Pay. They're not gonna be looking at my SMTP server. (And that's the same cert anyway.)

            • Nick Lamb says:

              You appear to have plenty of company at least:

              https://developer.apple.com/forums/thread/663425

              There are other threads and I'd say the general thrust is, it sometimes works and there's no clear rhyme or reason.

              There's some sign that what Apple is trying to do here is verify that dnalounge.com is still the same business that signed up for Apple Pay and not a different business which bought the domain name. That's not what certificates in the Web PKI are for, and so they can't reliably achieve what they want, but I think that's what their engineers are trying for. Whatever the goal, this process clearly doesn't work for lots of small businesses, which if they were a tiny startup might actually matter to somebody, but presumably Apple couldn't give a shit. Sorry about that.

              • Nick F says:

                https://crt.sh/?id=5004905514
                That one?
                Maybe Apple are crawling CT logs for this? Would be strange but sadly not surprising.

                • Nick Lamb says:

                  It seems unlikely that Apple went to the bother of writing a CT monitor, I actually built one for a previous employer and have one of my own from years back, and they're a considerable maintenance pain, yet one that doesn't really address Apple's use case at all. It's also unlikely that they are banging on Rob's site (crt.sh) either with an awful web scraper or using the actual database like a grown up.

                  It's doubly unlikely that they, on the one hand, implemented a CT monitor and yet on the other hand can't correctly discern that dnalounge.com has a more recent certificate.

                  My guess would be that something is broken in their periodic certificate checking code and when it fails rather than go "Oh, I'm broken, somebody should fix that" or even "I should tell the customer I broke", it goes "Eh, I have this out-dated information, I'll treat that as success" with the result that many (most?) businesses just have to do the full re-validation process every few months. Apple's own engineers seem uncertain about what, if anything, the periodic check does, e.g. whether it tries to read a particular file on your HTTPS site.

                  Speaking of which I note that the apple-developer-merchantid-domain-association.txt gets me a 403. I'm sure Jamie thought of that at least and has contemplated logs to see if Apple cares about this file after initially validating it.

                  It's certainly possible that the checking code snaps out of its coma if shown a very new certificate at the right moment, or that it's just broken entirely (and only initial validation works) for 90-day certificates even though Apple is supposedly onboard with further reductions in certificate lifespan.

                  But as I said, why should they give a shit? Even in this thread Jamie is reluctant to blame Apple for the problem with their system wasting his time and money, preferring to blame volunteers and a not-for-profit instead.

              • Derpatron9000 says:

                Itjustworks.png

  2. SteveB says:

    Have you tried “--force-renewal”?

  3. Rev Matt says:

    I always just force renew

  4. Jason Cook says:

    fwiw I gave up on certbot and switched to using lego for dealing with letsencrypt: https://go-acme.github.io/lego/

    The hell that is the out of tree plugin ecosystem is what started me on the path of avoiding certbot sent me that way, just working was what kept me there.

  5. Alasdair L says:

    We use this at work, and it hasn't let us down. What can I say, I have a perverse love of giant shell scripts:

    https://github.com/acmesh-official/acme.sh

  6. Matt says:

    I was similarly getting notifications about certs expiring, but when going to renew them, certbot said it was not necessary.

    I think in my case what happened was that certbot created a cronjob at install time that automatically renewed certs as necessary, so my add-on cert expiry checking was a red herring.

Leave a Reply

Your email address will not be published. But if you provide a fake email address, I will likely assume that you are a troll, and not publish your comment.

You may use these HTML tags and attributes: <a href="" title=""> <b> <blockquote cite=""> <code> <em> <i> <s> <strike> <strong> <img src="" width="" height="" style=""> <iframe src="" class=""> <video src="" class="" controls="" loop="" muted="" autoplay="" playsinline=""> <div class=""> <blink> <tt> <u>, or *italics*.

  • Previously