Can anyone tell me in small words how to fix this?postfix/smtpd: warning: TLS library problem: 6374:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:s3_pkt.c:1493:SSL alert number 45:
Re-running "certbot renew" does nothing.
Running "openssl x509 -enddate -noout -in" on every .crt file I can find shows expiration dates in the future, including /etc/letsencrypt/
CentOS 7.9, certbot 1.11.0 which appears to be the latest.
Rather than actually providing instructions for how to fix whatever they fucked up, they just refer you to the "support" "forum", both words of which are highly aspirational.
@letsencrypt Nothing brightens my day like users asking me why something stopped working right as I'm wrapping up for the day, only to find a service I'm forced to rely on has changed something, and the first response in the forum thread they link to tells me I'm incompetent and don't care
Update: Great news, everybody! When you run "openssl x509 -enddate -noout -in" on a file that has a thousand certs in it, it only reports on the first one! Because obviously that's exactly what you'd want and expect. So it looks like /etc/
FYI - I saw this from "The Register" which has some links to errors at Let's Encrypt.
https://www.theregister.com/2021/09/30/lets_encrypt_xero_slack_outages/
A root cert expired - should have gone to a more recent cert but several software bits are buggy (including several iOS versions - list in the story). I hope this helps.
I'm sure it's related, but my problem is not with "old Android devices", it's with my CentOS machine being unable to use outbound SMTP TLS with Let's Encrypt's certs.
I looked further at some of the links. A comment in this one recommends deleting the expired root cert file from Linux, Mac, or Windows systems
https://stackoverflow.com/questions/69387175/git-for-windows-ssl-certificate-problem-certificate-has-expired
The Linux example is for Ubuntu but you might have the same issue.
Yeah, from what I understand: there were two certification paths available to get to one of LE's roots, and an intermediate in one of those paths was expiring. More up-to-date software handles it just fine and knows that it just needs to use that alternate path, but due to Redhat's (and therefore CentOS's) version/backport policy, EL7 boxes are still using an old version of openssl which has a bug which prevents that from working properly.
Getting that expired intermediate cert completely out of your CA trust store should theoretically take care of it for you... There was an update to their "ca-certificates" package a little while ago which gets rid of it, specifically ca-certificates-2021.2.50-72.el7_9. A "yum update ca-certificates" should get it updated for you. I wouldn't be surprised if some service restarts might be needed to pick up that change, after the fact.
(I can't actually guarantee that'll take care of your issue, but it's worth a shot, anyway.)
I imagine you've probably stumbled across these yourself, but some more info that I'd run across:
https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
https://blog.devgenius.io/rhel-centos-7-fix-for-lets-encrypt-change-8af2de587fe4
Er, and by "one of LE's roots" what I actually meant was "one of LE's signing certs." Meh.
I had simular problems on my Centos Machines as well. I had to update the CA-certificates package. And everything started working again. Hopefully this helps
I read this in May because they emailed me (and everyone else who has a cert from them):
What I think you're seeing now is other software being crap. Let's Encrypt certs are signed by an intermediary that's signed by both an old root cert that was widely trusted but would eventually expire (today) and a new root cert that's now widely trusted but wasn't on the day they started.
When shown a cert that could be verified with two different root certs, whatever TLS library you're using, that Let's Encrypt didn't write picks the expired one and then complains it's expired. I like what your other poster is saying, delete the expired "DST Root CA X3" cert.
The command you ran gave you false hope.
ca-bundle.crthas lots of certs concatenated together, youropensslcommand only printed the first one. I believe you can find the expired "DST Root CA X3" cert somewhere in/etc/pki/ca-trust/source/anchors/, delete it, then runupdate-ca-trustto rebuild the ca-bundle.crt file.I think that "yum update ca-certificates" fixed it. CentOS does not have "update-ca-trust".
Hm, it should; that's even supplied by ca-certificates. It's run by the ca-certificates postinstall script, too:
# rpm -q --scripts ca-certificates | grep -A99 postinstall
postinstall scriptlet (using /bin/sh):
#if [ $1 -gt 1 ] ; then
# # when upgrading or downgrading
#fi
/usr/bin/ca-legacy install
/usr/bin/update-ca-trust
Glad that the update seems to have straightened it out! There's been some drama about this update in #centos on Libera.
You, sir, are a lifesaver!!!
GFY
might be worth to take the time can check what other security updates your system is not getting correctly.
This can also be fixed server-side to immediately make it work for non-patched clients, simply by not including the "DST Root CA X3" certificate in the bundle provided with your server certificate. This works since "ISRG Root X1" is already (and also) a trusted CA client-side.
Same thing happens if you run certbot (>=1.6.0) with argument: --preferred-chain ISRG Root X1
Client-wise for older distributions, both CentOS 7 and Debian 9 have ca-certificate and gnutls/openssl updates that fixes the issue. Do note that apt-get for https://deb.debian.org/ won't work until the security update have been installed on Debian 9. Furthermore, I thought all hope was lost with CentOS 6, but CloudLinux have now provided updated openssl packages for EL6 as well.
Related: https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
Yeah, OpenBSD just sent out a patch for it, but it might take a bit of work to get the patch: http://undeadly.org/cgi?action=article;sid=20211001073034
Oh hey, I used to be an editor for undeadly. Doesn't look as if it is much work, really. For your average OpenBSD dev, and a lot of their users that reads like: crack open a beer and get it done before you've finished the brew.
Though, I am guessing since macOS also uses LibreSSL these days, we'll maybe be seeing Apple playing catch up eventually? It appears as if this bug existed in older versions of OpenSSL (and since LibreSSL was forked older versions of OpenSSL, they inherited it). LibreSSL 3.3.5 released on October 1st, 2021 rectifies it. It appears as if this was also rectified in OpenSSL 1.1.x (for reference, OpenSSL 1.1.1 was released in 2018).
However, macOS 11.6 (Big Sur) uses LibreSSL 2.8.3 (wow, they are really far behind. But then, I tend to rely on LibreSSL from macports, because I am not a masochist). Even if you are running LibreSSL from macports, it is only up to 3.2.3. It appears as if homebrew (yuck) is using 3.2.4, so contact your uhhh, port maintainers I guess?
I wonder if anyone has filed a bug with Apple yet? The last time I filed a bug against their OpenSSH drift, it got closed as a duplicate, and wasn't fixed for two years, with no attribution (either to me, or whomever else also filed bugs against it previously). It must be really hard for a company with $200 billion in cash to patch libre/free open source software I guess.(/sarcasm). It isn't so hard for Tim Cook to pay himself $750 million bonuses though! Even Steve Jobs had more decency than that guy.
Too bad I seem to be unemployable these days. ;-/
I'm guessing Apple will have a patch, eventually. In time for macOS upcoming Monterey's release? I won't be holding my breath.
The irony that jwz encountered this due to SMTP is not lost on me. Ah, remember when the S in SMTP stood for Simple, and it was considered an intractably insecure protocol by the likes of Robert Morris (chief scientist of the NSA and the guy who implemented the crypt command in Unix, also father of Robert Tappan Morris, author of the first widespread internet worm) so why even bother trying to encrypt it?
I suppose some use the acronym SMTPS to represent SMTP tunneled over TLS, but that is still not a default, just as plaintext FTP is still the norm even though tunneling FTP over TLS is older than the name change from SSL to TLS and was supported by warez FTP daemons such as glftpd. It's almost as if grafting encryption on as an afterthought rarely ever works out well, or something!
Really though, I haven't run CentOS in prod anywhere since 2007, and that was my employer's choice and was ripped out and replaced with a less intractably unhip Linux distro. I don't really keep up with what the Linux "cool kids" are doing these days, but I think it is something like arch or Manjaro. Ubuntu is presumably considered tainted after the Microsoft WSL debacle (where you can get 4X better performance by running Ubuntu in a Hyper-V VM than WSL provides, but at least you side step the annoying Cygwin DLL hells because everyone and their grandmother decided to use Cygwin for porting code instead of Visual Studio, apparently).
Sorry, I may have too much snark going on at the moment. At least it's Friday night and DJ Bat/Xorcist of House of Usher notoriety is streaming, which yields fonder memories of simpler times online where cyber had less humor or dullard infosec connotations and more scene deference.
It does seem sorta sus to pin this on Let's Encrypt, instead of OpenSSL bitrot, but that seems to be how the media is spinning it. Heaven forbid that there be any hints of collusion against a CA providing free certs in an industry rife with egregious profiteering and known malicious certs in the CA infrastructure though, but wait, I already apologized for too much snark, didn't I?
As an update, it appears as if homebrew has bumped their LibreSSL port to 3.3.5 (which is the stable release from liressl.org as of October 1st) which presumably rectifies this issue.
I did email the port maintainer of the MacPorts release, though perhaps the formality of a Trac ticket would have been a better course of action? Their Trac system already has a yellow flag indicating that the port is behind a newer release of 3.4.0. Albeit, the port maintainer also appears as if he has a lot of other ports he maintains, as well as a day job, so my guess is he has a lot on his plate already.
I also noticed that Macports has libressl-devel which at version 2.9.2 appears to be significantly behind in versions relative to the main port, and I am guessing is maybe an abandoned port? Albeit, 2.9.2 is still newer than what macOS Big Sur 11.6 ships with in base as previously mentioned. I am not running any Monterey beta builds to see what the LibreSSL is there, but hopefully that is addressed by people at Apple who actually get paid to do such things. I've already got too many unpaid hobbies and volunteer opportunities as it is.
As an additional update, I ended up opening a Trac ticket for MacPorts as it appeared as they still haven't done a version update to 3.3.5.
Follow the progress at home here: https://trac.macports.org/ticket/63615#ticket
Admittedly, before submitting that I changed a few lines to my local Portfile for LibreSSL and it rebuilt without issues. So if you are one of those users who can't stand Homebrew, and can't have this port out of drift, you can read my notes in the Trac ticket and presumably get it up to date yourself without much more than reading along with the instructions I referenced.
Of note: at least according to MacPorts, nothing depends on this port. Thankfully, if you use LibreSSL, everything which depends on OpenSSL will instead default to LibreSSL.
Since macOS itself uses LIbreSSL, I would imagine that maybe MacPorts would try to prioritize this port a little bit more, but I don't work for Apple (even though it appears as if the port maintainer of LibreSSL does?), so I have 0 idea how they prioritize anything there. To paraphrase nicoles: "New Apple product, now with 20% higher profit margins!" seemed to be their way.
This fuckup caused other widespread damage. It prevented a bunch of my systems from being able to get updates from kernel.org's mirror of distro repositories, for example.
Pain in the ass.
C.
For those of you boneheads who keep wanting to say "WELL ACTUALLY it wasn't Let's Encrypt's fault" -- it is certainly, 100%, their responsibiity. Let's Encrypt's pitch is "Look at how cool we are for making it trivially easy for you to get certificates and keep them up to date." Guess what they failed at doing? 1) Keeping the certificates functional 2) Making that easy.
That "certbot renew" does not repair, or even detect, whatever-it-was that went wrong yesterday is inarguably a bug in certbot and/or Let's Encrypt's underlying infrastructure.
I will not be taking any questions.