A few weeks ago I started using Cloudfront to CDN the images and MP4s on this blog, and it seemed to be working out ok, but now I'm occasionally seeing broken images where the CDN has seemingly cached a 503 result and won't let it go. E.g. I'm seeing a 503 on this image.But you probably aren't, because when I wget that image, it comes through fine. So Cloudfront's cache key must include the particular browser headers or something. (Which is, in itself, stupid and wasteful and costing me unnecessary bandwidth, but that's a problem for another day.)
Surprisingly, if I add "?1" to the end of the URL it does not bust the cache. And when I manually add an "Invalidation" of that URL, it also does not re-fetch it.
I have my 503 Error Caching Minimum TTL set to 30 (along with all other error codes) and yet it seems to cache it forever.
Suggestions?
Headers shouldn't come into play, unless you changed the default:
(from https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/header-caching.html )
You can actually control that fairly precisely if you edit the "Behavior", then you can set "Cache Based on Selected Request Headers", and similarly "Query String Forwarding and Caching" (not sure what the default is here).
... but otherwise I have no idea, creating an invalidation should definitely work.
Something that might be helpful when debugging is checking the headers that you get in response - for example "x-cache" should tell you whether the response comes from a cache hit or miss.
I'm using these settings:
Right now I'm getting a 503 on this in the browser. I get the same error with wget if I add "Accept-Encoding: gzip". Without that, no error. So I guess Cloudfront caches the gzipped and un-gzipped copy separately. Response headers include:
Ok, a few weird things about those dates.
The only file I have with the "Last-Modified" date is an error page that is only used when Apache is in "maintenance mode", and I'm pretty sure I haven't put it in that mode in years, certainly long before I started using Cloudfront. But the 503 error text it's sending back is not the text from that file! The text it is sending is from an error file that does not have that date.
Second, where's that 4 hour expiration coming from? I suppose that's the default expiration on the error page HTML itself. I have "Error Caching Minimum TTL" set to 30 on all of my Cloudfront custom error pages, but I guess it's using the expiration of my error page instead? Sigh.
I get a 503 from https://cdn.jwz.org/images/scaled/640/2017/birkhoff3.jpg.
while running Firefox 68.9.0esr under Windows 10. wget works fine.
Well now even my css is returning 502 and everything's fucked.
I've issued invalidations against "/*" three times now and it doesn't seem to help. I also reduced the expiration on my error page HTML to 30 seconds.
Look, here's Cloudfront getting a 304 Not Modified on my CSS, and then continuing to serve up 502:
64.252.82.141 - - [24/Jun/2020:10:32:10 -0700] "GET /jwz.css HTTP/1.1" 304 - "-" "Amazon CloudFront" 2225
Well I turned off generation of the CDN URLs on the blog so that things are readable again, but all those bogus 502s are still in Cloudfront. Double-you tee fuck.
Ok, I think what was going on was that my fail2ban rules ended up banning some Cloudfront IPs, as hax0rs tried to load wp-config.php and shit via the CDN. Sigh.
One thing that bites me when debugging this kind of thing is that Chrome and Firefox have their own DNS cache. You can easily end up talking to one CloudFront edge in the shell and a different one in the browser. That would explain the wget inconstancy.
I feel your Cloudfront pains. If you wanted to head off some of that WordPress exploit/spam traffic, you can very quickly turn on their "Web Application Firewall" (WAF) product that comes with a bunch of managed WordPress rulesets. It will probably cost about $7-8/month depending on traffic, but perhaps worth it for less of a headache there.