Broken images on ye olde blogge

A few weeks ago I started using Cloudfront to CDN the images and MP4s on this blog, and it seemed to be working out ok, but now I'm occasionally seeing broken images where the CDN has seemingly cached a 503 result and won't let it go. E.g. I'm seeing a 503 on this image.

But you probably aren't, because when I wget that image, it comes through fine. So Cloudfront's cache key must include the particular browser headers or something. (Which is, in itself, stupid and wasteful and costing me unnecessary bandwidth, but that's a problem for another day.)

Surprisingly, if I add "?1" to the end of the URL it does not bust the cache. And when I manually add an "Invalidation" of that URL, it also does not re-fetch it.

I have my 503 Error Caching Minimum TTL set to 30 (along with all other error codes) and yet it seems to cache it forever.

Suggestions?

Previously, previously.

Tags: , , , ,

10 Responses:

  1. not pj says:

    Headers shouldn't come into play, unless you changed the default:

    By default, CloudFront doesn't consider headers when caching your objects in edge locations. If your origin returns two objects and they differ only by the values in the request headers, CloudFront caches only one version of the object.

    (from https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/header-caching.html )

    You can actually control that fairly precisely if you edit the "Behavior", then you can set "Cache Based on Selected Request Headers", and similarly "Query String Forwarding and Caching" (not sure what the default is here).

    ... but otherwise I have no idea, creating an invalidation should definitely work.

    • not pj says:

      Something that might be helpful when debugging is checking the headers that you get in response - for example "x-cache" should tell you whether the response comes from a cache hit or miss.

    • jwz says:

      I'm using these settings:

      Cache Based on Selected Request Headers: None
      Object Caching: Use Origin Cache Headers
      Forward Cookies: None
      Query String Forwarding and Caching: None (I guess this is why ?1 isn't a cache-buster)

      Right now I'm getting a 503 on this in the browser. I get the same error with wget if I add "Accept-Encoding: gzip". Without that, no error. So I guess Cloudfront caches the gzipped and un-gzipped copy separately. Response headers include:

      Last-Modified: Tue, 26 Sep 2017 22:16:42 GMT
      Date: Wed, 24 Jun 2020 13:24:03 GMT
      Expires: Wed, 24 Jun 2020 17:24:03 GMT
      Cache-Control: max-age=14400
      X-Cache: Error from cloudfront

      Ok, a few weird things about those dates.

      The only file I have with the "Last-Modified" date is an error page that is only used when Apache is in "maintenance mode", and I'm pretty sure I haven't put it in that mode in years, certainly long before I started using Cloudfront. But the 503 error text it's sending back is not the text from that file! The text it is sending is from an error file that does not have that date.

      Second, where's that 4 hour expiration coming from? I suppose that's the default expiration on the error page HTML itself. I have "Error Caching Minimum TTL" set to 30 on all of my Cloudfront custom error pages, but I guess it's using the expiration of my error page instead? Sigh.

  2. Andrew Klossner says:

    I get a 503 from https://cdn.jwz.org/images/scaled/640/2017/birkhoff3.jpg.
    while running Firefox 68.9.0esr under Windows 10. wget works fine.

  3. jwz says:

    Well now even my css is returning 502 and everything's fucked.

    I've issued invalidations against "/*" three times now and it doesn't seem to help. I also reduced the expiration on my error page HTML to 30 seconds.

    • jwz says:

      Look, here's Cloudfront getting a 304 Not Modified on my CSS, and then continuing to serve up 502:
      64.252.82.141 - - [24/Jun/2020:10:32:10 -0700] "GET /jwz.css HTTP/1.1" 304 - "-" "Amazon CloudFront" 2225

    • jwz says:

      Well I turned off generation of the CDN URLs on the blog so that things are readable again, but all those bogus 502s are still in Cloudfront. Double-you tee fuck.

  4. jwz says:

    Ok, I think what was going on was that my fail2ban rules ended up banning some Cloudfront IPs, as hax0rs tried to load wp-config.php and shit via the CDN. Sigh.

  5. Spike says:

    One thing that bites me when debugging this kind of thing is that Chrome and Firefox have their own DNS cache. You can easily end up talking to one CloudFront edge in the shell and a different one in the browser. That would explain the wget inconstancy.

  6. pgib says:

    I feel your Cloudfront pains. If you wanted to head off some of that WordPress exploit/spam traffic, you can very quickly turn on their "Web Application Firewall" (WAF) product that comes with a bunch of managed WordPress rulesets. It will probably cost about $7-8/month depending on traffic, but perhaps worth it for less of a headache there.

Leave a Reply

Your email address will not be published. But if you provide a fake email address, I will likely assume that you are a troll, and not publish your comment.

You may use these HTML tags and attributes: <a href="" title=""> <b> <blockquote cite=""> <code> <em> <i> <s> <strike> <strong> <img src="" width="" height="" style=""> <iframe src="" class=""> <video src="" class="" controls="" loop="" muted="" autoplay="" playsinline=""> <div class=""> <blink> <tt> <u>, or *italics*.

  • Previously