The Great Firehose of China is aimed at me again

Our friends in China took a dump on my server again this morning, despite the ~900 networks I had blocked in iptables.

I updated the list from here but that didn't help.

So I grabbed every IP that hit me this morning with an incorrect "Host:" header, constructed a /24 from that, and blocked that too. That was 3000+ netmasks, and is obviously not the right netmask, but it took the traffic down to a more manageable dull roar.

Has anyone yet worked out a more effective way of coping with this Chinese DNS poisoning?

I'm also really curious about how my IP keeps getting targeted. Are they really just picking a random number and I keep getting lucky, or is there some dickwad in a Chinese ISP smirking as he aims the firehose at me specifically? Is my blog or my bar considered counterrevolutionary?

Previously.

Tags: , , ,

26 Responses:

  1. jwz says:

    Incidentally, to figure out what IPs are hitting your server with the wrong Host header, a relatively painless trick is: 1) make your first, default, VirtualHost be for "ServerName example.com" and put its logs somewhere else; 2) add "%{Host}i" to the end of the "LogFormat" used by that host; 3) The VirtualHost section for your real site or sites follow that.

    That segregates traffic with an unmatched Host: header into its own set of log files and keeps them out of your real document tree. Anything that shows up in those logs is bogus and can be nuked.

    E.g.,

    # include the host header
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{Host}i" combinedhost

    # default host (for bogus Host: headers)
    #
    <VirtualHost *:80>
     ServerName example.com
     DocumentRoot "/var/www/nullhost"
     CustomLog logs/nullhost/access_log combinedhost
     ErrorLog  logs/nullhost/error_log

     ErrorDocument 401 "<H1>Your DNS is broken!</H1>"
     ErrorDocument 403 "<H1>Your DNS is broken!</H1>"
     ErrorDocument 404 "<H1>Your DNS is broken!</H1>"
     ErrorDocument 500 "<H1>Your DNS is broken!</H1>"

     AddDefaultCharset Off
     <Location ~ "^/announc">
       ErrorDocument 403 "d14:failure reason13:not a tracker8:retry in5:nevere"
       ErrorDocument 404 "d14:failure reason13:not a tracker8:retry in5:nevere"
     </Location>

    </VirtualHost>

    # Virtual hosts for sites actually served here:
    #
    <VirtualHost *:80>
     ServerName http://www.your-site.com
     ...etc...
    </VirtualHost>

  2. Other Jamie says:

    Perhaps changing the logo to a big red star would help.

  3. Seth says:

    Why not just "fix" your webserver to serve up a nice page about the tiananmen square massacre for these requests?

    • jwz says:

      Because if I don't drop these requests on the floor at the packet level, my server falls over.

      • Seth says:

        I mean, serve that to the clients your block is not matching (the dull roar). I doubt you need to serve it up that many times for them to notice.

        • jwz says:

          WTF? If I could reliably identify those I'd just block them too!

          • Seth says:

            I thought you could reliably detect them at the HTTP layer by the Host header. It's just that it's expensive. So you block as much as you can at the IP layer, and deal with the rest via HTTP (serve the "bad" page to people with the wrong Host header).

  4. Mark Welch says:

    I wonder if putting up Falun Gong material on your site would make a difference.

  5. Mark Welch says:

    I wonder if putting up Falun Gong material on your site would make a difference.

  6. The Nutcase says:

    Sorry for the off-topic but I'm sure others would love your feedback, but what do you think of this?

  7. CJ says:

    It appears you're being targeted intentionally. I'm mystified as to why.

  8. axle says:

    What distro/kernel version are you running? Does any of the commands iptables -m string -h and iptables -m set -h result in a "No such file or directory" message? Do you have the program ipset installed? My approach would be to put all known Chinese subnets into an ipset and add stray ones dynamically to a second one by matching against a whitelist of Host: values at the packet level. Then you can test whether an incoming connection is present in any of the ipsets and block it on the first packet. Here's what I came up with:

    ipset create china_static hash:net family inet hashsize 1024 maxelem 65536 timeout 0
    ipset create china_dynamic hash:ip family inet hashsize 1024 maxelem 65536 netmask 24 timeout 86400
    ipset create china list:set size 8 timeout 0
    ipset add china china_static
    ipset add china china_dynamic
    # this command requires Bash
    source <(sed 's@^@ ipset add china_static @' china_netaddrs.txt)
    # china_netaddrs.txt has entries pasted from wizcrafts.net

    iptables -I INPUT 1 -i eth0 -p tcp --tcp-flags PSH,ACK PSH,ACK --dport 80:443 \
    -m string --to 1000 --algo bm \! --string 'Host: www.dnalounge.com' \
    -m string --to 1000 --algo bm \! --string 'Host: dnalounge.com' \
    -m string --to 1000 --algo bm \! --string 'Host: www.dnapizza.com' \
    -m string --to 1000 --algo bm \! --string 'Host: dnapizza.com' \
    -m string --to 1000 --algo bm \! --string 'Host: cerebrum.dnalounge.com' \
    -j SET --add-set china_dynamic src --exist
    iptables -I INPUT 1 -i eth0 -p tcp --dport 80:443 -m set --match-set china src -j REJECT

    This should block all hostnames not explicitly listed. I've only tested it on my loopback device and not very thoroughly, so I don't know how it will perform. Maybe someone more knowledgeable has ideas for improving it. Depending on just how many entries end up in the ipsets you might have to increase hashsize and maxelem (I couldn't find maximum values for them) or even resort to nested ipsets. Running ipset list china_dynamic gives you all its entries and their timeout, use ipset save/restore $SETNAME to save/restore using files.

    Note: The iptables command with the hostnames relies on the assumption that the packet containing the 'Host:' header is also the one having the PSH and ACK flags set. To me this appears to be a sound assumption, but might turn out to be a source of error (omit --tcp-flags PSH,ACK PSH,ACK in that case). The part --dport 80:443 assumes that you run no other services in that port range as they'll be blocked, too. If you do, you'll have to split the commands in two for each port individually or omit :443 if HTTPS is unaffected. china_dynamic takes entire /24 subnets, grow/shrink its netmask value if you need to. Also note that the timeout values for the entries in china_dynamic don't get reset since they're filtered by the REJECT target. Switch the order of the iptables commands if you want them to be reset to 24 hours every time they try to connect.

    I hope it helps.

    • yuubi says:

      I don't recall the part of the HTTP spec that requires the `Host` header be unfragmented. Just a GET of a content page with firefox is around 660 bytes; no idea what any cookies for any users buying stuff, or any other headers from other browsers, might weigh.

    • axle says:

      I just noticed that the first iptables statement will block all HTTPS traffic since it can't match on encrypted TLS traffic. Change --dport 80:443 to --dport 80. On the second iptables statement it can remain.

  9. Jesse Mullan says:

    I believe that fail2ban could automate this process for you by scanning that errant log and auto blocking bad ips.

  10. What about simply changing your static IP address? I would imagine that your provider could accomodate that. Or obtaining a second and temporarily switching things over to use that, leaving the original unroutable. It would no doubt be a bit of a pain, but surely less so than this.

    Or if the DNS poisoning is making use of your real DNS records in your domain, using Amazon Route 53 to handle your DNS with their Geolocation Routing could potentially help. jwz.org could resolve to some Chinese government IP for requests based in China, or even 127.0.0.1.

  11. Kit Sunde says:

    Permanent redirect somewhere appropriate like Beijing smog statistics or anything wildly offensive like gaotse would both be more appropriate and let you drain the flow of people are re-visiting.

    • Nate says:

      Isn't this traffic coming from Bittorrent clients? You know, that strange kind of thing on the Internets that is not a browser?

  12. Shrubs Ho says:

    Much as I enjoy your site, it's not bleeding edge revolutionary: block the country.

  13. Scott says:

    I just block every class A assigned to APNIC... really cuts down the problems.