After a bit of logging and searching I found out that some Chinese ISP (probably CERNET according to the results of whatsmydns.net) and some Turkish ISP (probably TTNET) respond to dns queries such as a.tracker.thepiratebay.org with various IPs that have nothing to do with piratebay or torrents. In other words they seem to do some kind of DNS Cache Poisoning for some bizarre reason.
So hundreds (if not thousands) of bittorrent clients on those countries make tons of 'announces' to my webservers which result pretty much in a DDoS attack filling up all Apache's connections.
So basically, entire countries' worth of porn hounds randomly start hammering on my server all at once, even though no BitTorrent traffic has ever passed to or from the network it's on, because for some unknown reason, the now-long-defunct piratebay tracker sometimes resolves to my IP address. Hooray.
This (possibly) mitigates the problem somewhat, by returning an actual BitTorrent error message to the clients so they stop retrying sooner:
<Location ~ "^/announc">
ErrorDocument 404 "d14:failure reason13:not a tracker8:retry in5:nevere"
</Location>
We had previously blackholed the entire Chinese IP space because of this, but we missed some.
Hi, how's your day going?
Update: No, I don't think that ErrorDocument helps. I see the same BitTorrent clients hitting me every couple minutes regardless. (At first I thought it might have been a problem that literal ErrorDocuments like that send as text/html, but I changed it to text/plain and that didn't change anything.) There are still more subnets to killfile, I guess...
Update 2: Some suggest that returning a 410 instead of a 404 to BitTorrent clients may be more effective. I have not verified this myself, but to do that, do this:
RewriteEngine On
RewriteRule ^/announc - [G]
<Location ~ "^/announc">
ErrorDocument 410 "d14:failure reason13:not a tracker8:retry in5:nevere"
</Location>
You're welcome. That sucked.
You're welcome. That sucked.
For a long time I felt bad for blackholing the entire CN IP space for interminable bot traffic and comment spam, so this makes me feel better. Thanks!
Another writeup on the Great Firehose of China.
If this is that CYBER-WAR thing we've been hearing about for so long, it's way worse than I imagined! This inconvenienced me for hours!
This one was also interesting.
http://blog.devops.co.il/post/108740168304/torrent-ddos-attack
On the whole, I'd rather prefer that a team of prepsychotic special-forces hackers descend upon my by ultralights dropped from orbit.
jack in now, Case
https://news.ycombinator.com/item?id=8931827
I like the idea of replying to invalid requests with something about Tiananmen. It should get you blocked from the Chinese end quite quickly.
Of no use in this case of course.
I much preferred the idea of responding with
HTTP/1.1 301 Moved PermanentlyLocation: http://www.gov.cn/
https://news.ycombinator.com/item?id=8931827
You could automate your CN block by using a geoIP database updated on a regular basis.
And I forgot to post the link. Need moar coffee. http://dev.maxmind.com/geoip/legacy/geolite/
I'm a Chinese , and feel sorry for all these and shame on my gov.
We have had very good success with:
(Note the 410, as recommended in http://engineering.bittorrent.com/2015/01/29/a-note-on-the-ddos-attacks/, and actually triggering a 410 to make the ErrorDocument take effect.)
That is wrong wrong wrong, and I wrote that guy to tell him that and he still hasn't updated the wrong wrong wrong things he is telling people -- and crediting to me.
"ErrorDocument 410" does not mean "send a 410 for this document."
It means "were you already in the mind of sending a 410, this is the error message you should send along with it."
So if that's what you did, you're still sending a 404 with your same old default 404 document, and you don't know this because you didn't actually test it.
The RewriteRule [G] sends the 410.
Ah, so it does.
That's still not what the cited link tells you to do.
There are some interesting characteristics in traffic patters:
- list of affected IP addresses seem to be static
- the traffic only comes in certain hours to the affected IPs
- the affected IPs are typically hostings (i.e. no ADSL or otherwise home addresses)
- different IPs get different shares of traffic
- and many more!
I'm a security researcher writing an extended article about this.
I'd be interested to speak with people who are affected by this kind of "bittorrent DDoS". The magazine I'm writing the article for is willing to cover some of the costs related to this DDoS (your hosting cost, compensate for your time) if you help us track this attack better - please contact me at tchm at virtall dot com for details.