They downgraded 404 errors from "error" to "info" meaning they no longer show up in your error_log unless you crank up LogLevel to "useless firehose".This means that all of my nice fail2ban honeypot rules that blackhole your IP address if you probe /phpMyAdmin/ and stuff like that no longer trigger.
"Thaaaaanks."
Super great. In the mean time, until a better work-around is produced, I guess you could make a CustomLog directive to pipe your 404s to their own log file, and then point fail2ban at that.
Wow, they changed the behavior after hemming and hawing about this for fifteen years! And after closing and reopening it twice, they didn't even make it an administrator option? And they did it on an early minor update? Way to fuck up people's workflow!
Have you used Apache before? Their version numbering seems to mostly be a cruel joke designed to imply safety and consistency, but instead means "Put whatever the hell we feel like into this minor release. Enjoy."
I have, and you know what? Your comment explains a lot.
After they changed how they did virtual hosts... I switched to lighttpd. Now I'm glad I switched. Looks like they're methodically nuking their own product.
I believe you can set your own 404 error handler script that could log according to your specifications, but as you are clearly stating, you shouldn't have to.
As the person who reported the original issue...
If you want to take action on 404 or 400 or whatever else you want to key off of, use your access_log. It even gives you the requested URI instead of a potentially mangled filesystem path. If you were keying off of error_log, that was a bug in your handling.
The problem with putting 404s in error_log is that it turned THAT into an unmanaged firehose by default, dumping bad client requests into the mix with actual server errors. Most configurations manage their access_log outputs with rotation, indexing, what have you.
By contrast, error_log is, by default, a place to show SERVER errors needing administrative attention (not client malformed or improper requests).
And if you are worried about fail2ban taking in too much churn by having to parse all the non 404 entries in error_log...
Yeah, that's EXACTLY the reason you can create CustomLog outputs tailored to specific status values. That ability has also been around a very long time.
So because of some dumb argument about ideological purity, I should "just" rewrite a whole bunch of scripts, many of which I didn't even write in the first place, to work completely differently.
"Thaaaaanks."
This is as asinine as when Apache completely changed how "allow" and "deny" worked in a non-backward-compatible way.
"Thaaaaanks."
The thing that I, and many other people, had been relying on that was working just fine for our needs, was a "bug" and we were wrong all along and now you've "helped" us see the error of our ways.
"FFFFfffffffffuuuuuck you."
The reason this was a major problem is because it was flooding the error log of data center and large-scale installations with attempts by script kiddies to find vulnerabilities, drowning out all of the information about real server errors that needed intervention.
You can call this some sort of ideological argument if you want, but that doesn't change the fact that even Apache's predecessor, NCSA httpd, DEFINED the access log as the place to go for client request information. It's even in a format tailor made for this kind of parsing. Ignoring that fact does make your "ideological" argument here a bit silly.
Your misinterpretation of buggy behavior of the error log doesn't make it any less of a bug.
Let me be clear. It's not like I'm saying you're a bad geek or made some sort of fatal mistake. You simply had a conceptual bug in your original automation: using a non machine parseable, arbitrary output data source in a machine parser. Computing was founded on hacks like this, but they're still hacks, and sometimes hacks have to be updated to keep up with the times.
It's not like I haven't done the same thing hundreds of times over the years. I'm fully willing to admit when I make a mistake like that, dust myself off, and fix my own bug to use the tool or data that's designed to address my use case. So you had a bug; who cares?
The Apache error log is a free-form output that has no specific guarantee of any data in it. Messages can change, be added, or be removed arbitrarily without consulting anyone. The access log has been there since the dawn of web servers, and has had machine parsable format this entire time. Use it and move on. Another personal bug squashed.
Oh, fuck off. You changed a file format that other programs were relying on for NO REASON. Yes, it's all a house of cards, but you're the one running around with the leaf-blower and then saying, "Nyaah nyaah that's what you get."
You literally just said, "We at Apache only give a shit about massive data centers and huge, drupal-requiring websites. We don't care about backward compatibility, we don't care about any website run by a company with fewer than two hundred people, and we will happily mash together a change of opinion that's decades old with crucial security updates. We are God and you are a worm."
Why are these massive data centers reading disjoint lines of free-form text and being sad that it's slow and sometimes confusing? Why is it we have to rewrite our CustomLog rules, and the corporate overlords don't? Why is it our fault for using the error log for a certain reason (reading output about the server with a machine parser), but not their fault for doing the exact same thing? (Don't try to convince me that massive data centers don't have a script monitoring error_log just like everyone else, or that reading error_log from a script is unreasonable and should not be relied upon.)
"file format"...
error.log is not a file format. It's free-form, not intended to be machine-parseable, and no one is under any obligation to make it somehow consistent for your hack of a use case.
If that's the holier-than-thou way you want to play it, fine, but the rest of the world has known, since the early 1990s, the correct place to get client request data from in a ready-to-parse format: the access log.
Oh, and you should also already know that Apache 2.4.1 was a feature release, not a security release.
Feature releases can change plenty of functionality, including error messages that are not intended for machine parsing.
I gave you the nice, level-headed explanation above, but if you want to keep thinking that the fail2ban rules weren't the bug here, I have a bridge for sale too.
"An Apache proceeds to 'well, actually' the Internet on the dictionary definition of words."
Both sides of this argument have merit and the solution is easy: provide a backwards compatibility capability. My software vendor employer faces the same problem all the time. We find a bug which when fixed will change behavior. Everyone who reviews the proposed change agrees that the current behavior is definitely buggy and should change by default. But somewhere out there there’s a user who depends on the buggy behavior. Maybe they learned to exploit the bug. Maybe they just don’t want to tell their boss that they’ve been using bad data. Who knows?
Solution: fix the default behavior and provide a simple switch to revert to the legacy behavior.
Solution: leave the default behavior unchanged and provide a simple switch to enable the new behavior.
And let the bugs accumulate forever? That could work if there was a way to develop code without releasing a bug.
OMG this one slippery slope digested my universe! You monster!
Have you tried testing?
Yeah, agreed. I mean, you have to get rid of old bugs sometimes to make room for new ones.
I assume you meant more testing Lloyd. Yeah that will reduce the number of released bugs. But no amount of testing will catch bugs in the spec as this Apache situation seems to be.
404 is literally defined as an error.
jwz didn't write fail2ban, and he doesn't maintain it.
What are you on about, you monster?
I hesitate to point out you're using a webserver that lampoons an entire First Nation tribe to make a weak joke about how riddled it is with bugs. Knock me down with a, well, a feather which wasn't theirs to begin with.
(I hope, jwz, you understand from our prior interactions that I'm not posting the following in any attempt to tell you what to do or how to do it.)
Does fail2ban grok nginx's logs? (If not, the rest of this is meaningless.)
I've had pretty good luck professionally (a couple different places) presenting flaky web services through an nginx proxy (in my case, to force sane SSL handling or just to sanitize inputs where they don't provide good ways to deal). One of those applications was not WordPress, but from what I can recall of it, it "should" work…
But: as with several items above, this is in no way a thing where you should have to go change config as the result of a software upgrade. Boo, Apache, boo.
(When I get around to moving eclipsed.net around—mostly when my current VPS hosting cost finally gets my financial attention—I'm planning to just ditch Apache entirely for nginx, but everything I serve is flat files, which is TURBO not true for your use case.)
+1 I use fail2ban + nginx. I moved my (mostly aws [centos/ubuntu/aws-linux]) sites to nginx as well as all removed/replaced apache with nginx on my macs and it was like a huge weight lifted off my shoulders. All the same crap listed above disappeared plus I didn't have to re-write, re-remember all sorts of arcane httpd server crap that didn't have shit-all to do with providing me value.
I'm not suggesting a switch because it's not my first day on this site but the grass is certainly greener over here. @Todd Verling's response perfectly illustrates why - you are not their customer.
((It's interesting now that I think about it - this is exactly "The Innovator's Dilemma" in web-servers. The big-shot is so busy worrying about its big clients that it feels it can shit all over^H^H^H^H^H dismiss the small guys. Meanwhile, its competitors (nginx, lighttpd) are like "You need a simple solution? Try us!" See how well that strategy worked with Detroit or US Steel.))
That all sounds really promising for what I was planning to do!
But, just to be clear: for use cases similar to jwz's, I was only suggesting inserting nginx as a reverse proxy. Maybe WordPress plays ball with nginx now? But even if it does, that'd mean touching a lot of moving parts that just inserting it in between doesn't. I think?
I can tell you more about WP + nginx sometime soon, when I get around to porting my `ghost` blog to `WP`. Ghost craps the bed even worse than apache on upgrade day. The WP landing page says they recommend apache or nginx so take that for what it's worth.
I mean, like I said:
(I'm thinking mostly of migrating a bunch of Apache config over to nginx, as opposed to telling Apache to only listen to requests from localhost on an undisclosed port number.)
Have you considered directing this rant at the fail2ban authors, who had the idea of parsing error_log with regexen in the first place, rather than Apache's authors for daring to change anything at all? You sound like the overheating spacebar emacs guy.
Here, I did it for you, you lazy fuck.
How dare they gather data from the only place that it was available.
It was available in the access log, designed for machine parsing.
awk '$9 == 404 {print $7}' access_logInstead, the fail2ban scripters looked for it in the error log, whose human readable text changes over time so they put that into the regex... and now the humans have decided they're sick of having to write
grep -v faviconto read the error log, so they took that spam right out.fail2ban already reads the entire access log to look for suspicious user agent names, so I'm not sure why they can't also use it to read dodgy URLs.
Kyzer, I think you're misinterpretting that xkcd strip. The joke there isn't on spacebar guy (although he is an extreme example on purpose): the joke is that software updates always break things, but then also: maybe they don't have to?
Certainly, now that Apache is providing an API of sorts to collect errors like this, fail2ban should probably go use that API. It's just switching off the old behavior without any clear way to revert it ("No, I really do want 404s to still be logged at ERROR, and I don't want to have to get all the INFO stuff too") that's shitty.
(Given Sergey's response to your Github comment was mostly shrugging, initially, I pointed him at Todd's explanation above, which I think does explain what Apache thinks is the Right Way to get these log entries now.)
There was a bug report and maybe someone else can find it – I can't. IIRC it was a Solaris or HPUX bug that said something like "If I bang on the keyboard like a drunken monkey then something bad happens..." The response was more or less: "Problem discovered. Will be fixed next release. Workaround: Do not bang on keyboard like a drunken monkey until patched."
It would have been easy to dismiss the problem as an obvious: "Only an idiot would bang on the keyboard like a drunken monkey and not expect something bad to happen. Case closed." but instead they took ownership of the issue, discovered a problem, and fixed it.
Everyone came out ahead. The product was better, the customer was happy, nobody blamed anyone for using the product in an unexpected manner, and nobody was called a 'lazy fuck.'
Comments like this make me wish jwz's implementation of WordPress had a (generally stupid) "like" or "heart" button like the Book of Faces and Twitter. :^>
But also: I seem to recall that jwz and Kyzer have an established relationship, and I read that as friendly ribbing, rather than hateful derision.
I'm here primarily for the unbridled snark so I'm good with that. However, after a career in tech customer service I cringe when I read comments like Todd Vierling's – "You're too stupid to use our product correctly and too insignificant for us to care."
Yeah, you'll note that I don't really express an opinion on Todd in public.
FYI, it was Solaris. I no longer have a SunSolve login (or whatever Oracle has renamed it to this week), but IIRC, the first bug was "dtlogin crashes if you bang on the keyboard like a wild monkey." The workaround was as expected and whatever X library was causing it got patched, as stated above.
I say first because there was inevitably a follow-up: "dtlogin crashes if you click on the mouse like a wild monkey." Similar workaround, additional patch ultimately provided by Sun.