[linux-elitists] web server software for tarpitting?
Sun Feb 10 23:06:30 PST 2008
The other day we posted an article  about excessive traffic
for DTD files on www.w3.org: up to 130 million requests/day, with
some IP addresses re-requesting the same files thousands of times
per day. (up to 300k times/day, rarely)
The article goes into more details for those interested, but the
solution I'm thinking will work best (suggested by Don Marti
among others) is to tarpit the offenders.
I just followed up on slashdot  about the implementation I
have in mind, but that thread is pretty stale and this is
probably a better place to ask anyway, so:
Does anyone have specific web server software to recommend that
is able to keep tens of thousands of concurrent connections open
on a typical cheap Linux box? (Lighttpd? Nginx? Varnish? Yaws?)
It also needs to be able to proxy other requests to an Apache
server running elsewhere.
Bonus marks for:
- ability to do content negotiation
- ability to set different delays for different IP addresses
- HTTP compliance
I'll research this myself as well, I'm just wondering if anyone
has recommendations based on experience.
The implementation I'm thinking might work well is:
Switch www.w3.org to use some lightweight server software
that is able to keep lots of connections open, and configure
it to serve DTD files with an artificial 5 second delay.
Proxy all the other requests to our existing Apache server
running elsewhere (possibly on another port on the same
Most people shouldn't notice or care about the delay for DTD
files, only the apps that are requesting them hundreds or
thousands of times in a row will notice.
W3C's current traffic is something like:
- 66% DTD/schema files (.dtd/ent/mod/xsd)
- 25% valid HTML/CSS/WAI icons
- 9% other
So we'd probably want to configure the lightweight server to
serve those icons too (but then it would have to do conneg as
(that's not really tarpitting, but has similar requirements)
Gerald Oskoboiny <email@example.com>
More information about the linux-elitists