[linux-elitists] web server software for tarpitting?

[Gerald Oskoboiny]

The other day we posted an article [1] about excessive traffic
for DTD files on www.w3.org: up to 130 million requests/day, with
some IP addresses re-requesting the same files thousands of times
per day. (up to 300k times/day, rarely)

The article goes into more details for those interested, but the
solution I'm thinking will work best (suggested by Don Marti
among others) is to tarpit the offenders.

I just followed up on slashdot [2] about the implementation I
have in mind, but that thread is pretty stale and this is
probably a better place to ask anyway, so:

Does anyone have specific web server software to recommend that
is able to keep tens of thousands of concurrent connections open
on a typical cheap Linux box? (Lighttpd? Nginx? Varnish? Yaws?)
It also needs to be able to proxy other requests to an Apache
server running elsewhere.

Bonus marks for:

   - ability to do content negotiation
   - ability to set different delays for different IP addresses
   - HTTP compliance

I'll research this myself as well, I'm just wondering if anyone
has recommendations based on experience.

thanks!

[1] http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic

[2] http://developers.slashdot.org/comments.pl?sid=447350&cid=22376594
     which says:

     The implementation I'm thinking might work well is:

     Switch www.w3.org to use some lightweight server software
     that is able to keep lots of connections open, and configure
     it to serve DTD files with an artificial 5 second delay.
     Proxy all the other requests to our existing Apache server
     running elsewhere (possibly on another port on the same
     system)

     Most people shouldn't notice or care about the delay for DTD
     files, only the apps that are requesting them hundreds or
     thousands of times in a row will notice.

     W3C's current traffic is something like:

       - 66% DTD/schema files (.dtd/ent/mod/xsd)
       - 25% valid HTML/CSS/WAI icons
       - 9% other

     So we'd probably want to configure the lightweight server to
     serve those icons too (but then it would have to do conneg as
     well)

(that's not really tarpitting, but has similar requirements)

-- 
Gerald Oskoboiny <gerald@impressive.net>
http://impressive.net/people/gerald/