[linux-elitists] Re: FW: MS claims w2k is better than solaris 8 (fwd)

Jay Sulzberger jays@panix.com
Fri Feb 18 10:06:15 PST 2000


On Fri, 18 Feb 2000, John Goebel wrote:

> On Fri, Feb 18, 2000 at 12:24:39PM -0500, Jay Sulzberger wrote:
> > 
> > I think that to some degree the free software propaganda effort has fallen
> > down on the benchmark front.  We should state clearly that in the past all
> > these claims have been shown to be false.  Since we are satisfied that
> > Microsoft lies, we are only willing to discuss their claims after they pay
> > us to run the benchmarks ourselves, without their being in the room.
> > 
> 
> Jay,
> 
> Benchmarks are a game. Even the "objective" benchmarks like SPEC are a
> mafia. If you want to benchmark an application, that's fine, but
> general FLOPS and crap like that are either a no-duh (alpha fpu vs.
> PIII), or radically cooked.
> 
> If my application needs to perform within a certain time, or produce
> so many FLOPS (and I have the application, not linpack et al), then
> cool, that will be fruitful.
> 
> Too much hay is made out of benchmarks. Real applications running on
> systems, that's reasonable. SPEC, linpack, top 500 lists are just
> bragging rights.
> 
> Sorry about the soapbox-ing. 
> 
> Thanks,
> John 

I agree with this taken as a specific criticism of the failure of
benchmarks as used today in the computer industry.  And of course
Microsoft's recent efforts are as nothing compared to the great benchmark
frauds of the past.  I think here of the compilers which would take note
that a benchmark was being run, and output the hand coded highly optimized
object file.

But, as you point out, some benchmarks might be useful for some purposes.

And the only way even these can be useful is to have them run in public,
with fully specified hardware and software.

It is a question as to how much effort to put in on stuff like this.

I think not much, but we could occasionally whip up something, and
naturally invite Microsoft, Sun, Apple, etc., to take part.

In a circus atmosphere, of course.

The Functional Programming Contest has a special prize which will usually
be awarded for Most Ingenious Cheat.

This year the Microsoft Team came in second in the regular prize list.  
This team publishes a fine compiler under the BSD license.

http://www.ai.mit.edu/extra/icfp-contest
http://www.cs.virginia.edu/~jks6b/icfp
http://www.haskell.org/ghc

oo--JS.





More information about the linux-elitists mailing list