Site Presented By
Tuesday, Nov 24, 2009
The test lab looked for help online. But did it really want answers?
Apr 27, 1999 | On April 13, a test lab named Mindcraft released a report commissioned by Microsoft that declared Windows NT to be far superior in performance to Linux. NT, trumpeted the study from Mindcraft, an "independent testing lab," is "2.5 times faster than Linux as a file server and 3.7 times faster as a Web server." Numbers like that can get Linux lovers hopping mad. But what really has the Linux community incensed is the way Mindcraft apparently went about seeking information and help from Linux experts.
Sure, Mindcraft -- either itself or through intermediaries -- did ask for help in an Internet-based discussion forum. But it did so under false pretences: It didn't tell its would-be helpers that it needed the information because it was comparing the performance of Windows NT and Linux at the commercial request of Microsoft. And its request for advice originated from a computer with a Microsoft address. It refused to answer any requests for more information about its test. Finally, it apparently promised to redo the test with the input of Linus Torvalds himself -- but isn't giving him the info he needs to do the job right.
The original news of Mindcraft's test sent the Linux world into immediate convulsions -- the sound of open-source geeks gnashing their teeth could be heard all over the Internet. Were all those claims for Linux's technical superiority just so much overheated hooey?
Not so fast, said the Net. As demonstrated in a recent report by Linux front man Eric Raymond, if there is one thing that the Internet is really, really good at, it is marshaling the combined forces of thousands and thousands of watchful eyes.
In combination with some aggressive trade press reporting, numerous details undermining the Mindcraft report soon emerged. First, the study had been commissioned by Microsoft -- a fact not mentioned in the original press release, and buried deep within the Mindcraft Web site. Second, the test had been able to take advantage of extensive "tuning" information from Microsoft -- details on how to make NT work especially well on the particular hardware configuration used by Mindcraft. And third, Mindcraft's attempts to obtain similar tuning information from the Linux community were half-hearted, at best -- and here is where the story gets interesting.
The Linux community swears by its accessibility, by its willingness to help answer questions in a variety of forums. But it isn't perfect. The convoluted story of Mindcraft's attempt to get information from one of the most popular such forums -- Usenet newsgroups -- illustrates a key Linux weakness, the lack of centralized sources of information on the newest and most advanced technical issues.
But at the same time, the story underlines the essential worthlessness of commercially sponsored comparison tests. The purpose of these tests is to please the customer who commissions them. Why expend too much energy attempting to find information that your customer probably won't appreciate?
