Sunday, December 2, 2007

When Microsoft Attacks!

We at the exo.performance.network pride ourselves on having relatively "thick skins." After all, when you have the audacity to actually take a position on the issues - and then to back them up with hard data - you're bound to bring out the zealots on the other side. However, it's a rare treat when the shots sent our way hail from no less an industry authority than Microsoft.

Yes, it seems our little foray into Windows benchmarking has drawn the ire of Mr. Nick White, official "spokesmouth" for the Windows Vista team. In his blog/rant, Mr. White blabbers on a  bit about unrealistic benchmarking, then proceeds to badmouth our OfficeBench test script by calling it nothing more than a "window-open, window-close" routine. He even includes a ridiculously accelerated video of our test script in action, using the "speed" at which it executes as an excuse for discrediting its validity.

Of course he's wrong, both in his description of the test script and his justification for discrediting it. However, since his agenda likely had little to do with actually clearing the air - and was more in line with a classic Microsoft "hit" piece ordered from on high - we won't bother responding to Mr. White directly. However, we will take a moment to explain how OfficeBench works and why it's garnered so much respect over the years:

  1. OfficeBench's origins can be traced to our sister company, Competitive Systems Analysis, Inc. It was designed by Mr. Randall C. Kennedy while his company was under contract to Intel's Desktop Architecture Labs (DAL). This was way back in the 1999/2000 timeframe, when CSA was responsible for a great deal of internal benchmarking and white paper development surrounding the Pentium III and Pentium 4 CPU launches, among other projects.

    Why is this important? Because it helps to establish the origins of the technology.

  2. The OfficeBench test script was designed from the beginning to be a "run anywhere" benchmark. By "run anywhere" we mean that the script will execute reliably under almost any Windows runtime environment. At the time it was being developed, this meant Windows 2000 and Terminal Server. As Windows evolved, so did OfficeBench, until now it supports every version of Windows since 2000, including XP, Vista, Server 2003, Server 2008, all flavors of Terminal Server and all known application and desktop virtualization environments. When we say "anywhere," we really mean it.

    Why is this important? Because it allows us to test across multiple generations of Windows.

  3. OfficeBench is also version independent. That is, it's designed to work with any version of Microsoft Office. When it was originally conceived, the state of the art was Office 2000. Since then, Microsoft has rolled out 3 additional versions: XP, 2003 and, most recently, 2007. Once again, OfficeBench runs unmodified across all four versions of Office. Combine this with the support for the various Windows platform releases and you begin to see why OfficeBench is so powerful: It is the only test script of its kind that allows you to compare performance across multiple generations of Windows and Office. Mix them, match them - it just works.

    Why is this important? Because it allows us to test across multiple generations of Office.

  4. Mr. White's hatchet job aside, OfficeBench is in fact a fairly complex test script. For starters, it uses OLE automation to drive the applications. This is different from most test scripts, which use window messages or key stroke/mouse click simulation. Using OLE automation has numerous benefits, including allowing us to run unmodified across Office versions. It also factors out any input-related anomalies while eliminating the chance that a UI change or 3rd party modification will somehow break the script.

    As for Mr. White's assertion that it's a simple "window-open, window-close" script, I offer the following summary of key OfficeBench tasks:

    a. Reformat all section headers and subheads in Word.
    b. Generate multiple chart objects in Excel.
    c. Generate complete multi-slide presentation in PowerPoint.
    d. Multi-page scroll w/copy paste of chart objects into Word.
    e. Slide sort/apply multiple templates in PowerPoint.
    f. Multi-page scroll/print preview/print-to-file in Word.
    g. Multi-chart print preview/print-to-file in Excel
    h. Global search/replace in word (multiple).
    i. Multi-slide preview/print-to-file in PowerPoint
    j. Navigate simulated research web site in IE (multiple).

    Again, the above are just some highlights. There's a lot more going on than meets the eye, and the key is that it's the exact same set of tasks executing across all versions of Office.

    Why is this important? Because it shows that OfficeBench is a sophisticated test script that does more than merely "open and close windows."

  5. OfficeBench doesn't exist in a vacuum. It's delivered as part of a sophisticated, extensible, multi-process testing framework we call DMS Clarity Studio. With DMS Clarity Studio, we provide a variety of scalable workload objects for testing everything from client/server database connections to MAPI-based message store access to streaming multimedia. OfficeBench has been engineered to run in parallel with these workloads, providing for a rich variety of targeted test scenarios spanning the range of Windows client and server platforms. It's all coordinated through the DMS Clarity Studio framework and also seamlessly integrated with the exo.performance.network's Clarity Analysis Portal.

    Note: DMS Clarity Studio is offered for free as part of the exo.performance.network. It's also part of the larger DMS Clarity Suite framework in use across thousands of trading workstations and other mission critical systems in the financial services sector. Some of the largest trading firms in the world trust us - Devil Mountain Software, Inc. - to tell them when their systems are under-performing. Enough said.

    Why is this important? Because it shows that OfficeBench is part of a proven testing ecosystem that spans the range of Windows platforms and runtime scenarios.

In summary, OfficeBench is much more than a simple "window-open, window-close" script. It is a sophisticated, version-independent benchmarking tool that executes reliably under almost any Windows runtime environment. As such, it is the only tool of its kind that allows IT organizations to accurately assess multi-generational performance across all versions of Windows and Office.

And that's why OfficeBench scares the hell of out Microsoft. For the first time ever the industry has the tools necessary to call the company to the mat for its bloated, CPU cycle-sucking ways.

Microsoft's response? Slam the benchmark! Try to discredit the author/source! And crank-up the FUD machine!

Sorry, guys! You can run, but you can't hide, from OfficeBench.

20 comments:

Brian said...

You guys are doing a great job, keep up the good work. Don't let MS scare you! Continue exposing the bloat across the software industry. It is for their own good anyway. The new companies making fast, tiny software based online are lurking.

Watch out for the hardware manufacturers next, though. Without the continuing bloat, why does anyone need the latest, greatest hardware?

Anonymous said...

While your testing methodology seems to be centered around the Windows platform, the readers of your reports may also be interested in drawing wider conclusions which are reaching beyond bounds of this platform. This raises the following question:

Is your testing script applicable to, say, OpenOffice? Can the testing be conducted on Linux platform? If so, what are the results and how they compare to Microsoft platforms?

Best Regards,
/Chris

Anonymous said...

[[the only tool of its kind]]
Actually that's where half the problem lies.

carlcamera said...

I have some experience having performed benchmarking tests as a software engineering lead at Dell and there is no end to this argument as all benchmarks are imperfect in some sense.

One inherent problem with "run anywhere" for example is that it forces the benchmark to the lowest common denominator. Benefits in acceleration or feature enhancements of present hardware or software might not be exploited because it would not be backwards compatible.

In addition, when you benchmark Vista vs XP do you close the Vista Sidebar because it doesn't exist in XP? How do you test sleep/resume against Windows 2000 since it doesn't sleep? Do you turn off wifi-scanning in XP and Vista because that feature doesn't exist in Windows 98? Or do you allow these processes to run with full knowledge that they will impact the results? Regardless of your decision, you're biasing the results.

I would also suggest that your benchmark credibility is damaged when you describe your benchmark as a "tool to [expose Microsoft's] bloated cpu-sucking ways."

When I read that statement, the word objective in my mind is suddenly replaced by biased. You seem to have an agenda separate and distinct from assisting the I/T community.

Anonymous said...

Haha, a true corporate pet:

"I'm a Product Manager at Microsoft working on the Windows Vista launch team. I also work with key influencers in our user community. This means I get to do cool stuff, play with lots of electronic toys, travel the world, and blog about it at the same time. I know you're jealous ;)"

Traduction:

"I'm trying to tell that i'm almost like anyone, even like you who are reading this, to become identificated with me, and because i'm a good boy and play dead for my masters, i got a good ration of money and a good stroke in the head, then i shake my tail in approbation for my owner and go to my little house. I just need to slam anything that could annoy my masters, like hints or evidences that our product are crappy. They throw me toys to play if i do it good, is a cool cool cool pink life"

Anonymous said...

Amazing, how Microsoft people want to look, if they market so many marketing names like superfetch, readyboost and that mumbo jumbo that supposedly adds responsiveness, and is one of the reasons that users should condone the obscene amount of ram that it eats, when they want to blame a test because it doesn't gift dead time between inputs??
Hilarious!
It would be like amd/ati blaming testers of their actual hardware for their less than optimum performance!
If that is an employee of microsoft, i must say that they look now even more kiddie than before.

Anonymous said...

Carl
Why you are messing win98 ?! The dudes have very goo writed that their bench is supporting everything from Win2K to Vista!So the bench is running on NT cores.
Did you close the Vista side bar when you are using desktop app-s for example IE!??!And now what and DELL are not feeling good huh !? Cause they are shiping their products only with Vista ! And now that bench is hitting DELL`s marketing commercials "Get DELL be slower with our new NB(NoteBook) and MS WIN VISTA".Did you dudes actualy tryed to work on NB with Vista!?!?!It is a real suicide.It will be funny which will be the next company that rely on Vista and will come here to challenge the results!

Anonymous said...

How about giving proper details of the Office Bench methodology and what exactly it measures? A whitepaper with the details will do and I'm sure knowledgeable people can form their own opinions. Sorry, your explanations aren't detailed enough. You claim its not a high speed UI test, yet you mention at least some operations that are multi-page scroll/preview without mentioning at what speed those are done.


Instead of using colorful language to present your results in a sensational manner, if it had been done professionally with clear breakdown of which operations have slowed down by how much (after all by using OLE automation you can track each of them) then perhaps people would take your results more seriously.

You expect people to notice your results and yet you don't provide any details or information for someone to make an informed decision.

I'll let the readers here decide what that is pointing to.

Research Staff said...

Because then it would be *boring* and nobody would read us! :-)

As for OfficeBench, you can learn almost anything you need to know from the Online Help documentation at the DMS Clarity Analysis Portal site...

RS

Anonymous said...

Ah, yes, your test results should have been provided along with a proper white paper which explained the testing methodology, so that the Micro$oft hacks would have had clear indication from someone with their head in the real world where, exactly, the problems with Vistard are. Armed with complete information about the testing methodology, it would have helped the hacks to accelerate their preparation of the FUD storyline about why your test was worthless, and greatly helped them to refine their bogus explanations of same. As it is, they've been reduced to very messy interaction with the general public instead, in an attempt to try and save face and marketshare. Specifically, I'm referring to their manufacturing of evidence for use in a Microsoft employee's blog post, and resorting to anonymous critical postings in your own blog.

So shame on you Devil Mountain, for having so little regard for the plight of the Micro$oft hacks. If you had only done your part to assist them in disseminating their propaganda about the joys of The Micro$oft Way, it would have been much easier for them to keep this entire episode from getting traction in the press. Again, I must say, shame on you.

Anonymous said...

This isn't the only benchmark showing Vista to perform worse than XP. Vista has been slower in games as well. It also uses more memory for everything (GDI+ in Vista, one cause).

SP1 might fix some of the networking problems, but it looks like overall performance will remain slower than XP on identical hardware.

Anonymous said...

I must have missed the part in the Nick White's blog entry where he mentions Office Bench... even once.

I didn't see him "badmouthing" it, doing his job as "official spokesmouth" either.

You guys are hacks trying to make it look like you're under some sort of persecution from the EvilBigCorporation Microsoft.

Pathetic.

Your 15 minutes of fame in the IT press are over. Apparently even after all the articles based on your "study", people aren't finding OfficeBench all that compelling.

If that post in the linked blog by "rkennedy01" is legit, that's even more pathetic - spamming someone else's blog to pimp your software?

Anonymous said...

"the first time ever the industry has the tools necessary.."

you guys are new at this I see...

Anonymous said...

Thank you for this information. There is quite the bru-ha-ha regarding the new Principled Technology testing as well now on the Windows Vista blogs:

http://windowsvistablog.com/blogs/windowsvista/archive/2008/02/26/principled-technologies-tests-windows-vista-sp1.aspx

Anonymous said...

Problem is that your methods aren't disclosed enough to know if they are valid.

For example, you can't slap Vista and XP on identical hardware and do a benchmark in the same day and get meaningful results for many reasons.

Just the fact that my system runs faster with Vista SP1 with all features enabled than it did with XP with all bloat features disabled calls you into question.

If you publish the details of your test methodology, then we can assess whether you had problems understanding the issues involved and ended up running a naive and non-scientific test. Having an automated tool does not make you scientific.

For example: Did you defrag the systems? Did you apply updates? Did you give Vista time to complete content indexing and track worker habits to prime the smart cache? etc, etc.

Anonymous said...

Microsoft is basically an insecure system with many vulnerabilities.

Anonymous said...

A reason more not to use microsoft at all. Although I always wanted to run Linux, I got used to microsoft.

Ticket Point said...

Microsoft has in its systems always somewhere vulnerabilities.

Anonymous said...

no day without such messages I suppose...:-)

Anonymous said...

I think Linux systems generally better than those of Microsoft.