We at the exo.performance.network pride ourselves on having relatively "thick skins." After all, when you have the audacity to actually take a position on the issues - and then to back them up with hard data - you're bound to bring out the zealots on the other side. However, it's a rare treat when the shots sent our way hail from no less an industry authority than Microsoft.
Yes, it seems our little foray into Windows benchmarking has drawn the ire of Mr. Nick White, official "spokesmouth" for the Windows Vista team. In his blog/rant, Mr. White blabbers on a bit about unrealistic benchmarking, then proceeds to badmouth our OfficeBench test script by calling it nothing more than a "window-open, window-close" routine. He even includes a ridiculously accelerated video of our test script in action, using the "speed" at which it executes as an excuse for discrediting its validity.
Of course he's wrong, both in his description of the test script and his justification for discrediting it. However, since his agenda likely had little to do with actually clearing the air - and was more in line with a classic Microsoft "hit" piece ordered from on high - we won't bother responding to Mr. White directly. However, we will take a moment to explain how OfficeBench works and why it's garnered so much respect over the years:
- OfficeBench's origins can be traced to our sister company, Competitive Systems Analysis, Inc. It was designed by Mr. Randall C. Kennedy while his company was under contract to Intel's Desktop Architecture Labs (DAL). This was way back in the 1999/2000 timeframe, when CSA was responsible for a great deal of internal benchmarking and white paper development surrounding the Pentium III and Pentium 4 CPU launches, among other projects.
Why is this important? Because it helps to establish the origins of the technology.
- The OfficeBench test script was designed from the beginning to be a "run anywhere" benchmark. By "run anywhere" we mean that the script will execute reliably under almost any Windows runtime environment. At the time it was being developed, this meant Windows 2000 and Terminal Server. As Windows evolved, so did OfficeBench, until now it supports every version of Windows since 2000, including XP, Vista, Server 2003, Server 2008, all flavors of Terminal Server and all known application and desktop virtualization environments. When we say "anywhere," we really mean it.
Why is this important? Because it allows us to test across multiple generations of Windows.
- OfficeBench is also version independent. That is, it's designed to work with any version of Microsoft Office. When it was originally conceived, the state of the art was Office 2000. Since then, Microsoft has rolled out 3 additional versions: XP, 2003 and, most recently, 2007. Once again, OfficeBench runs unmodified across all four versions of Office. Combine this with the support for the various Windows platform releases and you begin to see why OfficeBench is so powerful: It is the only test script of its kind that allows you to compare performance across multiple generations of Windows and Office. Mix them, match them - it just works.
Why is this important? Because it allows us to test across multiple generations of Office.
- Mr. White's hatchet job aside, OfficeBench is in fact a fairly complex test script. For starters, it uses OLE automation to drive the applications. This is different from most test scripts, which use window messages or key stroke/mouse click simulation. Using OLE automation has numerous benefits, including allowing us to run unmodified across Office versions. It also factors out any input-related anomalies while eliminating the chance that a UI change or 3rd party modification will somehow break the script.
As for Mr. White's assertion that it's a simple "window-open, window-close" script, I offer the following summary of key OfficeBench tasks:
a. Reformat all section headers and subheads in Word.
b. Generate multiple chart objects in Excel.
c. Generate complete multi-slide presentation in PowerPoint.
d. Multi-page scroll w/copy paste of chart objects into Word.
e. Slide sort/apply multiple templates in PowerPoint.
f. Multi-page scroll/print preview/print-to-file in Word.
g. Multi-chart print preview/print-to-file in Excel
h. Global search/replace in word (multiple).
i. Multi-slide preview/print-to-file in PowerPoint
j. Navigate simulated research web site in IE (multiple).
Again, the above are just some highlights. There's a lot more going on than meets the eye, and the key is that it's the exact same set of tasks executing across all versions of Office.
Why is this important? Because it shows that OfficeBench is a sophisticated test script that does more than merely "open and close windows."
- OfficeBench doesn't exist in a vacuum. It's delivered as part of a sophisticated, extensible, multi-process testing framework we call DMS Clarity Studio. With DMS Clarity Studio, we provide a variety of scalable workload objects for testing everything from client/server database connections to MAPI-based message store access to streaming multimedia. OfficeBench has been engineered to run in parallel with these workloads, providing for a rich variety of targeted test scenarios spanning the range of Windows client and server platforms. It's all coordinated through the DMS Clarity Studio framework and also seamlessly integrated with the exo.performance.network's Clarity Analysis Portal.
Note: DMS Clarity Studio is offered for free as part of the exo.performance.network. It's also part of the larger DMS Clarity Suite framework in use across thousands of trading workstations and other mission critical systems in the financial services sector. Some of the largest trading firms in the world trust us - Devil Mountain Software, Inc. - to tell them when their systems are under-performing. Enough said.
Why is this important? Because it shows that OfficeBench is part of a proven testing ecosystem that spans the range of Windows platforms and runtime scenarios.
In summary, OfficeBench is much more than a simple "window-open, window-close" script. It is a sophisticated, version-independent benchmarking tool that executes reliably under almost any Windows runtime environment. As such, it is the only tool of its kind that allows IT organizations to accurately assess multi-generational performance across all versions of Windows and Office.
And that's why OfficeBench scares the hell of out Microsoft. For the first time ever the industry has the tools necessary to call the company to the mat for its bloated, CPU cycle-sucking ways.
Microsoft's response? Slam the benchmark! Try to discredit the author/source! And crank-up the FUD machine!
Sorry, guys! You can run, but you can't hide, from OfficeBench.
Read more...
Posts
