Windows "Workstation" 2008 - Vista Done Right?

If you've been paying attention to the various industry news outlets you've no doubt come across the story about the Microsoft engineer advocating Windows Server 2008 as a "workstation" OS. According to him, if you make the right tweaks - installing the Desktop Experience feature, adding a few missing utilities, tuning the scheduler - you can turn Server 2008 into a fairly convincing Vista knock-off, one that's faster and more scalable than the original.

Curious, we decided to see for ourselves just how well Server 2008 stacks-up to Vista with Service Pack 1. To make the comparison as even as possible, we disabled all of the UI goodies on Vista (i.e. set the Visual Effects to "Adjust for Best Performance") and installed the Desktop Experience feature under Server 2008. We also enabled SuperFetch and the Indexing services on the Server 2008 installation (both are disabled by default) and adjusted the "Processor scheduling" option to favor Programs (i.e. the way it's set under Windows Vista).

For hardware, we reused our Dell XPS M1710 test bed (Core 2 Duo T7200 at 2GHz, 2GB RAM, 80GB 7200RPM disk) from our previous Vista testing projects. Both OS were configured to use the entire disk as a single partition, and we installed the same device drivers under each version.

The actual test scenarios involved a straight execution of the OfficeBench test script (in a 10-iteration loop) as well as a separate multi-process workload package featuring the ADO, MAPI and WMP Stress workload generation objects (executing continuously for 10 minutes in a 3x3x3 multi-instance configuration).

Given all the press surrounding Vista Service Pack 1 and the supposed parity of the SP1 and Server 2008 kernels, we were expecting to find little or no performance delta between the two platforms. So we were understandably surprised when repeated test runs showed Windows Server 2008 outperforming Windows Vista w/SP1 by a margin of 11-17%.

Clearly, there is more going on within Server 2008 than simply a few boot-time kernel switches. The very tangible performance disparity between our "Workstation" 2008 configuration and Vista, even with Service Pack 1 installed, shows that Microsoft is capable of squeezing more out of the shared "Windows 6.1" code base if/when they choose to do so.

As for what's dragging Vista down (the number of running processes and services was nearly identical across both OS and in each test scenario), that's a bit harder to define. Perhaps the Server 2008 team decided to eschew some of the more desktop-centric and/or consumer-focused (i.e. CPU cycle-sapping) features of the Vista core (DRM comes to mind). Regardless, now that we know how much better things could have been, it'll be that much harder to settle for the sluggishness and bloat of Windows Vista.

Our recommendation: If you have an MSDN account or otherwise have access to a Server 2008 license, check it out for yourself. You may find that Windows "Workstation" 2008 is the Windows Vista you've been waiting for all along.

Read more...

The Widgets are Coming!

Now for something totally different: Widgets!

What are widgets? Generically speaking, they're small mini-applets that run on your desktop or in a dashboard-type page or view. In the case of the exo.performance.network, widgets are small slices of functionality that have been broken-out from the main Analysis Portal site and presented as a stand-alone UI for a specific data type.

For example, we're working on a Systems Health widget that uses the Portal site's Report Card engine to generate daily and weekly summary charts of a system's overall CPU, Memory and I/O health. Here's a screenshot of the widget in action:

Figure 1 - The Systems Health Widget

We've also built a Process Monitoring widget that lets you keep track of collected metrics data for a specific process on a specific system. Here's a quick look at the widget monitoring SQL Server CPU utilization over a one week period:

Figure 2 - The Process Monitor Widget

Our goal is to produce a whole library of similarly styled widgets, each with a specific function that exposes part of the Portal's tools set. We'll likely offer these together as a kind of "roll-your-own-dashboard" kit where you can point and click your way to the ideal combination of charts and gauges.

And as always, we welcome your feedback. Tell us what you'd like to see in an "exo.widget" library and we'll see what we can cook-up together! :-)

Read more...

New Server (Yippee!)

We're pleased to announce that we've completed our move to the new server. The exo.performance.network is now hosted on a state of the art PowerEdge 2950 with 8x 2.66Ghz Xeon CPU Cores (2x E5430), 8GB of RAM and 1.2TB of storage in a six-disk RAID. It's a nice step-up from our previous "Prestonia" Xeon dinosaur and should give us the capacity to reach our goal of 10K users by summer.

Also, we switched co-location facilities. We dumped the poorly managed DigiPort Miami (don't go there!) in favor of a swanky new suite at Host.net in Boca. It's closer to our offices in Wellington and a truly high-class operation. We'll sleep better at night. :-)

Note: We're still tweaking and tuning the new config, so pardon any intermittent service disruptions. All of your accounts have been moved over and both portal access and tracker upload should work as before. Regardless, we'll do our best to keep the "dust" to a minimum - and as always, thanks for helping to make the exo.performance.network a success!

Read more...

Office 2007 SP1 Yields Modest Performance Gains

During our most recent round of testing we covered Windows Vista with the Service Pack 1 Release Candidate bits running both Office 2007 and Office 2003. Now that Service Pack 1 for Office 2007 has been released we decided to revisit the tests with the updated bits for Word, Excel and PowerPoint in place.

Once again, we ran the tests on the same 2GHz Core 2 Duo Laptop (Dell XPS M1710) with 2GB of memory. Test methodology consisted of booting Vista, running an initial pass of OfficeBench, then running OfficeBench in a 10-iteration loop and recording the average loop time across the iterations.

The net result: A 3% improvement in OfficeBench completion times.

It would seem that, unlike the Vista team - which has the habit of over-promising and under-delivering - when the Office team talks about a performance boost they actually provide one. And while the results are far from earth-shattering - Vista SP1 + Office 2007 SP1 is still nearly 2x slower than the combination of Windows XP SP3 + Office 2003 - it's at least a step in the right direction.

Note: You can grab your free copy of OfficeBench, plus the rest of the DMS Clarity Suite tools, from the exo.performance.network web site. Just register for your free portal account, hook-up a couple of systems and help us build the world's first global repository of computer performance-related knowledge and data.

Read more...

When Microsoft Attacks!

We at the exo.performance.network pride ourselves on having relatively "thick skins." After all, when you have the audacity to actually take a position on the issues - and then to back them up with hard data - you're bound to bring out the zealots on the other side. However, it's a rare treat when the shots sent our way hail from no less an industry authority than Microsoft.

Yes, it seems our little foray into Windows benchmarking has drawn the ire of Mr. Nick White, official "spokesmouth" for the Windows Vista team. In his blog/rant, Mr. White blabbers on a  bit about unrealistic benchmarking, then proceeds to badmouth our OfficeBench test script by calling it nothing more than a "window-open, window-close" routine. He even includes a ridiculously accelerated video of our test script in action, using the "speed" at which it executes as an excuse for discrediting its validity.

Of course he's wrong, both in his description of the test script and his justification for discrediting it. However, since his agenda likely had little to do with actually clearing the air - and was more in line with a classic Microsoft "hit" piece ordered from on high - we won't bother responding to Mr. White directly. However, we will take a moment to explain how OfficeBench works and why it's garnered so much respect over the years:

1. OfficeBench's origins can be traced to our sister company, Competitive Systems Analysis, Inc. It was designed by Mr. Randall C. Kennedy while his company was under contract to Intel's Desktop Architecture Labs (DAL). This was way back in the 1999/2000 timeframe, when CSA was responsible for a great deal of internal benchmarking and white paper development surrounding the Pentium III and Pentium 4 CPU launches, among other projects.
   
   Why is this important? Because it helps to establish the origins of the technology.
   
   
2. The OfficeBench test script was designed from the beginning to be a "run anywhere" benchmark. By "run anywhere" we mean that the script will execute reliably under almost any Windows runtime environment. At the time it was being developed, this meant Windows 2000 and Terminal Server. As Windows evolved, so did OfficeBench, until now it supports every version of Windows since 2000, including XP, Vista, Server 2003, Server 2008, all flavors of Terminal Server and all known application and desktop virtualization environments. When we say "anywhere," we really mean it.
   
   Why is this important? Because it allows us to test across multiple generations of Windows.
   
   
3. OfficeBench is also version independent. That is, it's designed to work with any version of Microsoft Office. When it was originally conceived, the state of the art was Office 2000. Since then, Microsoft has rolled out 3 additional versions: XP, 2003 and, most recently, 2007. Once again, OfficeBench runs unmodified across all four versions of Office. Combine this with the support for the various Windows platform releases and you begin to see why OfficeBench is so powerful: It is the only test script of its kind that allows you to compare performance across multiple generations of Windows and Office. Mix them, match them - it just works.
   
   Why is this important? Because it allows us to test across multiple generations of Office.
   
   
4. Mr. White's hatchet job aside, OfficeBench is in fact a fairly complex test script. For starters, it uses OLE automation to drive the applications. This is different from most test scripts, which use window messages or key stroke/mouse click simulation. Using OLE automation has numerous benefits, including allowing us to run unmodified across Office versions. It also factors out any input-related anomalies while eliminating the chance that a UI change or 3rd party modification will somehow break the script.
   
   As for Mr. White's assertion that it's a simple "window-open, window-close" script, I offer the following summary of key OfficeBench tasks:
   
   a. Reformat all section headers and subheads in Word.
   b. Generate multiple chart objects in Excel.
   c. Generate complete multi-slide presentation in PowerPoint.
   d. Multi-page scroll w/copy paste of chart objects into Word.
   e. Slide sort/apply multiple templates in PowerPoint.
   f. Multi-page scroll/print preview/print-to-file in Word.
   g. Multi-chart print preview/print-to-file in Excel
   h. Global search/replace in word (multiple).
   i. Multi-slide preview/print-to-file in PowerPoint
   j. Navigate simulated research web site in IE (multiple).
   
   Again, the above are just some highlights. There's a lot more going on than meets the eye, and the key is that it's the exact same set of tasks executing across all versions of Office.
   
   Why is this important? Because it shows that OfficeBench is a sophisticated test script that does more than merely "open and close windows."
   
   
5. OfficeBench doesn't exist in a vacuum. It's delivered as part of a sophisticated, extensible, multi-process testing framework we call DMS Clarity Studio. With DMS Clarity Studio, we provide a variety of scalable workload objects for testing everything from client/server database connections to MAPI-based message store access to streaming multimedia. OfficeBench has been engineered to run in parallel with these workloads, providing for a rich variety of targeted test scenarios spanning the range of Windows client and server platforms. It's all coordinated through the DMS Clarity Studio framework and also seamlessly integrated with the exo.performance.network's Clarity Analysis Portal.
   
   Note: DMS Clarity Studio is offered for free as part of the exo.performance.network. It's also part of the larger DMS Clarity Suite framework in use across thousands of trading workstations and other mission critical systems in the financial services sector. Some of the largest trading firms in the world trust us - Devil Mountain Software, Inc. - to tell them when their systems are under-performing. Enough said.
   
   Why is this important? Because it shows that OfficeBench is part of a proven testing ecosystem that spans the range of Windows platforms and runtime scenarios.
   

In summary, OfficeBench is much more than a simple "window-open, window-close" script. It is a sophisticated, version-independent benchmarking tool that executes reliably under almost any Windows runtime environment. As such, it is the only tool of its kind that allows IT organizations to accurately assess multi-generational performance across all versions of Windows and Office.

And that's why OfficeBench scares the hell of out Microsoft. For the first time ever the industry has the tools necessary to call the company to the mat for its bloated, CPU cycle-sucking ways.

Microsoft's response? Slam the benchmark! Try to discredit the author/source! And crank-up the FUD machine!

Sorry, guys! You can run, but you can't hide, from OfficeBench.

Read more...

Community Snapshot 01: Who's Using What?

As an ongoing service to the greater Windows IT community, we're publishing the first of our weekly "snapshots" of OS and application usage rates as measured within our own little corner of the industry. These numbers are current as of 06:00 GMT and based on a sample of set of 1,106 contributor systems:

Figure 1 - Contributor Usage by OS Type/SKU

Interpretation: While Windows XP (55%) still makes up the lion's share of our user base, Windows Vista - in its various incarnations - is running a closing second (37%). We'll be checking back each week to chart the progress of XP, Vista and their variations over time, and also to monitor adoption rates for Vista SP1 and XP SP3 when they're released early next year.

Read more...

How to Make Vista Run Like XP (Sort of)

In an effort to further clarify our previous test results, we decided to experiment a bit with Vista to see how it would perform with a majority of its newer UI elements and background services turned off. In the process, we believe we've come up with a roadmap of sorts for how to "make Vista run like XP" (sort of):

1. Shutdown "Unnecessary" Services - This means killing all those new Vista goodies, like SuperFetch and the WSearch indexer. CPU cycles are a precious commodity - use them wisely.
   
2. Ditch the UI - Use the Advanced System Settings dialog to change the Appearance settings to "adjust for best performance." It's like taking a trip back in time...to 1995!
   
3. Drop the Resolution/Color Depth - This helps to mitigate any sluggishness in the newer Vista-model drivers. 1024x768 with 16-bit color should be good for most video adapters - and it looks "real sharp" stretched across that new 21 inch LCD!
   
4. Handicap XP - Most Windows XP users are running Office 2000, XP or 2003. Upgrade the XP config with Office 2007 to ensure that you get a nice, entirely unrepresentative (of the real world) hybrid scenario.

Do the above and you'll be rewarded by a much closer net experience. Instead of being ~2x slower than XP, Vista in this new, "bare metal" configuration is *only* (drum roll, please)...40% slower!

Bottom Line: Even with the OS stripped to the core, and with all of the new eye-candy and CPU-sucking background services turned off, Vista is *still* 40% slower than XP (SP3) at a variety of business productivity tasks. Time to ask Santa for that new PC for Christmas!

Read more...