The Great Virtual PC 2007 CPU Gobble!

One of the unexpected side effects of moving from Virtual PC 2007 to Virtual Server as development and testing environment is lower CPU utilization. For some reason, when I load up a test scenario (two client VMs collecting data and uploading to a single server VM) in Virtual PC 2007, the virtualpc32.exe process hosting the scenario chews-up 50% or more of the available CPU cycles on my dual-core workstation. By contrast, when I load up the same scenario under Virtual Server, the VMs chew-up almost no CPU, which is what I would have expected given that the workloads running within them are very light (i.e. Task Manager inside the Guest OS sessions shows nearly zero CPU utilization).

I'm going to forward my findings to Ben Armstrong (i.e. the "Virtual PC Guy" at Microsoft) for analysis, however, I already have a theory on the source of the excessive CPU utilization. For starters, it seems isolated the client VMs, both of which are running a high-resolution monitoring agent (DMS Clarity Metrics Tracker). This agent makes frequent calls to PDH, WMI and the Registry, and I'm guessing that Virtual PC 2007 is generating a lot more overhead when processing these calls than Virtual Server. The agent also uses a set of high-resolution timer objects that likewise seems to give Virtual PC fits.

As a simple test, I tried disabling the agents on each VM. CPU utilization for virtualpc32.exe immediately to dropped to below 10%. Case closed.

Bottom Line: For testing applications that use high-resolution timers, or that make frequent calls to certain system libraries, Virtual Server 2005 R2 SP1 does a much better job of handling what should normally be a very fast, lightweight operation. And with VMRC Plus, you don't have to sacrifice usability in order to reap the rewards of better real-time application support under Virtual Server. Read more...

Vista + Virtualization = Poor Performance

Everyone knows by now that Vista is slower than Windows XP. In fact, my own testing shows it to be roughly twice as slow on the same hardware (see our upcoming Test Center study for more details). That's because Vista is a far more complex operating system, with many additional features and background services that simply don't exist under XP.

What many users don't know, however, is that this gap widens even further under virtualization. Fire-up Vista under VMware or Virtual PC and you'll find that the delta is more like 3x - i.e. tasks take over three times as long to complete under virtualized Vista as they do under virtualized XP.

Let that preceding statement sink in for a minute. In real world terms it means that applications are taking as much as a 50% greater performance hit from being virtualized than would be expected given the aforementioned 2x delta in a native, non-virtualized comparison. As a veteran IT professional, I would expect to take a hit moving to Vista - just not one that's so out of whack with the established norms.

Clearly, there's something going on here that makes Vista particularly difficult to virtualize. I have some theories, however, at this point I can only go on what I've observed. And that is:

1. When testing Vista on a Dell OptiPlex 745 with 4GB of RAM, the performance delta - as measured by the OfficeBench test script - is roughly 2x. The script took twice as long to complete under the Vista/Office 2007 combination as it did under Windows XP/Office 2003.

2. When repeating these tests under VMware Workstation 6.0 and Virtual PC 2007, running on two different hardware test beds (the aforementioned OptiPlex and an XPS M1710 laptop), the script took 3x longer under Vista.

3. All of the tests were conducted using OfficeBench, which is part of the Clarity Studio test framework that is freely available via the exo.performance.network web site (www.xpnet.com).
For the record, I was skeptical of the original results, so much so that I re-ran the VMware scenarios multiple times and then confirmed them against a different installation running under Virtual PC 2007. I also presented my findings to both VMware and Microsoft, but neither could explain the phenomena I observed.

Bottom Line: Vista is significantly slower under virtualization than it should be, and I'll be damned if I know why.

Anyone have an idea what might be going on here? Anyone? Bueller? B-u-e-l-l-e-r? Read more...

The Vista Aero vs. Battery Life Myth

Lately, there has been a rash of unscientific reporting around the issue of Windows Vista and notebook battery consumption. Some customers have apparently reported decreased battery life under Vista, however, most of these reports have been entirely anecdotal in nature: Someone quoting someone else who claims that some notebook has somehow lost some of its battery life under some version of Vista (wow, that's a lot of...err..."something").

Many pundits have pointed to Vista's Aero Glass UI as the source of the power-drain. They assume (wrongly, as it turns out) that rendering the UI via a dedicated graphics processing chip - instead of using the primary CPU to draw a series of bitmaps images - somehow consumes more battery power. Since my own experiences with Vista (six months as my full-time OS on three different notebooks, with Aero enabled on all of them) seem to contradict these reports, I decided to do some objective benchmarking to set the record straight.

To ensure a representative test bed I selected two systems from different vendors operating at opposite ends of the notebook power/performance spectrum:

1. A Dell XPS M1710 with 2GHz Core 2 Duo (T7200) CPU, nVidia GeForce 7900GS graphics, 2GB of DDR-2 (667MHz) RAM and an 80GB, 7200RPM hard disk. This is hardly a "power-miser" rig. In fact, the various components - in particular, the 7900GS card - are notoriously power-hungry, at least during 3D graphics/gaming tasks.

2. A Lenovo ThinkPad R60e with 1.66GHz Core 2 Duo (T5500) CPU, integrated Intel 945 series graphics, 2GB of DDR-2 (533MHz) RAM and a 60GB, 5400 RPM hard disk. This is more of a mainstream system for business class users. It lacks the power-hungry discrete graphics, oversized LCD screen, etc.

The test consisted of multiple iterations (10x) of the OfficeBench test script (part of the DMS Clarity Suite toolset - see http://www.xpnet.com/). I configured both systems to use Vista's Power saver battery scheme and also configured OfficeBench to pause frequently (1-3 seconds per test section) to allow an opportunity for the CPU to throttle-down and/or power-management features to kick-in.

Starting from a fully-charged (100% as reported by Vista's battery meter) state, I pulled the plug on each system and allowed them to complete the test script. I then repeated this sequence, only this time I manually disabled desktop composition (i.e. turned-off the Aero UI) via the Compatibility tab in the Clarity Studio application shortcut. This caused Vista to stop the "dwm" service and render the entire script workload - the application windows, dialogs, animations and transitions - using the older, non-GPU-accelerated model.

As I suspected, the battery consumption for the non-Aero scenario was within 1-2% of the consumption with Aero enabled. In other words, disabling Aero had little or no measurable impact on battery consumption under Windows Vista Ultimate when running a mix of common business productivity (Internet Explorer, Word, Excel and PowerPoint) applications.

So much for that myth... Read more...

Quad Core Nostalgia

As I watch Intel launch its latest quad-core CPU I can't help but wax nostalgic about my time as contract test engineer for the company's Desktop Architecture Labs (DAL). It was early 2000 and the first Pentium 4 was still in preproduction testing. I had just received my prototype system for evaluation - an 800Mhz box with dual-channel RAMBUS (remember those guys?) RDRAM. I knew they were in trouble when my first round of tests - mostly linear office productivity tasks - showed the chip to be 30-40% *slower* than its predecessor, the Pentium III. I communicated my findings back to Intel and they blamed it on a buggy BIOS and poorly tuned chipset.

Weeks later, as I evaluated the now 1.5GHz production-level chip, I became convinced that a traditional linear benchmark approach wasn't going to cut it. The Pentium 4's longer pipeline simply clobbered throughput, with most tests showing performance barely on par with the older P6 core. Fortunately, I was already hard at work on my first parallel-processing test suite, Benchmark Studio, and tests with multiple, concurrent tasks had the Pentium 4 pulling away from the Pentium III by a sizable margin.

Once again, I communicated my findings to Intel, even suggesting a possible marketing spin for the data: More performance for demanding workloads. It would have dovetailed nicely with the related work I had been doing around the company's "Constant Computing" initiative, however, ultimately my findings were canned. Apparently, my message of "more torque for heavy multitasking loads" (i.e. the "SUV" argument) wasn't sexy enough. They wanted a "sports car" message. Better linear performance. Ever higher clock speeds (4GHz was the long term goal). The rest, as they say, is history.

Of course, the Pentium 4 architecture (a.k.a. "NetBurst") ultimately flopped, allowing AMD to each Intel's lunch for many years. When Intel finally dumped NetBurst in favor of a revamped Pentium III design (a.k.a. Core 2), the industry had finally caught-up with where I was over 7 years ago. Now, virtually all business productivity benchmarks emphasize parallel execution performance, a necessity now that most CPUs have 2 or more cores on board. Symmetrical Multiprocessing (SMP), once the purview of engineering workstations, is now de rigeur, and the current mainstream OS - Windows XP - is completely at home on multiple CPUs.
I guess I can take some small measure of satisfaction in knowing that I was right about where benchmarking was headed, and that if Intel had followed my lead they might have fared better (at least in terms of marketing success).

Note: You can download the latest incarnation of my test suite, Clarity Studio, for free from the exo.performance.network (www.xpnet.com) web site. Read more...

Vista Aero: What a CPU Hog!

Microsoft's new Aero Glass GUI - one of the cornerstones of the company's Windows Vista marketing message - is a joy to behold. Aero's sleek, semi-transparent facades serve to enhance the user experience by providng a stronger sense of "depth" and cohesion. Combined with Vista's enhanced UI metaphors (love those "breadcrumbs" in explorer), Aero is a major improvement over the XP GUI.

It's also a CPU hog. Despite Microsoft's claims about leveraging 3D accelerator technology to offload the GUI workload, Aero still chews-up more CPU cycles (an average of 22%) with desktop composition enabled (i.e. 3D accelerated mode) than with it disabled (i.e. non-accelerated "legacy" mode). In other words, turn on the "bling" and you toss nearly a quarter of your CPU bandwidth out the window.

Note: I measured the above using the recently updated DMS Clarity Tracker Agent, which is now part of the new public exo.performance.network (xpnet.com) project. You can reproduce the test scenario by downloading the companion DMS Clarity Studio tool (also at xpnet.com) and running the OfficeBench test script, first with composition enabled and then with it disabled.

Other interesting tidbits:

• The ratio of the increased CPU overhead is roughly 4:1 in favor of User vs.Privileged (i.e. kernel mode) time - a good thing.
  
  
• That User mode bump can be traced directly to the participating applications - for example, Word 2007 using 48% more CPU time with composition enabled.
  
  
• Enabling desktop composition also causes windows to chew another 16% of the available physical RAM as measured by the Committed Bytes counter.
  
  
• Though CPU overhead was higher with composition enabled, overall script completion times - as measured by OfficeBench - were only 3% slower.
  
  
• One alarming statistic: Processor Queue Length, a measure of how many threads are queued and waiting for CPU time, increases by 28% with composition enabled - not good, especially for multitasking.
  

Bottom Line: For single-tasking users running generic productivity application scenarios, the added overhead from enabling desktop composition shouldn't be much a factor. However, more demanding users - especially those needing maximum performance - may find that turning composition off let's them regain some of those long lost CPU cycles.

I guess the old saw still applies: What Intel (and AMD) giveth, Microsoft taketh away...

Read more...

The Secret Life of SuperFetch

One of the more mysterious new features of Windows Vista is its SuperFetch memory management subsystem. Billed as a "smart" pre-caching mechanism, SuperFetch is supposed to improve system responsiveness by monitoring application usage patterns and then pre-loading application code in anticipation of the next task. SuperFetch uses the time of day and other behavioral markers to determine what to load when, using the Windows memory manager's Standby List as its entry point (the ever illuminating Mark Russinovich goes into more detail in his recent TechNet Article).

Of course, it all sounds good on paper. But how do you quantify the impact of something that's designed to work in the background and to be essentially undetectable (beyond some vague sense that the OS is more responsive)? For starters, it helps to know where to find it. SuperFetch runs as a Windows Service under the SVCHOST alias. The actual filename is sysmain.dll, so a quick scan of Task Manager to correlate the Process ID with the instance of SVCHOST in question gets you in touch with SuperFetch.

Next, you need a way to monitor both SuperFetch's behavior and its impact on system responsiveness. The first part is easy - any Vista-compatible metrics agent will do the trick, though we prefer the one we provide through the xpnet: DMS Clarity Tracker. Measuring the second part - how SuperFetch impacts the system - is a bit trickier.

Here we used another xpnet tool, DMS Clarity Studio, to generate a scripted productivity workload spanning Microsoft Word, Excel, PowerPoint and Internet Explorer. By comparing before/after results with SuperFetch enabled/disabled (and rebooting after each test run) we were able to determine that Microsoft's new VMM magic is indeed having a positive impact on application responsiveness. With the service enabled, Startup times - as measured by the OfficeBench test script - were cut in half, this after multiple "training" runs to allow SuperFetch to map the usage pattern (i.e. "We run Office after booting").

We'll be conducting additional research and testing around SuperFetch and other new Vista technologies (Integrated Search, ReadyBoost) in future blog entries for the exo.performance.network. Stay tuned! Read more...

Monster Excel Workbooks Exposed

Everyone knows that Microsoft Office is a bit of a memory hog. In fact, few products can claim as much credit for driving the memory upgrade cycle as the ubiquitous combination of Word, Excel, PowerPoint and Outlook. However, while many power users may think they’ve pushed the envelope on one or more of these applications – massive documents, huge spreadsheets, media-rich presentations – none can compare to those captains of industry that make their home at the corner of Wall and Broadway.

I’m referring, of course, to financial services traders. The “Type A” personality crowd – risk takers, deal makers, the rock stars of wealth creation. They live life on the edge, balancing risk vs. reward in constant battle with each other and the market itself. And the fuel that drives their engines is…data. Lots and lots of data – analyzed, quantified and extrapolated in nearly every conceivable way.

Massaging that data falls on the shoulders of Microsoft Excel. Through myriad templates and macros and real-time data connections, these traders push Excel to the extreme as they tweak and tune their customized (and highly proprietary) financial models. All of which consumes a tremendous amount of hardware resources. In at least one shop we found that traders ran, on average, six concurrent instances of the Excel application process, each one occupying a peak memory footprint of from 300-500MB during a normal trading session (for a total of 1.8GB).
CPU utilization was also high, with each instance consuming ~60% of the available CPU cycles on a 4-CPU workstation, or 300% out of a total CPU capacity of 400% (100% x 4 CPU). Then there was the thread count. At any given time these systems were asked to juggle up to 230 concurrent execution threads just from the various Excel instances.

That’s approximately 1/2 the total thread workload for a typical business productivity user, yet this is just one application among many. These systems are also running proprietary trading software (plus various real-time feeds, like Bloomberg), which is why even a high-end PC isn’t adequate. Hence their reliance on top-of-the-line workstation hardware. And even then, with dual-cores and gigabytes of memory, many traders still need more than one system in order to handle their computational load. In fact, it’s not uncommon to find 3 or more high-end boxes under a trader’s desk – thanks in large part to the overhead of their massive Excel workbooks.

1.8GB of RAM. 300% CPU utilization. 230 concurrent execution threads. And you thought your spreadsheets were complicated! Read more...