Friday, February 19, 2010

(Editorial): Rebutting Ars Technica

Upated: Thanks to an industrious reader, we finally have outside confirmation that our understanding of this issue was dead on. Yes, Virginia, this means we were right and you were wrong. So before you go posting yet another idiotic comment parroting yet another clueless blogger from yet another obscure corner of the Internet, please read our editorial titled “What took you so long?

The remainder of this post has been retracted due to it being superseded by the above linked follow-up post.

Read more...

Thursday, February 18, 2010

(WCPI): An In-Depth Look at the Methodology

The Windows Composite Performance Index (WCPI) is a new industry metric built upon the foundations of one the most robust and sophisticated performance analysis frameworks ever devised. The DMS Clarity Suite framework was created by a former Intel Corporation performance engineer who specialized in the design and implementation of next generation benchmarking methodologies. It has been deployed commercially at some of the largest financial services companies in the world and is in use today on trading floors throughout Wall Street.

The methodology behind the framework is straightforward: Monitor critical Windows performance counters; evaluate them against a series of weighted thresholds; and flag those events wherein one or more of the counters exceeds its configured target limit. To further qualify these events – which are defined as contiguous sample periods in which a threshold has been exceeded – each one is evaluated both for severity (how far above the threshold) and duration (how long it remained that way). This has the effect of skewing the analysis away from isolated “spikes” – which are often meaningless - and towards prolonged sequences of threshold exceptions of the type that, cumulatively, can cause significant system-wide delays.

For example, when evaluating memory utilization, one of the metrics monitored is the Memory\Pages Input/Second counter. As the DMS Clarity framework’s analysis engine reviews sample records for a given system (over 10,000 such samples are recorded per system, per week), it documents each incident in which this counter has exceeded the user-defined threshold (in the case of WCPI, the threshold is set at 300). However, the engine also records how many contiguous (i.e. sequential in time) records show this counter – and the other counters it evaluates – running above the threshold, and will generally discount a short duration event with a high counter value (i.e. a sudden spike from an application startup or large memory mapped file I/O) in favor of a longer duration event (i.e. a sustained value associated with a prolonged disk I/O operation, like extensive VMM paging) in which the individual counter values may be lower but their corresponding negative side effects more detrimental to system performance.

Naturally, the number of threshold violation “hits” for a given counter or group of counters correlates directly to the frequency of the sample periods and also the restrictiveness of the threshold values. In commercial deployments, the framework is configured to sample a system’s runtime environment every second, and then average the collected values every 15 seconds before storing the results as a single record. It’s also configured to use very aggressive thresholds in order to identify degraded runtime conditions that might impact a real-time computing environment(e.g. a trading workstation in a Wall Street brokerage).

For WCPI calculations, the sample frequency is retained but the storage schedule widened to one record per averaged 60 second interval (vs. 15 seconds in commercial deployments). Likewise, the individual counter thresholds are loosened to reflect a more general purpose runtime scenario. The net result is an analysis methodology that ignores events that would otherwise have triggered a violation in a commercial setting, essentially making the WCPI calculations much more lenient and forgiving of runtime conditions – an important consideration when casting such a wide net across a diverse population like the exo.performance.network.

Bottom Line: With a robust, sophisticated analysis framework behind it, and the world’s largest repository of raw Windows metrics data on the planet the work on, WCPI is by far the most accurate metric of its kind. No other industry metric provides equivalent insight into the runtime behavior of Windows systems as they exist in the real-world.

The following is a technical break-down of the three key sub-indices that are used to generate the WCPI value. These indices are recalculated daily across all participating exo.performance.network sites, and the resulting data is then combined and averaged to achieve the final WCPI number:

  • The Peak CPU Saturation Index is calculated by comparing a series of 4 independent Windows metrics – The System\Processor Queue Length counter, the Process Instant Delay factor (a custom metric unique to DMS Clarity), the Process Cumulative Delay factor (another DMS Clarity custom metric) and the event duration – against a series of user-defined threshold values. The resulting individual ratios are then weighted and combined to create the single number Peak CPU Saturation Index.

    Note: Peak CPU Saturation is the not the same thing as 100% CPU utilization. A system can use 100% of its CPU bandwidth and yet still remain responsive. Rather, it is a measure of the amount of delay, in real-time, that the system is experiencing as a result of contention for CPU resources as well as the impact that delay will likely have for any a given process running on the system under those conditions.

  • The Peak Memory Pressure Index is calculated by comparing a series of 4 independent Windows metrics – the Memory\Committed Bytes counter, the Memory\Pages Input/sec counter, the PageFile\% Usage counter and the aforementioned event duration value – against a set of user-defined threshold values. The resulting individual ratios are then weighted and combined to create the single number Peak Memory Pressure Index value.

    Note: Peak Memory Pressure is a gradual, as opposed to sudden or instantaneous, metric. The index normally builds over time as the number of active processes increases, putting pressure on the Windows VMM to re-shuffle the physical memory deck and page portions of certain processes to disk where necessary.

  • The Peak I/O Contention Index is calculated by comparing a series of 4 independent Windows metrics – the Physical Disk\Current Disk Queue Length counter, the Physical Disk\Disk Bytes/Sec counter, the Processor\Interrupts/Sec counter and the aforementioned even duration – against a set of user-defined threshold values. The resulting individual ratios are then weighted and combined to create the single number Peak I/O Contention Index.

    Note: The Peak I/O Contention Index often mirrors the Peak Memory Pressure Index during low-memory situations where VMM activity is dominating the disk channels.

Note: You can keep tabs on all of our research findings by visiting our web site: www.xpnet.com. There you’ll find a wide selection of interactive chart objects and free monitoring tools that you can use to compare your own systems to the WCPI and similar independent research metrics published by the exo.performance.network.

Read more...

Wednesday, February 17, 2010

(WCPI): 85% of Win 7 PCs are Underperforming

Updated: Thanks to an industrious reader, we finally have outside confirmation that our understanding of this issue was dead on. Yes, Virginia, this means we were right and you were wrong. So before you go posting yet another idiotic comment parroting yet another clueless blogger from yet another obscure corner of the Internet, please read our editorial titled “What took you so long?


Windows XP holdouts take heart: New data from the exo.repository shows that better than 8 in 10 Windows 7 systems monitored by the exo.performance.network are running alarmingly low on physical memory. And nearly the same number are demonstrating significant delays in I/O processing – ostensibly due to heavy virtual memory activity as Windows compensates for insufficient RAM.

Note: We’ve published a new blog entry that explains our methodology in depth. If you have questions about how we came up with the numbers in this post, this is the place to look.

This disturbing trend was identified by exo.performance.network researchers as they compiled data in support of the newly released Windows Composite Performance Index (WCPI). Computed values for the Peak CPU Saturation, Peak Memory Pressure and Peak I/O Contention indices – all specialized components of the more generalized WCPI metric – showed that Windows 7 systems had much higher memory pressure values than systems running Windows XP. This was true despite the fact that the Windows 7 systems in question had, on average, nearly twice as much (3.3GB vs. 1.7GB for Windows XP) RAM installed.

 

Win XP

Win Vista

Win 7

CPU Saturated

32%

45%

44%

Low on RAM

40%

80%

86%

I/O Backlogged

36%

83%

85%

Table 1 – WCPI Percentages by Platform

Such a discrepancy in recorded memory consumption is not surprising given Windows 7’s much higher RAM footprint. However, when you factor in the generally more advanced hardware that’s being deployed with Windows 7 (over a third of the legacy XP systems sampled have CPUs that pre-date the launch of Windows Vista), the realization that Microsoft’s new OS is quickly expanding to consume all available resources is still quite alarming. Add to this Windows 7’s higher CPU utilization (44% of sampled Windows 7 PCs are CPU bound, vs. 32% for XP) and it’s clear that Microsoft’s latest and greatest is well on its way to overtaking Moore’s Law.

Note: The above data points were compiled from the hundreds of millions of system and process metrics records that have been uploaded to the exo.repository by participating member sites. You can keep tabs on all of our research findings by visiting our web site: www.xpnet.com. There you’ll find a wide selection of interactive chart objects and free monitoring tools that you can use to compare your own systems to the WCPI and similar independent research metrics published by the exo.performance.network.

Read more...

(WCPI): Rethinking Windows Performance

The exo.performance.network is proud to announce the debut of our latest industry performance metric. The Windows Composite Performance Index (WCPI) is a comprehensive measurement of Microsoft Windows PC performance as observed at IT sites from around the globe.

Based on data collected by our network of nearly 24,000 registered users, the WCPI delivers an easy-to-read, single number index representing the aggregate runtime performance of systems participating in our free monitoring services. As such, it provides a uniquely authoritative look at the internal behavior of PCs running Microsoft’s ubiquitous desktop operating systems.

Figure 1 – The WCPI Interactive Widget

Note: The WCPI is re-calculated daily, with supporting data drawn from system metrics records that are uploaded hourly to the exo.repository by participating exo.performance.network member sites. The resulting index value is itself an aggregate of three component-specific hardware performance indices:

  • The Peak CPU Saturation Index has to do with the PC’s workload as it relates to its central processing unit – specifically, how much of a computational backlog (in terms of real-world delay in completing tasks) is accumulating during normal system operation.


  • The Peak Memory Pressure Index relates to the PC’s use of physical memory – how close it is to running out of RAM and also how often it relies on virtual memory to satisfy demand.


  • The Peak I/O Contention Index describes the PC’s performance when interacting with peripheral devices, including the system’s hard disk(s) and any network adapters.

Together, these indices paint a comprehensive picture of performance across the three critical areas of PC system architecture. The exo.performance.network is offering the WCPI, and its supporting free system, process and network monitoring services, as a free, vendor-independent metric of Windows performance in the real-world.

Combined with our exo.charts library and OfficeBench benchmarking tool, the WCPI represents the latest addition to the industry’s most authoritative reference for all things Windows performance-related. You can learn more about the exo.performance.network by visit our web site: www.xpnet.com.

Read more...