(WCPI): An In-Depth Look at the Methodology

The Windows Composite Performance Index (WCPI) is a new industry metric built upon the foundations of one the most robust and sophisticated performance analysis frameworks ever devised. The DMS Clarity Suite framework was created by a former Intel Corporation performance engineer who specialized in the design and implementation of next generation benchmarking methodologies. It has been deployed commercially at some of the largest financial services companies in the world and is in use today on trading floors throughout Wall Street.

The methodology behind the framework is straightforward: Monitor critical Windows performance counters; evaluate them against a series of weighted thresholds; and flag those events wherein one or more of the counters exceeds its configured target limit. To further qualify these events – which are defined as contiguous sample periods in which a threshold has been exceeded – each one is evaluated both for severity (how far above the threshold) and duration (how long it remained that way). This has the effect of skewing the analysis away from isolated “spikes” – which are often meaningless - and towards prolonged sequences of threshold exceptions of the type that, cumulatively, can cause significant system-wide delays.

For example, when evaluating memory utilization, one of the metrics monitored is the Memory\Pages Input/Second counter. As the DMS Clarity framework’s analysis engine reviews sample records for a given system (over 10,000 such samples are recorded per system, per week), it documents each incident in which this counter has exceeded the user-defined threshold (in the case of WCPI, the threshold is set at 300). However, the engine also records how many contiguous (i.e. sequential in time) records show this counter – and the other counters it evaluates – running above the threshold, and will generally discount a short duration event with a high counter value (i.e. a sudden spike from an application startup or large memory mapped file I/O) in favor of a longer duration event (i.e. a sustained value associated with a prolonged disk I/O operation, like extensive VMM paging) in which the individual counter values may be lower but their corresponding negative side effects more detrimental to system performance.

Naturally, the number of threshold violation “hits” for a given counter or group of counters correlates directly to the frequency of the sample periods and also the restrictiveness of the threshold values. In commercial deployments, the framework is configured to sample a system’s runtime environment every second, and then average the collected values every 15 seconds before storing the results as a single record. It’s also configured to use very aggressive thresholds in order to identify degraded runtime conditions that might impact a real-time computing environment(e.g. a trading workstation in a Wall Street brokerage).

For WCPI calculations, the sample frequency is retained but the storage schedule widened to one record per averaged 60 second interval (vs. 15 seconds in commercial deployments). Likewise, the individual counter thresholds are loosened to reflect a more general purpose runtime scenario. The net result is an analysis methodology that ignores events that would otherwise have triggered a violation in a commercial setting, essentially making the WCPI calculations much more lenient and forgiving of runtime conditions – an important consideration when casting such a wide net across a diverse population like the exo.performance.network.

Bottom Line: With a robust, sophisticated analysis framework behind it, and the world’s largest repository of raw Windows metrics data on the planet the work on, WCPI is by far the most accurate metric of its kind. No other industry metric provides equivalent insight into the runtime behavior of Windows systems as they exist in the real-world.

The following is a technical break-down of the three key sub-indices that are used to generate the WCPI value. These indices are recalculated daily across all participating exo.performance.network sites, and the resulting data is then combined and averaged to achieve the final WCPI number:

• The Peak CPU Saturation Index is calculated by comparing a series of 4 independent Windows metrics – The System\Processor Queue Length counter, the Process Instant Delay factor (a custom metric unique to DMS Clarity), the Process Cumulative Delay factor (another DMS Clarity custom metric) and the event duration – against a series of user-defined threshold values. The resulting individual ratios are then weighted and combined to create the single number Peak CPU Saturation Index.
  
  Note: Peak CPU Saturation is the not the same thing as 100% CPU utilization. A system can use 100% of its CPU bandwidth and yet still remain responsive. Rather, it is a measure of the amount of delay, in real-time, that the system is experiencing as a result of contention for CPU resources as well as the impact that delay will likely have for any a given process running on the system under those conditions.
  
  
• The Peak Memory Pressure Index is calculated by comparing a series of 4 independent Windows metrics – the Memory\Committed Bytes counter, the Memory\Pages Input/sec counter, the PageFile\% Usage counter and the aforementioned event duration value – against a set of user-defined threshold values. The resulting individual ratios are then weighted and combined to create the single number Peak Memory Pressure Index value.
  
  Note: Peak Memory Pressure is a gradual, as opposed to sudden or instantaneous, metric. The index normally builds over time as the number of active processes increases, putting pressure on the Windows VMM to re-shuffle the physical memory deck and page portions of certain processes to disk where necessary.
  
  
• The Peak I/O Contention Index is calculated by comparing a series of 4 independent Windows metrics – the Physical Disk\Current Disk Queue Length counter, the Physical Disk\Disk Bytes/Sec counter, the Processor\Interrupts/Sec counter and the aforementioned even duration – against a set of user-defined threshold values. The resulting individual ratios are then weighted and combined to create the single number Peak I/O Contention Index.
  
  Note: The Peak I/O Contention Index often mirrors the Peak Memory Pressure Index during low-memory situations where VMM activity is dominating the disk channels.

Note: You can keep tabs on all of our research findings by visiting our web site: www.xpnet.com. There you’ll find a wide selection of interactive chart objects and free monitoring tools that you can use to compare your own systems to the WCPI and similar independent research metrics published by the exo.performance.network.