But despite years of real-world experience with both sides of the duopoly, few organizations have taken the time to directly quantify what my colleagues and I at Intel used to call “The Great Moore’s Law Compensator (TGMLC).” In fact, the hard numbers above represent what is perhaps the first ever attempt to accurately measure the evolution of the Windows/Office platform in terms of real-world hardware system requirements and resource consumption.
Over the next several sections I hope to further quantify the impact of TGMLC and to track its effects across four distinct generations of Microsoft’s desktop computing software stack. To accomplish my goal I’ll be employing a cross-version test script – OfficeBench – and executing it against different combinations of Windows and Office: Windows 2000 + Office 2000; Windows XP (SP1) + Office XP; Windows XP (SP2) + Office 2003; and Windows Vista + Office 2007. Tests will first be conducted in a controlled virtual machine environment under VMware and then repeated on different generations of Intel desktop and mobile hardware to assess each stack’s impact on hardware from the corresponding era.
Click Image to View Our Interactive Results Table
About OfficeBench: The OfficeBench test script is a version-independent benchmark tool that uses OLE automation to drive Microsoft Word, Excel, PowerPoint and Internet Explorer through a series of common business productivity tasks. These include assembling and formatting a compound document and supporting workbooks and presentation materials, as well as data-gathering through simulated browsing of a web-based research database. OfficeBench is available for free download from the exo.performance.network (http://www.xpnet.com/) web site as part of the DMS Clarity Studio testing framework.
The Stone Age
Back in 1999, when I was working as an advisor to Intel’s Desktop Architecture Labs (DAL), I remember how thrilled we all were to get our hands of Windows 2000 and Office 2000. Finally, a version of the Windows/Office stack that could leverage all of the desktop horsepower we were building in to the next generation Pentium 4 platform. I remember it was also the first time I had a fully-scriptable version of the Office Suite to work with (previous versions had supported OLE automation only in Word and Excel). Shortly thereafter, the first version of OfficeBench was born and I began my odyssey of chronicling TGMLC through the years.
First-off, let me characterize the state-of-the-art at the time. The Pentium 4 CPU was about to be unveiled and the standard configuration in our test labs was a single-CPU system with 128MB of RDRAM and an IDE hard disk. While a joke by today’s standards, this was considered a true power-user configuration suitable for heavy number-crunching or even lightweight engineering workstation applications. It was also only marginally faster than the previous generation Pentium III, a fact that Intel tried hard to hide by cranking-up the CPU clock to 1.5GHz and turning its competition with rival AMD into a drag race. It’s a decision that would come back to haunt them well into the next century.
Sadly, I didn’t have access to an original Pentium 4 system for this article. My engineering test bed was long ago scrapped for parts, and I doubt that many of these old i840 chipset-based boxes are still in use outside of the third world. However, we can at least evaluate the software stack itself. Through the magic of virtualization we can conclude that, even with only 128MB of RAM, a Windows 2000-based configuration had plenty of room to perform. During OfficeBench testing, the entire suite consumed only 9MB of RAM, while the overall OS footprint never exceeded 50% of the available memory. Clearly this was a lean, mean version of Windows/Office and it chewed through the test script a full 17% faster than its nearest competitor, Windows XP (SP1) + Office XP.
The Bronze Age
The introduction of Windows XP in 2001 marked the first mainstream (i.e. not just for business users) version of Windows to incorporate the Windows “NT” kernel. In addition to improved Plug & Play support and other improvements, XP sported a revamped user interface with true-color icons and lots of shiny, beveled effects. Not wanting to look out of style, and also smelling another sell-up opportunity, the Office group rushed out Microsoft Office XP (a.k.a. “Office 10”), which was nothing more than a slightly tweaked version of Office 2000 with some UI updates.
Hardware had evolved a bit in the two years since the Windows 2000 launch. For starters, Intel had all but abandoned its ill-fated partnership with RAMBUS. New Intel designs featured the more widely supported DDR-SDRAM, while CPU frequencies were edging above 2GHz. Intel also upped the L2 cache size of the Pentium 4 core from 256KB to 512KB (i.e. the “Northwood” redesign) in an attempt to fill the chip’s stall-prone 20 stage integer pipeline. Default RAM configurations were now routinely in the 256MB range while disk drives sported ATA-100 interfaces.
Windows XP, especially in the pre-Service Pack 2 timeframe, wasn’t all that more resource intensive than Windows 2000. It wasn’t until later, as Microsoft piled-on the security fixes and users started running anti-virus and anti-spyware tools by default, that XP began to put on significant “weight.” Also, the relatively modest nature of the changes from Office 2000 to Office XP translated into only a minimal increase in system requirements. For example, overall working set size for the entire suite during OfficeBench testing under VMware was only 1MB higher than Office 2000, while CPU utilization actually went down 1% across the three applications (Word, Excel and PowerPoint). This did not, however, translate into equivalent performance. As I noted before, Office XP on Windows XP took 17% longer than Office 2000 on Windows 2000 to complete the same OfficeBench test script.
I was fortunate enough to be able to dig-up a representative system of that era: A 2GHz Pentium 4 system with 256MB of RAM and integrated Intel Extreme graphics (another blunder by the chip maker). Running the combination of Windows XP (SP1) and Office XP on bare iron allowed me to evaluate additional metrics, including the overall stress level being placed on the CPU. By sampling the Processor Queue Length (by running the DMS Clarity Tracker Agent in parallel with Clarity Studio and OfficeBench), I was able to determine that this legacy box was only moderately stressed by the workload. With an average Queue Length of 3 ready threads, the CPU was busy but still not buried under the computing load. In other words, given the workload at hand, the hardware seemed capable of executing it while remaining responsive to the end-user (a trend I saw more of as testing progressed).
The Industrial Revolution
Office 2003 arrived during a time of real upheaval at Microsoft. The company’s next major Windows release, code named “Longhorn,” was behind schedule and the development team was being sidetracked by a string of security breaches in the Windows XP code base. The resulting fix, Windows XP Service Pack 2, was more of a re-launch than a mere update. Whole sections of the OS core were either replaced or rewritten, and new technologies – like Windows Defender and a revamped firewall – added layers of code to a rapidly bloating platform.
Into this mess walked Office 2003 which, among other things, tried to bridge the gap between Windows and the web through support for XML and the ability to store documents as HTML files. Unlike Office XP, Office 2003 was not a minor upgrade but a major overhaul of the suite. And the result was, not surprisingly, more bloating of the Windows/Office footprint. Overall memory consumption went up modestly to 13MB during OfficeBench testing while CPU utilization remained constant vs. previous builds, this despite the fact that the suite was spinning an extra 4 execution threads (overall thread count was up by 15).
Where the bloat took its toll, however, was in raw application throughput. Completion times under VMware increased another 8% vs. Office XP, putting the Windows XP (SP2) + Office 2003 combination a full 25% off the pace of the original Windows 2000/Office 2000 numbers from 3 years earlier. In other words, with all else being equal – hardware, environment, configuration – Microsoft’s desktop computing stack was losing in excess of 8% throughput per year due to increased code path complexity and other delays.
Of course, all else was not equal. Windows XP (SP2) and Office 2003 were born into a world of 3GHz CPUs, 1GB RAM, SATA disks and Symmetrical Multithreading (a.k.a. Hyper-threading). This added hardware muscle served to offset the growing complexity of Windows/Office, allowing a newer system to achieve OfficeBench times slightly better (~5%) than a legacy Pentium 4 system, despite the latter having a less demanding code path (TGMLC in action once again).
Welcome to the 21st Century
Given the extended delay of Windows Vista and its accompanying Office release, Microsoft Office System 2007, I was understandably concerned about the level of bloat that might have slipped into the code base. After all, Microsoft was promising the world with Vista, and early betas of Office showed a radically updated interface (the Office “Ribbon”) as well as a new, open file format and other nods to the anti-establishment types. Little did I know that Microsoft would eventually trump even my worst predictions: Not only is Vista + Office the most bloated desktop software stack ever to emerge from Redmond, its system requirements are so out of proportion with recent hardware trends that only the latest and greatest from Intel or AMD can support its epically porcine girth.
Let’s start with the memory footprint. The average combined working set for Word, Excel and PowerPoint when running the OfficeBench test script is 109MB. By contrast, Office 2000 consumed a paltry 9MB, which translates into a 12x increase in memory consumption (i.e. 170% per year since 2000). To be fair, previous builds of Office benefited from a peculiar behavior common to all pre-Office 12 versions: When minimized to the Task Bar, each Office application would release much of its non-critical working set memory. This resulted in a much smaller memory footprint, as measured by the Windows performance counters (which are employed by the aforementioned DMS Clarity Tracker Agent).
Microsoft has discontinued this practice with Office 2007, resulting in much higher average working set results. However, even factoring this behavioral change, the working set for Office 2007 is truly massive. Combined with an average boot-time image of over 500MB for just the base Windows Vista code base, it seems clear that any system configuration that specifies less than 1GB of RAM is a non-starter with this version. And none of the above explains the significantly higher CPU demands of Office 2007, which are nearly double (73% vs. 39%) that of Office 2003. Likewise, the number of execution threads spawned by Office 2007 (32) is up, as is the total thread count for the entire software stack (615 vs. 370 – again, almost double the previous version).
Clearly, this latest generation of the Windows/Office desktop stack was designed with the next generation of hardware in mind. And in keeping with the TGMLC pattern, today’s latest and greatest hardware is indeed up to the challenge. Dual (or even Quad) cores, combined with 4MB or more of L2 cache, have helped to sop-up the nearly 2x greater thread count, while 2GB standard RAM configurations are mitigating the nearly 1GB memory footprint of Vista + Office 2007.
The net result is that, surprise, Vista + Office 2007 + state of the art hardware delivers throughput that’s nearly on par (~22% slower) with the previous generation of Windows XP + Office 2003 + the previous state of the art hardware. In other words, the hardware gets faster, the code base gets fatter and the user experience, as measured in terms of application response times and overall execution throughput, remains relatively constant. The Great Moore’s Law Compensator is vindicated.
Conclusion
As I stated in the beginning, the conventional wisdom regarding PC evolution could be summed up in this way: “What Intel giveth, Microsoft taketh away.” The testing I conducted here shows that the wisdom continues to hold true right up through the current generation of Windows Vista + Office 2007. What’s shocking, however, is the way that the IT community as a whole has grown to accept the status quo. There is a sense of inevitability attached to the concept of the “Wintel duopoly,” a feeling that the upgrade treadmill has become a part of the industry’s DNA. Forces that challenge the status quo – Linux, Google, OS X – are seen as working against the very fabric of the computing landscape.
But as recent events have shown us, the house that “Wintel” built exists largely because of a fragile balance between hardware evolution and software complexity. When that balance gets out of whack – as was the case when Vista became delayed, leaving Intel with a hard sell for many of its newer offerings – the house can quickly destabilize. And when that happens, it will be up to one of the aforementioned outside forces to seize the initiative, topple the “Wintel” structure and perhaps change the very nature of desktop computing as we know it. Read more...
Posts
