Thursday, September 4, 2008

IE 8/Chrome Follow-Up

After posting our controversial findings about memory usage under IE 8 and Chrome (see previous post here and here), we received feedback from some our our readers criticizing our use of the Process\Working Set counter values to size the various browser processes. The complaint was that this counter does not factor code (OCX, DLLs, et al) shared across the various process instances.

Fortunately, the DMS Clarity Tracker agent also collects the Page File Bytes counter for each process (i.e. the “Private Bytes” counter as represented in Task Manager), which is a more accurate measure of the RAM footprint (as backed by reserved page file space) of each process instance. And indeed, PF Bytes paints a slightly different picture than Working Set.

image 
Figure 1 – Page File (i.e. Private) Bytes under Windows XP

For starters, IE 8’s average memory consumption under Windows XP, as tallied across all process instances, drops from 271MB to 242MB when viewed through PF Bytes. Likewise, Chrome’s total process working set (i.e. all 12 instances) drops from 267MB to 208MB, while Firefox’s single process footprint goes from 104MB to 94MB. Interesting, IE 7’s Working Set and PF Bytes values are almost identical – it appears to be sharing nothing with any other process.

image
Figure 2- % Shared Code Across All Browser Instances

In a way, we’re glad that we received the aforementioned critical feedback because it forced us to take a look at the level of code re-use present in each competing browser. And here is where Google’s clean design reaps real dividends: By comparing the total Working Set and PF Bytes values, we can conclude that Chrome is sharing roughly 28% of its code image across its various instances. Compare this to IE 8 (12%) and Firefox (11%), and you begin to see why its sometimes better to start with a clean slate.

Bottom Line: IE 8 is still “fat” by any measure, while Google Chrome’s “chubbiness” is offset somewhat by its more efficient use of shared code across process instances.

Read more...

Wednesday, September 3, 2008

Google Chrome the “Fattest” of them All

Earlier this week, we posted our findings from a 3-way browser comparison we had conducted featuring Internet Explorer 7, Internet Explorer 8 (Beta 2) and Firefox 3.01. As we were going to “print,” we learned that Google was readying its own foray into the browser market – Google Chrome – prompting us to quickly revisit our test scenario to add this high-profile newcomer to the mix.

Note: For more information on the exo.performance.network, or to register for your free Analysis Portal account, visit www.xpnet.com.

What we found was truly shocking: After re-executing our 10-site, multi-tab scenario across all 4 browsers, we discovered that it is Google Chrome, not Internet Explorer 8, that is the true memory consumption leader.

With a peak working set (324MB) just shy of IE 8’s updated 332MB mark, Chrome put a massive hit on available memory for our 2GB Windows XP (SP3) test system. And when we switched our focus to the average working set, Chrome actually topped IE 8 with an impressive 267MB footprint (compared to 211MB for IE 8).

image
Figure 1 – Average and Peak Working Sets

Of course, both browser look absolutely porcine when compared to the lean, mean Firefox 3.01 (151MB peak, 104MB average working set size). And lest we forget, IE 7 continues to hover somewhere between the fit & trim Firefox and the obesity that defines Chrome/IE 8 (209MB peak, 142MB average).

One area where the Mozilla.org byproduct shares common ground with Chrome is average CPU utilization. Both products chew-up CPU bandwidth much more aggressively than either of the IE variants, with Chrome actually outpacing Firefox by 2 percentage points (45% to 42% for FF). Contrast these numbers with IE 8 (24%) and IE 7 (13%) and you begin to see that, when it comes to gobbling up CPU cycles, Chrome and Firefox are in fact birds of a feather.

Interestingly, the one place where Chrome completely befuddled us was in its use of threading. Both Firefox and IE 7 spawn a relatively modest number of threads (25 and 43, respectively), a fact related to their reliance on a single process instance to handle all tabbed sessions. By contrast, IE 8 spawns potentially hundreds of threads (153 in our latest test round), and spreads them out across its various instances (in our case, 6 discrete copies of iexplore.exe).

image
Figure 2 – Average and Peak Thread Count 

Given its use of a similar, multi-process model, we would have expected Google Chrome to introduce a comparable thread workload. So we were understandably surprised to discover that Chrome had spun a much more manageable 48 execution threads at the peak of the test scenario (10 sites loaded into 10 separate tabs). Even more surprising was how these threads were distributed: 25 of them belonged to the initial Chrome process instance (i.e. the one that’s created when you first start Chrome); the rest were distributed almost evenly across the other process instances – in our case, 2 threads per instance over 12 discrete instances (with one instance grabbing an extra 3 threads for unknown reasons).

Our guess – and it really is just that until we get confirmation – is that those initial 25 threads are there primarily to handle the base UI functions (client window, address bar, menus), and that all subsequent tabs/processes are created with just two execution threads to handle all of the the core rendering functions for that instance. It’s an interesting design, one likely made possible by Chrome’s modular architecture. And while it may not deliver the kind of massive parallelism that Microsoft seems to be shooting for with IE 8, it should serve to offset Chrome’s voracious appetite for memory. We shudder to think what Google’s browser would look like if it incorporated Microsoft’s overwhelming threading model.

Bottom Line: Chrome, like IE 8, is a browser designed with tomorrow’s hardware in mind. It’s use of a multi-process tabbing model – which, according to Google, helps isolate failures and protect complex web applications (like GMail or Google Docs) – means that it will always use more memory than Firefox, IE 7 and similar, single-process browsers. How such model will hold up under heavy use, especially on today’s hardware, remains to be seen.

Users may decide that the added protection of isolated tab processes is worth the cost in terms of browser “memory bloat.” However, Microsoft is also incorporating isolated tabs in IE 8, albeit on a smaller scale (just 6 process instances to host our 10 tabs – clearly not total isolation). And with its more granular threading model, Microsoft’s browser may hold an advantage as CPU core counts climb and customers embrace the next wave of PC platforms.

One thing’s for certain: The once sedate browser landscape just got a whole lot more interesting!

Read more...

Monday, September 1, 2008

Internet Explorer 8: Over 2x “Fatter” than Firefox

Internet Explorer 8 Beta 2 arrived last week amid great expectations and some heated controversy. The latter was due to Microsoft’s decision to adopt a standards-compliant rendering model that breaks a great many IE-specific web sites.

However, while various bloggers and pundits waxed breathlessly about all of the new bells and whistles (including the much-reported “porn mode”), we were busy putting Microsoft’s new browser under the microscope of the exo.repository.

Note: For more information on the exo.performance.network, or to register for your free Analysis Portal account, visit www.xpnet.com.

Over the course of several days, we evaluated IE 8 under both Windows XP (SP3) and Vista x86 (SP1) to determine how this major update behaves and what, if any, new burdens it might place on today’s over-taxed (in the case of Vista) Windows PCs.

What we found was another example of unchecked Microsoft  code “bloat,” complete with “shirt-bursting, waistline-stretching” memory consumption and the kind of CPU-hogging thread growth normally reserved for massively parallel server farms.

For example, on our Dell OptiPlex 745 (Core 2 Duo @ 2.66GHz) Vista/XP test bed with 2GB of RAM, IE 8 consumed just under 380MB of memory during a 10-site, multi-tab browsing scenario of popular general media, technical media and humor-related Web destinations (see bottom of this article for a list of sites).

By contrast, IE 7 consumed just under 250MB rendering this same workload, while Firefox 3.01 put both IE versions to shame by completing the same browsing scenario in just 159MB of RAM.

image
Figure 1 – Peak Browser Working Set Under XP/Vista

The story gets worse for IE 8 when you examine the number of threads spawned to complete the scenarios. Under Firefox, the count never exceed 29 concurrent threads. IE 7 spawned a hefty 65 execution threads, while IE 8 tried to choke the life out of the CPU with a massive 171 concurrent threads.

By comparison, the instance of Microsoft SQL Server 2005 x64 Edition that hosts the exo.repository rarely spins more than 100 concurrent threads – this on an 8-way, 8GB server machine servicing over 3000 actively monitored PCs.

image
Figure 2 – Peak Browser Thread Count Under XP/Vista

One area where IE 8 came out ahead of Firefox is in overall CPU utilization. Firefox consumed an average 33% Processor Time under XP (48% under Vista), while IE 8 took-up 22% of the CPU (33% under Vista) and IE 7 was the least aggressive at 13% (24% under Vista).

Considering how much faster Firefox is than IE 7/8 at displaying most web sites, the additional CPU cycles are understandable and likely attributable to a more efficient rendering engine employing fewer threads in exchange for faster linear performance.

Some observations:

  • With a massive working set approaching 20% of our test bed’s 2GB RAM configuration, IE 8 may be the long awaited “killer app” that drives customers towards 4GB+ systems and the 64-bit flavors of Windows Vista/7.
  • By greatly increasing the number of concurrent execution threads, and then spreading them out across multiple, discrete processes (in our case, 6 separate instances of iexplore.exe), Microsoft seems to be positioning IE 8 to take advantage of the greatly expanded core counts of future Intel and AMD CPUs – at the cost of overwhelming today’s single and dual-core PCs.
  • By delivering superior performance while maintaining a nearly 2x smaller memory and CPU footprint across the board, Firefox 3.01 proves once again that the open source community can often trump even the most well-funded commercial projects when it comes to delivering efficient, streamlined code.

Bottom Line: No matter how you slice the data, IE 8 represents a massive expansion of the baseline runtime requirements for Microsoft’s Internet Explorer web browser. Meanwhile, the Firefox folks continue to embarrass Microsoft by “doing it better” (including delivering superior performance and overall standards compliance) while consuming fewer hardware resources. How these efficiencies will play in the aforementioned multi-core future remains to be seen.

It may be that Microsoft’s focus on parallelism will reap dividends once the hardware catches up with the software - much like Vista has looked better as the Windows PC ecosystem has evolved to support its epically porcine girth. Such massively-threaded products will likely feel more at home on the 4, 8 and 16-core systems of tomorrow, but for right now they represent the worst kind of code bloat.


List of web sites visited as part of our test scenario:

Note: To ensure an accurate, rich-media experience, we installed both the Adobe Flash 9 and Microsoft SilverLight 2.0 (Beta) plug-ins under each tested browser/OS combination.

Read more...

Thursday, August 21, 2008

Tracking Service Pack Adoption – XP vs. Vista

One of the items we like to track within the exo.repository is the adoption rate for new OS service packs. The results serve as a kind of customer satisfaction survey for each new version of Windows.

Note: For more information on the exo.performance.network, or to register for your free Analysis Portal account, visit www.xpnet.com.

For example, when we take a look at Windows Vista, over the past 4 months the adoption rate for Service Pack 1 (i.e. the percentage of users with SP1 installed vs. the overall Vista installed base) grew from 69% at the end of April to over 86% by the end of July.

image
Figure 1 – SP1 vs. SP3 Adoption Rates 

By contrast, the adoption rate for Windows XP Service Pack 3 has been far less robust. In April, just 34% of users had deployed Service Pack 3 (or the beta version). And in July, that number had crept up to only 47% – less than half of the user base.

These numbers tell us two things:

  1. There was tremendous pent-up demand for Vista SP1, either due to customer dissatisfaction with the product or because users were convinced of its benefits from all of the media hype surrounding its release.

  2. No similar demand existed for Windows XP Service Pack 3, most likely because, overall, the majority of users are quite happy with Windows XP + Service Pack 2 (we found just a handful of users still running Service Pack 1).

Of course, there are exceptions to every rule. For Example, we’re finding that some of the IT organizations we work with on our commercial side – specifically, those in the financial services and investment banking sectors - are planning to address their need for more workstation memory capacity by deploying the 64-bit version of Windows XP, a platform to which Service Pack 3 doesn’t apply (the x64 edition of XP uses the Windows Server 2003 code base).

Regardless, what seems clear from these numbers is that Microsoft’s customers felt Windows Vista was in urgent need of a Service Pack 12-18 months after it was released, while those still using Windows XP were relatively happy with the platform and in no rush to patch it.

Read more...

Wednesday, March 5, 2008

Windows "Workstation" 2008 - Vista Done Right?

If you've been paying attention to the various industry news outlets you've no doubt come across the story about the Microsoft engineer advocating Windows Server 2008 as a "workstation" OS. According to him, if you make the right tweaks - installing the Desktop Experience feature, adding a few missing utilities, tuning the scheduler - you can turn Server 2008 into a fairly convincing Vista knock-off, one that's faster and more scalable than the original.

Curious, we decided to see for ourselves just how well Server 2008 stacks-up to Vista with Service Pack 1. To make the comparison as even as possible, we disabled all of the UI goodies on Vista (i.e. set the Visual Effects to "Adjust for Best Performance") and installed the Desktop Experience feature under Server 2008. We also enabled SuperFetch and the Indexing services on the Server 2008 installation (both are disabled by default) and adjusted the "Processor scheduling" option to favor Programs (i.e. the way it's set under Windows Vista).

For hardware, we reused our Dell XPS M1710 test bed (Core 2 Duo T7200 at 2GHz, 2GB RAM, 80GB 7200RPM disk) from our previous Vista testing projects. Both OS were configured to use the entire disk as a single partition, and we installed the same device drivers under each version.

The actual test scenarios involved a straight execution of the OfficeBench test script (in a 10-iteration loop) as well as a separate multi-process workload package featuring the ADO, MAPI and WMP Stress workload generation objects (executing continuously for 10 minutes in a 3x3x3 multi-instance configuration).

image

Given all the press surrounding Vista Service Pack 1 and the supposed parity of the SP1 and Server 2008 kernels, we were expecting to find little or no performance delta between the two platforms. So we were understandably surprised when repeated test runs showed Windows Server 2008 outperforming Windows Vista w/SP1 by a margin of 11-17%.

Clearly, there is more going on within Server 2008 than simply a few boot-time kernel switches. The very tangible performance disparity between our "Workstation" 2008 configuration and Vista, even with Service Pack 1 installed, shows that Microsoft is capable of squeezing more out of the shared "Windows 6.1" code base if/when they choose to do so.

As for what's dragging Vista down (the number of running processes and services was nearly identical across both OS and in each test scenario), that's a bit harder to define. Perhaps the Server 2008 team decided to eschew some of the more desktop-centric and/or consumer-focused (i.e. CPU cycle-sapping) features of the Vista core (DRM comes to mind). Regardless, now that we know how much better things could have been, it'll be that much harder to settle for the sluggishness and bloat of Windows Vista.

Our recommendation: If you have an MSDN account or otherwise have access to a Server 2008 license, check it out for yourself. You may find that Windows "Workstation" 2008 is the Windows Vista you've been waiting for all along.

Read more...

Wednesday, January 30, 2008

The Widgets are Coming!

Now for something totally different: Widgets!

What are widgets? Generically speaking, they're small mini-applets that run on your desktop or in a dashboard-type page or view. In the case of the exo.performance.network, widgets are small slices of functionality that have been broken-out from the main Analysis Portal site and presented as a stand-alone UI for a specific data type.

For example, we're working on a Systems Health widget that uses the Portal site's Report Card engine to generate daily and weekly summary charts of a system's overall CPU, Memory and I/O health. Here's a screenshot of the widget in action:

image

Figure 1 - The Systems Health Widget

We've also built a Process Monitoring widget that lets you keep track of collected metrics data for a specific process on a specific system. Here's a quick look at the widget monitoring SQL Server CPU utilization over a one week period:

image

Figure 2 - The Process Monitor Widget

Our goal is to produce a whole library of similarly styled widgets, each with a specific function that exposes part of the Portal's tools set. We'll likely offer these together as a kind of "roll-your-own-dashboard" kit where you can point and click your way to the ideal combination of charts and gauges.

And as always, we welcome your feedback. Tell us what you'd like to see in an "exo.widget" library and we'll see what we can cook-up together! :-)

Read more...

New Server (Yippee!)

We're pleased to announce that we've completed our move to the new server. The exo.performance.network is now hosted on a state of the art PowerEdge 2950 with 8x 2.66Ghz Xeon CPU Cores (2x E5430), 8GB of RAM and 1.2TB of storage in a six-disk RAID. It's a nice step-up from our previous "Prestonia" Xeon dinosaur and should give us the capacity to reach our goal of 10K users by summer.

Also, we switched co-location facilities. We dumped the poorly managed DigiPort Miami (don't go there!) in favor of a swanky new suite at Host.net in Boca. It's closer to our offices in Wellington and a truly high-class operation. We'll sleep better at night. :-)

Note: We're still tweaking and tuning the new config, so pardon any intermittent service disruptions. All of your accounts have been moved over and both portal access and tracker upload should work as before. Regardless, we'll do our best to keep the "dust" to a minimum - and as always, thanks for helping to make the exo.performance.network a success!

Read more...