6 secs, oh my god…

Great article, thanks for taking the time to do this.
I have a question about your methodology. Did you do all tests of a given browser consecutively, or switch browsers after each trial?

The tests were all run consecutively for each of the browsers. Even if we didn’t run them consecutively and CNN got hit with a surge of traffic all of a sudden that would essentially render results prior to the “surge” inaccurate compared to the others.

The only way that you can really overcome this problem is to do the testing on a local network where you can control the fluctuations in traffic. But if we would have done that then people would have said that this isn’t a “real world” test.

I’m curious why you thought the Sunspider benchmark is biased. I’ve seen stats from Apple, Mozilla, and Microsoft using it for javascript performance. (Apple developed it, Mozilla has done a ton of blogs on their speed on it, Microsoft used it at their presentation at Mix, dunno about Opera).

It’s not me, it’s some of our readers. The last time we used the Sunspider test we received several comments and emails stating that we shouldn’t have used a test developed by one of the development companies that create a browser.

Before commenting on that I did some quick tests on an XP machine that I have access to, and the results were proportionally the same. Meaning the ranks did really seem to differ.

Trying to find a JavaScript test that all of the browsers successfully complete is not an easy task. Even the Sunspider test has quite a few troubles with Opera.

The point of the graphs is not to try and sit there to interpret the values. If we would have not provided the detailed data with each test then we would have done a 2D graph, but all the graphs are there to represent is how large the differences are between some of the browsers. Again, it’s tough to please everyone on our comparisons which is one reason I think most sites refrain from doing them.

That’s true, and we’ve done an article on how Vista and XP report the memory usage differently.

Ryan wrote:

Steve wrote:

Any chance of WinXP stats, since most people are still on this OS?

I don’t think we’re going to rerun all of these tests on XP because for the most part the browsers are the same for XP. It wouldn’t really be worth spending another 6 hours testing the browsers when the results would still be pretty similar.

Do you know for a fact they will be similar, or is that an assumption? If it’s an assumption, then I know a saying about that

Actually I think it is much better to test on Vista rather than XP, if only for the memory tests. XP’s way of counting howmuch memory is being used pretty much sucks, and they are certainly not good enough for intensive testing like this. Vista’s way is actually IMO the best of any platform out there (though I don’t really know about Linux).

Steve wrote:

Any chance of WinXP stats, since most people are still on this OS?

I don’t think we’re going to rerun all of these tests on XP because for the most part the browsers are the same for XP. It wouldn’t really be worth spending another 6 hours testing the browsers when the results would still be pretty similar.

Do you know for a fact they will be similar, or is that an assumption? If it’s an assumption, then I know a saying about that
Also, you write “Pretty much none of the browsers were able to complete all of the tests error-free, and so we’re focusing purely on the speed”. Isn’t this also known as “If you don’t care what the result is, i can do it really fast”? Could you somehow indicate how accurate each browser was (how many errors were thrown)?

Anyway good job on this. Always nice to read a reasonably well executed comparison article. But as a final point I must berate you for using 3D graphs. They don’t look pretty and they make it harder to read and compare values. The only people who use such 3D graphs are marketing people and 1st year science students (and the students soon learn they’ll get laughed out the room if they do such a thing again :))

I actually think that the Slickspeed is more biased because it has more variables than just pure javascript speed - like whether certain selectors are implemented natively in the browser or have to be run through the library (ie. nth selectors). Really I think that it has more to do with comparing the speeds of the library rather than browser.

Although I certainly do agree with you that there is a surprising lack of tools to measure pageload. I mean, that’s about the most important performance there is, right?

Did you know it’s possible to have IE7 in stand alone mode? If not check out
http://weblogs.asp.net/jgallow.....2900_.aspx

Maybe this will with some of the testing so you don’t have to go back and forth so much.

Huh, didn’t think about that. But I wonder whether that would affect the performance? That’s one reason why I didn’t want to run portable versions of any of the browsers as well.

They do vary quite a bit, and that’s why I ran each browser three times and averaged the results. Generally, however, each of the results were rather close to each other.

Glad it helped you! I normally have a significant number of tabs open as well, and that’s why I really wanted to test out the memory usage.

I’ll definitely have to think about that, but I’m sure this is an article that I’ll keep updating. I’ll create a new post on the site when I do decide to update it.

Thanks for putting the effort into this little project - I’m definitely going to take on board what these results say - IE8 is getting uninstalled (am sick of it crashing, even with less than 5 tabs open) and am going to take a proper look at Firefox 3 Beta, it’s looking good from those memory stats - I typically end up with around 100 tabs open at times!

Since we had already done the Sunspider tests we didn’t want to do them again, while it may seem unfair to mix the test results from the libraries I believe that it makes sense. Many Web 2.0 sites use a mixture of those libraries, but regardless it would be difficult to put together a set of test results that everyone is content with.

well, fair enough
PS, I just did the test with safari 3.1 and firefox beta5 RC1, it comes out 715ms vs 834ms. and each browser varies alot when test multiple times.

Maybe this will with some of the testing so you don’t have to go back and forth so much.

Any chance of WinXP stats, since most people are still on this OS?

I don’t think we’re going to rerun all of these tests on XP because for the most part the browsers are the same for XP. It wouldn’t really be worth spending another 6 hours testing the browsers when the results would still be pretty similar.

Since we had already done the Sunspider tests we didn’t want to do them again, while it may seem unfair to mix the test results from the libraries I believe that it makes sense. Many Web 2.0 sites use a mixture of those libraries, but regardless it would be difficult to put together a set of test results that everyone is content with.

FX3b4 does win one or 2 out of 4 library tests.Its quite unfair to mix 4 libraries together since they are not used with same frequency.

Finally, I don’t think slickspeed is particularly better than sunspider, slickspeed are testing more about theoretical stuff, sunspider does more about day to day practical usage.