Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2

Author Topic: Browser Benchmarks v 3- March 2013  (Read 10944 times)

snadge

  • Kitizen
  • ****
  • Posts: 1450
Browser Benchmarks v 3- March 2013
« on: March 17, 2013, 12:22:12 AM »

EDIT: updated to v2 - 3 extra browsers and 2 extra benchmarks added, marked in Blue in lists - some new notes added too and new conclusion & winners and overall total scores per browser. Also added a closing note and post from Mr. M.C. Straver (the creator of Palemoon, another Firefox fork) which explains why benchmarks should be taken as partial indication and not overall result and he explains why 64-Bit performs worse than 32-Bit

I haven't done a round of bench-marking for a while so thought I would spend a few hours on a Saturday night several hours over the weekend running bench's on all the popular browsers AND 64-Bit iterations for your perusal - nothing special, just a quick zip through a few bench-marking tools. (I say 'quick zip' yet it took me two long evenings to complete & compile!!)

BROWSERS TESTED:
  • 32-Bit Browsers:
    - Firefox 19.0.1
    - Chrome 25.0.1364.172 m
    - Opera 12.14
    - Internet Explorer 9.0.8112.16421
    - Internet Explorer 10.0.9200.16521
    - Safari 5.1.7
  • 64-Bit Browsers:
    - Firefox 20.0A (Nightly Build)
    - Waterfox 18.0.1
    - Internet Explorer 9.0.8112.16421
    - Opera 12.14

BENCHMARK SOFTWARE USED:
Peacekeeper Browser Benchmark v2
BrowserMark 2.0
V8 Benchmark v7
SunSpider JavaScript Test 0.9.1 (lower score is better)
RoboHornet Alpha v1
Club Compy Real World Benchmarking



All tests run with no browser add-ons running and no background applications active in Windows 7 Pro x64 on a 3.6Ghz Ivy-Bridge Quad-Core CPU, 8Gb (2x4) 1333Mhz Crucial Memory and High Performance Samsung 840 SSD.

NOTE 1: IE10 64-Bit tests not included as unabled to enable 64-bit mode IE in Windows 7x64, the 'enable enhanced protected mode' workaround still shows processes for Tabs run in 32-Bit in Task manager, therefore not 64-Bit!

NOTE 2: I didn't include Mozilla's Kraken Benchmark because while Chrome flew through it, both Firefox and IE9 were taking so long I just couldn't bare to wait so cancelled them

NOTE 3: RoboHornet uses an 'index' scoring method, they weigh out scores across all tests so that an overall 'index' score of 100 is the 'average' performance of todays browsers, below 100 and its performing less than average and over 100 its performing more than average.


:RESULTS:

Click HERE to see larger version



CONCLUSION:
Well it really is a one horse race with Google Chrome taking the lead, IE10 and Opera 32/64 not far behind, Safari performing better than IE9 and all versions of Firefox be it 32-Bit, 64-Bit or Waterfox Project's 64-Bit fork...  these tests show Firefox too be the bottom feeder, especially the Waterfox version which performed the worst in all but 1 of the tests.

DOES 64-bit ENHANCE PERFORMANCE...?
In a word...NO, nearly all 64-Bit browsers performed worse than their 32-Bit counterparts apart from Opera which is more or less the same, better in some and not in others, borderline!  ....so no 'real gain' in performance there by using 64-Bit!

WINNERS TABLE
(Scores calculated by adding them all together and subtracting the SunSpider scores)

OUR WINNER - GOOGLE CHROME   8) 8) 8) 8) 8)
Score = 60,188
2nd Place - INTERNET EXPLORER 10  :D :D :D :D
Score = 44,316
3rd Place - OPERA 32/64    ;D ;D ;D
x32 Score = 28,926
x64 Score = 28,243
4th Place - SAFARI 5    :) :) :)
Score = 27,148
5th Place - INTERNET EXPLORER 9 x32   ??? ???
Score = 15,990
6th Place -  FIREFOX x32 :(
Score = 12,522
7th Place -  INTERNET EXPLORER 9 x64 :(
Score = 11,439
8th Place -  FIREFOX x64 :(
Score = 11,300
LOSER - WATERFOX x64  :'(
Score = 10,593


CLOSING NOTE:
I would like you to read the following post from Mr. M.C. Straver who is the creator of Palemoon (another Firefox (x32) fork) in which he states that these benchmark tools are merely an 'indication' and cant really be used to paint an overall picture of the browsers performance due to 'tight loop testing' and other issues, he also explains how 64-Bit browsers will perform worse than 32-Bit during benchmarks too... its a good read and I tend to believe what he says, however I can definitely say that after moving from Firefox 19 to Chrome 25 I can definitley feel the extra speed which is reflected in the results above.

http://forum.palemoon.org/viewtopic.php?f=24&t=650
Quote
What's the deal with browser benchmarks?

Short answer: They aren't always conclusive. They can be biased, incomplete, and should only be seen as an indication and not as hard fact.

(Very) long answer:

1) Definitions

I'll be using a few terms in this answer that may not be immediately clear to people:
JS: Short for "JavaScript", the universal scripting language that is used in webpages
JIT, and jitted: JIT is a "Just-In-Time" compiler, which converts text-based scripting into a machine code equivalent when it is encountered for (much) faster execution. If a function is "jitted", this means that code has been converted this way and isn't "interpreted"
Pure JS: This stands for "Pure JavaScript", and with this I mean the functions of JS that are most likely to be jitted like math operations, bitwise operations, etc.
DOM: the Document Object Model, and object-oriented structure that is the meat and potatoes of dynamic webpages

2) The Benchmarks

To know what exactly is tested and how it's tested, a closer look at the different popular benchmarks first:
Sunspider, Kraken and V8: These test almost exclusively Pure JS
Dromaeo: A benchmark that tests a mix of Pure JS, interpreted JS and DOM/CSS
Peacekeeper (Futuremark): A benchmark that tests a mix of JS, DOM and graphical elements

3) What is tested and what is not tested

All of the benchmarks rely very heavily on JS and its execution. Although JS is very important for modern webpages, it is certainly far from the end-all of things. just as important are the speeds at which DOM and CSS are handled, or how HTML is parsed by the browser. In addition, graphic rendering speed and network speed and buffering are important, as well as how efficiently the browser handles its memory. As a result, all of the current browser benchmarks out there don't provide you with a full picture of how a browser performs overall. Sunspider, Kraken and V8 can be considered the least interesting benchmarks because they only really focus on the JIT part of JS, and don't look at all at any of the other parts of the browser. Dromaeo is a bit better, but still a JS focused test. Peacekeeper extends the range a bit further by actually adding some rendering tests, although this benchmark has its own set of issues like the lack of statistical confidence and relying very heavily on hardware (and as such more of a hardware test than a browser test), and it still doesn't include HTML parsing, networking tests, memory handling.

This means that all of these benchmarks only give a partial image.

4) 32-bit versus 64-bit and tight loops

Benchmarks invariably perform their tests in "tight loops", which means a small bit of code that is looped through rapidly many times. If properly calibrated (meaning looping through it without performing anything to get a "reference" value) this kind of tight loop can quite efficiently measure how well a specific, single instruction is executed. This is, however, not the kind of behavior you'd encounter when browsing webpages, where you would normally have a large number of different instructions one after the other. Testing in tight loops may therefore not give you any sort of conclusive result; it can only be seen as an indication of what overall browser speed could be if not influenced by other factors.

This also brings me to testing 32-bit and 64-bit browsers against each other. Because of these tight loops and functions in a browser usually not using 64-bit address space or big variables, the benchmarks won't specifically test what a 64-bit browser is strong at. Tight loop testing also brings in another factor: because a 64-bit processor uses twice as large registers, you are inherently pushing twice as much data through your hardware. Tight loops are specifically sensitive to the amount of data that is passed to and from the processor, and you should always take this into account when comparing test results between a 32-bit and 64-bit browser. You can expect 64-bit browsers to score (quite a bit) lower than their 32-bit counterparts; this doesn't mean that the browser is slower, though, just that the test isn't measuring the full capacity of the browser.

5) Bias, bugs, and cheating

Benchmarks can be (horribly) biased to favor one particular browser. This is especially the case if the benchmark is created and operated by the people who also put out the browser to be tested (or are affiliated with) - it's not scientifically sound to write a test to prove your own theory, and writing a test to prove your browser faster than any other out there isn't too hard if you bias the test specifically to what you know your browser's strengths are. This is the reason why, even if I could, I have not written a benchmark myself; being the Pale Moon developer, I would be influenced in writing my benchmark by my in-depth knowledge of the code and optimizations, or be accused of such easily enough even if I write it fully objectively.

Benchmarks or browsers can also have bugs that cause the measured results to be wrong. You are inherently using the browser's internal scripting to measure the performance of the browser. You can't rely on using the very thing you test to provide unbiased, objective results.

When building a browser, especially when using profile-guided optimization (PGO, a technique that builds the browser, then lets you run it while recording "normal use" of the browser, and fine-tuning building to that by biasing the final build towards the functions that are actually seen used), you can also cheat benchmarks by making sure the specific functions used in benchmarking are compiled to execute as fast as possible, at the expense of all other functions of the browser. For example, if a build would be made where the "profiling run" is used to do nothing but mathematical calculations, then the final build would be exceedingly good at them, but at the expense of, e.g. rendering graphics or DOM or CSS.

6) Conclusion

The conclusion is that benchmarks can't be used to draw hard conclusions. Plain and simple: they are an indication, nothing more. They serve well if you compare siblings (e.g. Firefox and Pale Moon) to get a relative performance difference between the two on the limited subset of what is actually tested, but that's about as far as it goes.
« Last Edit: March 18, 2013, 10:38:04 PM by snadge »
Logged
Aquiss - 900/110/16ms - TP-Link AR73

sheddyian

  • Kitizen
  • ****
  • Posts: 1159
    • My Shed Blog
Re: Browser Benchmarks - March 2013
« Reply #1 on: March 17, 2013, 01:00:24 PM »

Interesting, thanks!

Chrome is my browser of choice on Windows XP and 7, as I find it nice and quick.

That said, I've just installed IE10 on my Windows 7 machine, and it seems faster than Chrome on the same platform!  I'd be interested in your results if you update to IE10.

My observations on IE10 are that it feels snappier at loading and rendering pages, opening new tabs etc, and it does a better job of playing full screen HD video than Chrome on my machine.

However, when I run the speedof.me HTML5 broadband speed checker, IE10 performs very badly and claims my ADSL line is managing only 4Meg/second download, whereas Chrome gives me a result that seems realistic (~16 meg/second)

Ian
Logged

burakkucat

  • Respected
  • Senior Kitizen
  • *
  • Posts: 38300
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: Browser Benchmarks - March 2013
« Reply #2 on: March 17, 2013, 02:20:08 PM »

Are there 64-bit versions of Chrome and Opera available? I would certainly like to see a similar comparison as to that you have perfomed for the 32-bit versions, Snadge:-\
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

snadge

  • Kitizen
  • ****
  • Posts: 1450
Re: Browser Benchmarks - March 2013
« Reply #3 on: March 17, 2013, 02:54:40 PM »

theres no 64-Bit versions of Chrome and I wasnt aware there was a 64-Bit version of Opera (there is, just checked) - as for IE10 I thought you could only get that with Windows 8 but appears not.

So... I will later add IE10 and Opera 64-Bit too the tests, I will also add some more benchmarks if anyone cares to point me to some good ones?  see if cant even things out a bit - I read on Toms Hardware recent tests that they didnt use V8 newer benchmark to the one I used because it "skews the results in favour of chrome" so just wondering if V8 Benchmark v7 is doing the same - that said I think it would still score high and still get its 1st place spot as it does on Toms Hardware...but they have benchmarks that I cant get a hold of without paying that put Firefox in second place

it was only meant to be a quick whizz through some bench's - not deep diagnosis but I wouldnt mind getting some more benchmarking tools and browsers too add to it, I will do that

I could of possibly ran each test 3 times to get an 'average' but i would have been on all night... took over 2 hours to do that lot.
« Last Edit: March 17, 2013, 02:57:11 PM by snadge »
Logged
Aquiss - 900/110/16ms - TP-Link AR73

sheddyian

  • Kitizen
  • ****
  • Posts: 1159
    • My Shed Blog
Re: Browser Benchmarks - March 2013
« Reply #4 on: March 17, 2013, 03:03:23 PM »

theres no 64-Bit versions of Chrome and I wasnt aware there was a 64-Bit version of Opera (there is, just checked) - as for IE10 I thought you could only get that with Windows 8 but appears not.

IE10 for Windows 7 hasn't been out very long - end of February I think, and unusually, Microsoft don't seem to have made a big deal about it.  Still keen to migrate customers from 7 to 8 perhaps!

Ian
Logged

HPsauce

  • Helpful
  • Kitizen
  • *
  • Posts: 2606
Re: Browser Benchmarks - March 2013
« Reply #5 on: March 17, 2013, 03:09:07 PM »

They'll be pushing IE10 out fairly soon, it's now showing as "Important" in my Windows Update whereas it was Optional at first.  ;)
Funny thing is I thought it needed the Platform Update (KB2670838 which has been reported as having a few issues in some situations) but that's only showing as Optional....
Logged

sheddyian

  • Kitizen
  • ****
  • Posts: 1159
    • My Shed Blog
Re: Browser Benchmarks - March 2013
« Reply #6 on: March 17, 2013, 03:25:34 PM »

They'll be pushing IE10 out fairly soon, it's now showing as "Important" in my Windows Update whereas it was Optional at first.  ;)
Funny thing is I thought it needed the Platform Update (KB2670838 which has been reported as having a few issues in some situations) but that's only showing as Optional....

Interesting - I hadn't seen it even listed as "optional", I had to go hunting for it on the MS site when I downloaded - think it was only about 7 - 10 days ago.

I assume I picked up the Platform Update or it got installed as part of the IE10 install, as it's not listed when I check for updates, though I've just found a security update I somehow missed!

Ian


Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33881
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Browser Benchmarks - March 2013
« Reply #7 on: March 17, 2013, 09:35:54 PM »

Thank you for doing this and posting the results.  Very interesting.

I too would be interested to see how ie10 performs.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33881
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Browser Benchmarks - March 2013
« Reply #8 on: March 17, 2013, 09:45:31 PM »

I'd also be interested if possible if there is such a benchmark test.. in how the browsers perform with flash.  From past experience I find that some browsers tend to cope much better than others with apps that use flash. 
 Without the lack of proper benchmarking, and observation only, I'd say that chrome seemed far superior, whilst (last yrs) opera seemed pretty pathetic and continually froze or crashed.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

snadge

  • Kitizen
  • ****
  • Posts: 1450
Re: Browser Benchmarks - March 2013
« Reply #9 on: March 17, 2013, 10:41:20 PM »

Iam busy doing the extra tests now using IE10 (32/64), Opera 64-Bit

I have included RoboHornet and ClubCompy Real World Benchmarking in the benchmarks (currently putting browsers through these at moment) - I was also going to include Mozilla's Kraken but while chrome zoomed through it, Firefox and IE where taking soooo looonnngg I just couldnt bare it lol... i mean twenty-thirty times slower (so it seemed), anyway I have laptop to use while testing so MIGHT run it - some of these tests take 5-10 minutes each, a couple of them take just seconds but times that by 10 browsers not included logging data into Excel etc...takes a while!!

I will let you know when I have the results...should be in next 2 hours (I hope!!)
Logged
Aquiss - 900/110/16ms - TP-Link AR73

asbokid

  • Kitizen
  • ****
  • Posts: 1286
    • Hacking the 2Wire
Re: Browser Benchmarks - March 2013
« Reply #10 on: March 17, 2013, 11:03:09 PM »

Excellent experiments! Look forward to seeing the results and conclusion, snadge   :)

cheers, a
Logged

snadge

  • Kitizen
  • ****
  • Posts: 1450
Re: Browser Benchmarks - March 2013
« Reply #11 on: March 18, 2013, 02:15:25 AM »

unfortunately I can no longer edit my post? seems there is a time limit on kitz for editing posts, hopefully Kitz will swap out my first post for the code below...I have added in the new browsers, benchmarks & results table, also added some notes thats worth reading and changed the conclusion and winners table
...

EDIT: updated to v2 - 3 extra browsers and 2 extra benchmarks added, marked in Blue in lists - some new notes added too and new conclusion & winners and overall total scores per browser. Also added a closing note and post from Mr. M.C. Straver (the creator of Palemoon, another Firefox fork) which explains why benchmarks should be taken as partial indication and not overall result and he explains why 64-Bit performs worse than 32-Bit

I haven't done a round of bench-marking for a while so thought I would spend a few hours on a Saturday night several hours over the weekend running bench's on all the popular browsers AND 64-Bit iterations for your perusal - nothing special, just a quick zip through a few bench-marking tools. (I say 'quick zip' yet it took me two long evenings to complete & compile!!)

BROWSERS TESTED:
  • 32-Bit Browsers:
    - Firefox 19.0.1
    - Chrome 25.0.1364.172 m
    - Opera 12.14
    - Internet Explorer 9.0.8112.16421
    - Internet Explorer 10.0.9200.16521
    - Safari 5.1.7
  • 64-Bit Browsers:
    - Firefox 20.0A (Nightly Build)
    - Waterfox 18.0.1
    - Internet Explorer 9.0.8112.16421
    - Opera 12.14

BENCHMARK SOFTWARE USED:
Peacekeeper Browser Benchmark v2
BrowserMark 2.0
V8 Benchmark v7
SunSpider JavaScript Test 0.9.1 (lower score is better)
RoboHornet Alpha v1
Club Compy Real World Benchmarking



All tests run with no browser add-ons running and no background applications active in Windows 7 Pro x64 on a 3.6Ghz Ivy-Bridge Quad-Core CPU, 8Gb (2x4) 1333Mhz Crucial Memory and High Performance Samsung 840 SSD.

NOTE 1: IE10 64-Bit tests not included as unabled to enable 64-bit mode IE in Windows 7x64, the 'enable enhanced protected mode' workaround still shows processes for Tabs run in 32-Bit in Task manager, therefore not 64-Bit!

NOTE 2: I didn't include Mozilla's Kraken Benchmark because while Chrome flew through it, both Firefox and IE9 were taking so long I just couldn't bare to wait so cancelled them

NOTE 3: RoboHornet uses an 'index' scoring method, they weigh out scores across all tests so that an overall 'index' score of 100 is the 'average' performance of todays browsers, below 100 and its performing less than average and over 100 its performing more than average.


:RESULTS:

Click HERE to see larger version



CONCLUSION:
Well it really is a one horse race with Google Chrome taking the lead, IE10 and Opera 32/64 not far behind, Safari performing better than IE9 and all versions of Firefox be it 32-Bit, 64-Bit or Waterfox Project's 64-Bit fork...  these tests show Firefox too be the bottom feeder, especially the Waterfox version which performed the worst in all but 1 of the tests.

DOES 64-bit ENHANCE PERFORMANCE...?
In a word...NO, nearly all 64-Bit browsers performed worse than their 32-Bit counterparts apart from Opera which is more or less the same, better in some and not in others, borderline!  ....so no 'real gain' in performance there by using 64-Bit!

WINNERS TABLE
(Scores calculated by adding them all together and subtracting the SunSpider scores)

OUR WINNER - GOOGLE CHROME   8) 8) 8) 8) 8)
Score = 60,188
2nd Place - INTERNET EXPLORER 10  :D :D :D :D
Score = 44,316
3rd Place - OPERA 32/64    ;D ;D ;D
x32 Score = 28,926
x64 Score = 28,243
4th Place - SAFARI 5    :) :) :)
Score = 27,148
5th Place - INTERNET EXPLORER 9 x32   ??? ???
Score = 15,990
6th Place -  FIREFOX x32 :(
Score = 12,522
7th Place -  INTERNET EXPLORER 9 x64 :(
Score = 11,439
8th Place -  FIREFOX x64 :(
Score = 11,300
LOSER - WATERFOX x64  :'(
Score = 10,593


CLOSING NOTE:
I would like you to read the following post from Palemoon Creator (another Firefox (x32) fork) in which he states that these benchmark tools are merely an 'indication' and cant really be used to paint an overall picture of the browsers performance due to 'tight loop testing' and other issues, he also explains how 64-Bit browsers will perform worse than 32-Bit during benchmarks too... its a good read and I tend to believe what he says, however I can definitely say that after moving from Firefox 19 to Chrome 25 I can definitley feel the extra speed which is reflected in the results above...

http://forum.palemoon.org/viewtopic.php?f=24&t=650
Quote
What's the deal with browser benchmarks?

Short answer: They aren't always conclusive. They can be biased, incomplete, and should only be seen as an indication and not as hard fact.

(Very) long answer:

1) Definitions

I'll be using a few terms in this answer that may not be immediately clear to people:
JS: Short for "JavaScript", the universal scripting language that is used in webpages
JIT, and jitted: JIT is a "Just-In-Time" compiler, which converts text-based scripting into a machine code equivalent when it is encountered for (much) faster execution. If a function is "jitted", this means that code has been converted this way and isn't "interpreted"
Pure JS: This stands for "Pure JavaScript", and with this I mean the functions of JS that are most likely to be jitted like math operations, bitwise operations, etc.
DOM: the Document Object Model, and object-oriented structure that is the meat and potatoes of dynamic webpages

2) The Benchmarks

To know what exactly is tested and how it's tested, a closer look at the different popular benchmarks first:
Sunspider, Kraken and V8: These test almost exclusively Pure JS
Dromaeo: A benchmark that tests a mix of Pure JS, interpreted JS and DOM/CSS
Peacekeeper (Futuremark): A benchmark that tests a mix of JS, DOM and graphical elements

3) What is tested and what is not tested

All of the benchmarks rely very heavily on JS and its execution. Although JS is very important for modern webpages, it is certainly far from the end-all of things. just as important are the speeds at which DOM and CSS are handled, or how HTML is parsed by the browser. In addition, graphic rendering speed and network speed and buffering are important, as well as how efficiently the browser handles its memory. As a result, all of the current browser benchmarks out there don't provide you with a full picture of how a browser performs overall. Sunspider, Kraken and V8 can be considered the least interesting benchmarks because they only really focus on the JIT part of JS, and don't look at all at any of the other parts of the browser. Dromaeo is a bit better, but still a JS focused test. Peacekeeper extends the range a bit further by actually adding some rendering tests, although this benchmark has its own set of issues like the lack of statistical confidence and relying very heavily on hardware (and as such more of a hardware test than a browser test), and it still doesn't include HTML parsing, networking tests, memory handling.

This means that all of these benchmarks only give a partial image.

4) 32-bit versus 64-bit and tight loops

Benchmarks invariably perform their tests in "tight loops", which means a small bit of code that is looped through rapidly many times. If properly calibrated (meaning looping through it without performing anything to get a "reference" value) this kind of tight loop can quite efficiently measure how well a specific, single instruction is executed. This is, however, not the kind of behavior you'd encounter when browsing webpages, where you would normally have a large number of different instructions one after the other. Testing in tight loops may therefore not give you any sort of conclusive result; it can only be seen as an indication of what overall browser speed could be if not influenced by other factors.

This also brings me to testing 32-bit and 64-bit browsers against each other. Because of these tight loops and functions in a browser usually not using 64-bit address space or big variables, the benchmarks won't specifically test what a 64-bit browser is strong at. Tight loop testing also brings in another factor: because a 64-bit processor uses twice as large registers, you are inherently pushing twice as much data through your hardware. Tight loops are specifically sensitive to the amount of data that is passed to and from the processor, and you should always take this into account when comparing test results between a 32-bit and 64-bit browser. You can expect 64-bit browsers to score (quite a bit) lower than their 32-bit counterparts; this doesn't mean that the browser is slower, though, just that the test isn't measuring the full capacity of the browser.

5) Bias, bugs, and cheating

Benchmarks can be (horribly) biased to favor one particular browser. This is especially the case if the benchmark is created and operated by the people who also put out the browser to be tested (or are affiliated with) - it's not scientifically sound to write a test to prove your own theory, and writing a test to prove your browser faster than any other out there isn't too hard if you bias the test specifically to what you know your browser's strengths are. This is the reason why, even if I could, I have not written a benchmark myself; being the Pale Moon developer, I would be influenced in writing my benchmark by my in-depth knowledge of the code and optimizations, or be accused of such easily enough even if I write it fully objectively.

Benchmarks or browsers can also have bugs that cause the measured results to be wrong. You are inherently using the browser's internal scripting to measure the performance of the browser. You can't rely on using the very thing you test to provide unbiased, objective results.

When building a browser, especially when using profile-guided optimization (PGO, a technique that builds the browser, then lets you run it while recording "normal use" of the browser, and fine-tuning building to that by biasing the final build towards the functions that are actually seen used), you can also cheat benchmarks by making sure the specific functions used in benchmarking are compiled to execute as fast as possible, at the expense of all other functions of the browser. For example, if a build would be made where the "profiling run" is used to do nothing but mathematical calculations, then the final build would be exceedingly good at them, but at the expense of, e.g. rendering graphics or DOM or CSS.

6) Conclusion

The conclusion is that benchmarks can't be used to draw hard conclusions. Plain and simple: they are an indication, nothing more. They serve well if you compare siblings (e.g. Firefox and Pale Moon) to get a relative performance difference between the two on the limited subset of what is actually tested, but that's about as far as it goes.
« Last Edit: March 18, 2013, 04:59:13 AM by snadge »
Logged
Aquiss - 900/110/16ms - TP-Link AR73

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43472
  • Penguins CAN fly
    • DSLstats
Re: Browser Benchmarks - March 2013
« Reply #12 on: March 18, 2013, 07:25:17 AM »

Excellent work snadge. I've added an update note to your original posting.
Logged
  Eric

guest

  • Guest
Re: Browser Benchmarks - March 2013
« Reply #13 on: March 18, 2013, 05:15:52 PM »

I have to admit Firefox on Win7 is getting to the annoying stage in terms of updates every week or so and whoever coded the handling for flash plugins is a moron - WTF were you thinking of when you allowed the plugin to completely hog the main FF thread upon deletion of temporary DRM-protected data? Its rare that I wish to inflict bodily harm on someone coding free software but you are the exception to the rule matey.

About all that keeps me using FF is NoScript and AdBlock+

Sad days when you have to consider MS or an advertising company for your browser :(
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33881
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Browser Benchmarks - March 2013
« Reply #14 on: March 18, 2013, 08:27:55 PM »

Thank you snadge for adding the additional results.   Ive updated your first post as requested.

whoever coded the handling for flash plugins is a moron -

My thoughts exactly, it doesnt seem to handle flash well at all and frequently crashes, hence my earlier remarks about a flash benchmark.

Once upon a time firefox was my browser of choice, but these days its too clunky :(


-----------------
Edited to add

For want of a better flash benchmark I tried this
http://themaninblue.com/experiment/AnimationBenchmark/flash/

Firefox gives me 38-40 FPS
Chrome gives me  54-56 FPS
« Last Edit: March 18, 2013, 08:47:41 PM by kitz »
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker
Pages: [1] 2