MozillaWiki

Platform/Games/Testing/32v64bit: Difference between revisions

< Platform‎ | Games‎ | Testing
 
(20 intermediate revisions by 2 users not shown)
Line 18: Line 18:
  |-
  |-
  | Firefox
  | Firefox
  | (user agent string with build ID) Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202
  | (user agent string with build ID) Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202
  |-
  |-
  | Processor
  | Processor
Line 24: Line 25:
  |-
  |-
  | Memory
  | Memory
  | (type, speed, amount) 8 GB DDR3 1033 Mhz
  | (type, speed, amount) 8 GB DDR3 1333 Mhz
  |-
  |-
  | GPU
  | GPU
Line 43: Line 44:
  | main-thread-poppler-cold
  | main-thread-poppler-cold
  | 0.452
  | 0.452
  | 0.729 (±22%!)
  | '''0.348'''
  |-
  |-
  | main-thread-poppler-warm  
  | main-thread-poppler-warm  
  | 0.428
  | 0.428
  | 0.523
  | '''0.372'''
  |-
  |-
  | main-thread-sqlite-cold
  | main-thread-sqlite-cold
  | 0.053 (±26%!)
  | '''0.053''' (±26%!)
  | 0.872
  | 0.062 (±25%!)
  |-
  |-
  | main-thread-sqlite-warm
  | main-thread-sqlite-warm
  | 0.033 (±24%!)
  | '''0.033''' (±24%!)
  | 0.043 (±27%!)
  | 0.048 (±29%!)
  |-
  |-
  | colspan="3" class="header" | Throughput
  | colspan="3" class="header" | Throughput
Line 61: Line 62:
  | box2d-throughput  
  | box2d-throughput  
  | 6.838
  | 6.838
  | 4.623
  | '''4.882'''
  |-
  |-
  | box2d-throughput-f32  
  | box2d-throughput-f32  
  | 6.316
  | 6.316
  | 4.258
  | '''4.567'''
  |-
  |-
  | lua-binarytrees  
  | lua-binarytrees  
  | 8.067
  | 8.067
  | 7.271
  | '''7.447'''
  |-
  |-
  | lua-scimark  
  | lua-scimark  
  | 10.545
  | 10.545
  | 12.160
  | '''11.730'''
  |-
  |-
  | poppler-throughput  
  | poppler-throughput  
  | 7.008
  | 7.008
  | 5.988
  | '''6.445'''
  |-
  |-
  | sqlite-throughput  
  | sqlite-throughput  
  | 10.891
  | 10.891
  | 8.404
  | '''8.773'''
  |-
  |-
  | colspan="3" class="header" | Preparation
  | colspan="3" class="header" | Preparation
Line 87: Line 88:
  | poppler-cold-preparation  
  | poppler-cold-preparation  
  | 0.939
  | 0.939
  | 0.864
  | '''0.926'''
  |-
  |-
  | poppler-warm-preparation  
  | poppler-warm-preparation  
  | 0.185
  | 0.185
  | 0.183
  | 0.185
  |-
  |-
  | sqlite-cold-preparation  
  | sqlite-cold-preparation  
  | 1.084
  | '''1.084'''
  | 1.260
  | 1.347
  |-
  |-
  | sqlite-warm-preparation  
  | sqlite-warm-preparation  
  | 0.098
  | '''0.098'''
  | 0.123
  | 0.123
  |-
  |-
Line 105: Line 106:
  | box2d-variance  
  | box2d-variance  
  | 1.466
  | 1.466
  | 1.216
  | '''1.261'''
  |-
  |-
  | poppler-variance  
  | poppler-variance  
  | 4.843
  | 4.843
  | 4.074
  | '''4.395'''
  |-
  |-
  | class="header" | Overall Score
  | class="header" | Overall Score
  | class="header" | 3,967
  | class="header" | 3,967
  | class="header" | 3,377
  | class="header" | 4,120
  |}
  |}


Line 126: Line 127:
  |-
  |-
  | main-thread-poppler-cold
  | main-thread-poppler-cold
  |
  | 0.461
  | 0.367
  | '''0.367'''
  |-
  |-
  | main-thread-poppler-warm  
  | main-thread-poppler-warm  
  |
  | 0.427
  | 0.361
  | '''0.361'''
  |-
  |-
  | main-thread-sqlite-cold
  | main-thread-sqlite-cold
  |
  | 0.065
  | 0.064
  | '''0.064'''
  |-
  |-
  | main-thread-sqlite-warm
  | main-thread-sqlite-warm
  |
  | '''0.040''' (±30%!)
  | 0.043 (±28%!)
  | 0.043 (±28%!)
  |-
  |-
Line 144: Line 145:
  |-
  |-
  | box2d-throughput  
  | box2d-throughput  
  |
  | 6.987
  | 4.666
  | '''4.666'''
  |-
  |-
  | box2d-throughput-f32  
  | box2d-throughput-f32  
  |
  | 6.317
  | 4.277
  | '''4.277'''
  |-
  |-
  | lua-binarytrees  
  | lua-binarytrees  
  |  
  | 7.899
  | 7.182
  | '''7.182'''
  |-
  |-
  | lua-scimark  
  | lua-scimark  
  |
  | 10.555
  | 12.055
  | '''12.055'''
  |-
  |-
  | poppler-throughput  
  | poppler-throughput  
  |  
  | 7.023
  | 5.959
  | '''5.959'''
  |-
  |-
  | sqlite-throughput  
  | sqlite-throughput  
  |
  | 10.733
  | 8.485
  | '''8.485'''
  |-
  |-
  | colspan="3" class="header" | Preparation
  | colspan="3" class="header" | Preparation
  |-
  |-
  | poppler-cold-preparation  
  | poppler-cold-preparation  
  |  
  | 0.953
  | 0.846
  | '''0.846'''
  |-
  |-
  | poppler-warm-preparation  
  | poppler-warm-preparation  
  |
  | 0.186
  | 0.180
  | '''0.180'''
  |-
  |-
  | sqlite-cold-preparation  
  | sqlite-cold-preparation  
  |
  | '''1.103'''
  | 1.209
  | 1.209
  |-
  |-
  | sqlite-warm-preparation  
  | sqlite-warm-preparation  
  |
  | '''0.109'''
  | 0.115
  | 0.115
  |-
  |-
Line 188: Line 189:
  |-
  |-
  | box2d-variance  
  | box2d-variance  
  |  
  | 1.758
  | 1.290
  | '''1.290'''
  |-
  |-
  | poppler-variance  
  | poppler-variance  
  |
  | 4.830
  | 4.036
  | '''4.036'''
  |-
  |-
  | class="header" | Overall Score
  | class="header" | Overall Score
  | class="header" | result
  | class="header" | 3,839
  | class="header" | 4.287
  | class="header" | 4,287
  |}
  |}


Line 210: Line 211:
  |-
  |-
  | main-thread-poppler-cold
  | main-thread-poppler-cold
  |
  | 0.456
  | 0.338
  | '''0.338'''
  |-
  |-
  | main-thread-poppler-warm  
  | main-thread-poppler-warm  
  |
  | 0.445
  | 0.356
  | '''0.356'''
  |-
  |-
  | main-thread-sqlite-cold
  | main-thread-sqlite-cold
  |
  | '''0.050'''
  | 0.063
  | 0.063
  |-
  |-
  | main-thread-sqlite-warm
  | main-thread-sqlite-warm
  |
  | '''0.034''' (±24%!)
  | 0.042 (±28%!)
  | 0.042 (±28%!)
  |-
  |-
Line 228: Line 229:
  |-
  |-
  | box2d-throughput  
  | box2d-throughput  
  |
  | 6.806
  | 4.643
  | '''4.643'''
  |-
  |-
  | box2d-throughput-f32  
  | box2d-throughput-f32  
  |
  | 6.316
  | 4.284
  | '''4.284'''
  |-
  |-
  | lua-binarytrees  
  | lua-binarytrees  
  |  
  | 8.002
  | 7.239
  | '''7.239'''
  |-
  |-
  | lua-scimark  
  | lua-scimark  
  |
  | 10.250
  | 12.185
  | '''12.185'''
  |-
  |-
  | poppler-throughput  
  | poppler-throughput  
  |  
  | 7.034
  | 6.003
  | '''6.003'''
  |-
  |-
  | sqlite-throughput  
  | sqlite-throughput  
  |
  | 10.759
  | 8.366
  | '''8.366'''
  |-
  |-
  | colspan="3" class="header" | Preparation
  | colspan="3" class="header" | Preparation
  |-
  |-
  | poppler-cold-preparation  
  | poppler-cold-preparation  
  |  
  | 0.937
  | 0.842
  | '''0.842'''
  |-
  |-
  | poppler-warm-preparation  
  | poppler-warm-preparation  
  |
  | 0.183
  | 0.173
  | '''0.173'''
  |-
  |-
  | sqlite-cold-preparation  
  | sqlite-cold-preparation  
  |
  | '''1.071'''
  | 1.203
  | 1.203
  |-
  |-
  | sqlite-warm-preparation  
  | sqlite-warm-preparation  
  |
  | '''0.110'''
  | 0.121
  | 0.121
  |-
  |-
Line 272: Line 273:
  |-
  |-
  | box2d-variance  
  | box2d-variance  
  |  
  | 1.467
  | 1.210
  | '''1.210'''
  |-
  |-
  | poppler-variance  
  | poppler-variance  
  |
  | 4.836
  | 4.083
  | '''4.083'''
  |-
  |-
  | class="header" | Overall Score
  | class="header" | Overall Score
  | class="header" | result
  | class="header" | 3,942
  | class="header" | 4.323
  | class="header" | 4,323
  |}
  |}


Line 294: Line 295:
  |-
  |-
  | main-thread-poppler-cold
  | main-thread-poppler-cold
  |
  | 0.448
  | 0.359
  | '''0.359'''
  |-
  |-
  | main-thread-poppler-warm  
  | main-thread-poppler-warm  
  |
  | 0.451
  | 0.362
  | '''0.362'''
  |-
  |-
  | main-thread-sqlite-cold
  | main-thread-sqlite-cold
  |
  | '''0.056''' (±22%!)
  | 0.071 (±31%!)
  | 0.071 (±31%!)
  |-
  |-
  | main-thread-sqlite-warm
  | main-thread-sqlite-warm
  |
  | '''0.033''' (±22%!)
  | 0.042 (±26%!)
  | 0.042 (±26%!)
  |-
  |-
Line 312: Line 313:
  |-
  |-
  | box2d-throughput  
  | box2d-throughput  
  |
  | 6.834
  | 4.674
  | '''4.674'''
  |-
  |-
  | box2d-throughput-f32  
  | box2d-throughput-f32  
  |
  | 6.317
  | 4.304
  | '''4.304'''
  |-
  |-
  | lua-binarytrees  
  | lua-binarytrees  
  |  
  | 8.219
  | 7.226
  | '''7.226'''
  |-
  |-
  | lua-scimark  
  | lua-scimark  
  |
  | 9.925
  | 12.005
  | '''12.005'''
  |-
  |-
  | poppler-throughput  
  | poppler-throughput  
  |  
  | 7.021
  | 5.951
  | '''5.951'''
  |-
  |-
  | sqlite-throughput  
  | sqlite-throughput  
  |
  | 10.768
  | 8.414
  | '''8.414'''
  |-
  |-
  | colspan="3" class="header" | Preparation
  | colspan="3" class="header" | Preparation
  |-
  |-
  | poppler-cold-preparation  
  | poppler-cold-preparation  
  |  
  | 0.953
  | 0.840
  | '''0.840'''
  |-
  |-
  | poppler-warm-preparation  
  | poppler-warm-preparation  
  |
  | 0.184
  | 0.173
  | '''0.173'''
  |-
  |-
  | sqlite-cold-preparation  
  | sqlite-cold-preparation  
  |
  | '''1.091'''
  | 1.212
  | 1.212
  |-
  |-
  | sqlite-warm-preparation  
  | sqlite-warm-preparation  
  |
  | '''0.120'''
  | 0.140
  | 0.140
  |-
  |-
Line 356: Line 357:
  |-
  |-
  | box2d-variance  
  | box2d-variance  
  |  
  | '''1.546'''
  | 2.258
  | 2.258
  |-
  |-
  | poppler-variance  
  | poppler-variance  
  |
  | 4.832
  | 4.067
  | '''4.067'''
  |-
  |-
  | class="header" | Overall Score
  | class="header" | Overall Score
  | class="header" | result
  | class="header" | 3,869
  | class="header" | 4.225
  | class="header" | 4,225
  |}
  |}


Line 378: Line 379:
  |-
  |-
  | main-thread-poppler-cold
  | main-thread-poppler-cold
  |
  | 0.453
  | 0.360
  | '''0.360'''
  |-
  |-
  | main-thread-poppler-warm  
  | main-thread-poppler-warm  
  |
  | 0.453
  | 0.351
  | '''0.351'''
  |-
  |-
  | main-thread-sqlite-cold
  | main-thread-sqlite-cold
  |
  | '''0.047''' (±21%!)
  | 0.060 (±27%!)
  | 0.060 (±27%!)
  |-
  |-
  | main-thread-sqlite-warm
  | main-thread-sqlite-warm
  |
  | '''0.038''' (±26%!)
  | 0.052 (±40%!)
  | 0.052 (±40%!)
  |-
  |-
Line 396: Line 397:
  |-
  |-
  | box2d-throughput  
  | box2d-throughput  
  |
  | 7.161
  | 4.666
  | '''4.666'''
  |-
  |-
  | box2d-throughput-f32  
  | box2d-throughput-f32  
  |
  | 6.589
  | 4.292
  | '''4.292'''
  |-
  |-
  | lua-binarytrees  
  | lua-binarytrees  
  |  
  | 8.364
  | 7.230
  | '''7.230'''
  |-
  |-
  | lua-scimark  
  | lua-scimark  
  |
  | 9.820
  | 12.195
  | '''12.195'''
  |-
  |-
  | poppler-throughput  
  | poppler-throughput  
  |  
  | 7.596
  | 5.989
  | '''5.989'''
  |-
  |-
  | sqlite-throughput  
  | sqlite-throughput  
  |
  | 11.282
  | 8.583
  | '''8.583'''
  |-
  |-
  | colspan="3" class="header" | Preparation
  | colspan="3" class="header" | Preparation
  |-
  |-
  | poppler-cold-preparation  
  | poppler-cold-preparation  
  |  
  | 1.035
  | 0.844
  | '''0.844'''
  |-
  |-
  | poppler-warm-preparation  
  | poppler-warm-preparation  
  |
  | 0.199
  | 0.177
  | '''0.177'''
  |-
  |-
  | sqlite-cold-preparation  
  | sqlite-cold-preparation  
  |
  | '''1.232'''
  | 1.236
  | 1.236
  |-
  |-
  | sqlite-warm-preparation  
  | sqlite-warm-preparation  
  |
  | '''0.112'''
  | 0.137
  | 0.137
  |-
  |-
Line 440: Line 441:
  |-
  |-
  | box2d-variance  
  | box2d-variance  
  |  
  | 1.967
  | 1.297
  | '''1.297'''
  |-
  |-
  | poppler-variance  
  | poppler-variance  
  |
  | 5.149
  | 4.086
  | '''4.086'''
  |-
  |-
  | class="header" | Overall Score
  | class="header" | Overall Score
  | class="header" | result
  | class="header" | 3,762
  | class="header" | 4,211
  | class="header" | 4,211
  |}
  |}
Line 461: Line 462:
  |-
  |-
  | Firefox 32.0a1 32-bit
  | Firefox 32.0a1 32-bit
  |
  | 3,762
  |
  | 3,967
  |  
  | 3,869
  |  
  | 3875.8
  |-
  |-
  | Firefox 32.0a1 64-bit
  | Firefox 32.0a1 64-bit
  |
  | '''4,120'''
  |
  | '''4,323'''
  |
  | '''4,225'''
  |
  | '''4233.2'''
  |-
  |-
  |}
  |}
''Best score is '''highlighted'''''


= Notes =
The benchmark freezes very often at the "sqlite-warm-preparation" test. I had to right click on the page, or to select a word, open/close a new tab, or stuff like that to make it "refresh" and continue
= Conclusions =
= Conclusions =
''Anthony will fill this in later''
* Overall, 64-bit builds seem to perform about 9% better than 32-bit builds
* 32-bit builds perform about 7% to 21% better on sqlite benchmarks

Latest revision as of 17:33, 4 June 2014

Summary

The following documents A-B testing comparing performance of 32-bit Firefox builds to 64-bit Firefox builds, utilizing the Massive benchmark.

Methodology

  1. Test using the same system, make note of the system profile
  2. Install the latest Firefox Nightly 32-bit build and start with a new profile
  3. Load http://kripken.github.io/Massive/ and click "Run the benchmark now!"
  4. Let the benchmark run
  5. Record the results below when the benchmark completes
  6. Include a screenshot of the result
  7. Repeat the benchmark 5 times on the same system and record the result separately
  8. Install the latest Firefox Nightly 64-bit and repeat the tests with a new profile

System Profile

Operating System Windows 7 64-bit
Firefox (user agent string with build ID) Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202

Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202

Processor (make, model, speed) AMD FX(tm)-8320 Eight-Core Processor 3.50 Ghz
Memory (type, speed, amount) 8 GB DDR3 1333 Mhz
GPU (make, model, speed, memory, driver version) AMD Radeon R7 200 Series, Core Clock 780 Mhz, 2GB DDR3 900 Mhz, Catalyst version 13.12

Results

Testrun #1

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.452 0.348
main-thread-poppler-warm 0.428 0.372
main-thread-sqlite-cold 0.053 (±26%!) 0.062 (±25%!)
main-thread-sqlite-warm 0.033 (±24%!) 0.048 (±29%!)
Throughput
box2d-throughput 6.838 4.882
box2d-throughput-f32 6.316 4.567
lua-binarytrees 8.067 7.447
lua-scimark 10.545 11.730
poppler-throughput 7.008 6.445
sqlite-throughput 10.891 8.773
Preparation
poppler-cold-preparation 0.939 0.926
poppler-warm-preparation 0.185 0.185
sqlite-cold-preparation 1.084 1.347
sqlite-warm-preparation 0.098 0.123
Variance
box2d-variance 1.466 1.261
poppler-variance 4.843 4.395
Overall Score 3,967 4,120

Testrun #2

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.461 0.367
main-thread-poppler-warm 0.427 0.361
main-thread-sqlite-cold 0.065 0.064
main-thread-sqlite-warm 0.040 (±30%!) 0.043 (±28%!)
Throughput
box2d-throughput 6.987 4.666
box2d-throughput-f32 6.317 4.277
lua-binarytrees 7.899 7.182
lua-scimark 10.555 12.055
poppler-throughput 7.023 5.959
sqlite-throughput 10.733 8.485
Preparation
poppler-cold-preparation 0.953 0.846
poppler-warm-preparation 0.186 0.180
sqlite-cold-preparation 1.103 1.209
sqlite-warm-preparation 0.109 0.115
Variance
box2d-variance 1.758 1.290
poppler-variance 4.830 4.036
Overall Score 3,839 4,287

Testrun #3

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.456 0.338
main-thread-poppler-warm 0.445 0.356
main-thread-sqlite-cold 0.050 0.063
main-thread-sqlite-warm 0.034 (±24%!) 0.042 (±28%!)
Throughput
box2d-throughput 6.806 4.643
box2d-throughput-f32 6.316 4.284
lua-binarytrees 8.002 7.239
lua-scimark 10.250 12.185
poppler-throughput 7.034 6.003
sqlite-throughput 10.759 8.366
Preparation
poppler-cold-preparation 0.937 0.842
poppler-warm-preparation 0.183 0.173
sqlite-cold-preparation 1.071 1.203
sqlite-warm-preparation 0.110 0.121
Variance
box2d-variance 1.467 1.210
poppler-variance 4.836 4.083
Overall Score 3,942 4,323

Testrun #4

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.448 0.359
main-thread-poppler-warm 0.451 0.362
main-thread-sqlite-cold 0.056 (±22%!) 0.071 (±31%!)
main-thread-sqlite-warm 0.033 (±22%!) 0.042 (±26%!)
Throughput
box2d-throughput 6.834 4.674
box2d-throughput-f32 6.317 4.304
lua-binarytrees 8.219 7.226
lua-scimark 9.925 12.005
poppler-throughput 7.021 5.951
sqlite-throughput 10.768 8.414
Preparation
poppler-cold-preparation 0.953 0.840
poppler-warm-preparation 0.184 0.173
sqlite-cold-preparation 1.091 1.212
sqlite-warm-preparation 0.120 0.140
Variance
box2d-variance 1.546 2.258
poppler-variance 4.832 4.067
Overall Score 3,869 4,225

Testrun #5

Benchmark Firefox 32-bit Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold 0.453 0.360
main-thread-poppler-warm 0.453 0.351
main-thread-sqlite-cold 0.047 (±21%!) 0.060 (±27%!)
main-thread-sqlite-warm 0.038 (±26%!) 0.052 (±40%!)
Throughput
box2d-throughput 7.161 4.666
box2d-throughput-f32 6.589 4.292
lua-binarytrees 8.364 7.230
lua-scimark 9.820 12.195
poppler-throughput 7.596 5.989
sqlite-throughput 11.282 8.583
Preparation
poppler-cold-preparation 1.035 0.844
poppler-warm-preparation 0.199 0.177
sqlite-cold-preparation 1.232 1.236
sqlite-warm-preparation 0.112 0.137
Variance
box2d-variance 1.967 1.297
poppler-variance 5.149 4.086
Overall Score 3,762 4,211

Overall

Build Lowest Score Highest Score Median Score Average Score
Firefox 32.0a1 32-bit 3,762 3,967 3,869 3875.8
Firefox 32.0a1 64-bit 4,120 4,323 4,225 4233.2

Best score is highlighted

Notes

The benchmark freezes very often at the "sqlite-warm-preparation" test. I had to right click on the page, or to select a word, open/close a new tab, or stuff like that to make it "refresh" and continue

Conclusions

  • Overall, 64-bit builds seem to perform about 9% better than 32-bit builds
  • 32-bit builds perform about 7% to 21% better on sqlite benchmarks
Retrieved from "https://wiki.allizom.org/index.php?title=Platform/Games/Testing/32v64bit&oldid=985869"