Platform/Games/Testing/32v64bit

Summary

The following documents A-B testing comparing performance of 32-bit Firefox builds to 64-bit Firefox builds, utilizing the Massive benchmark.

Methodology

1. Test using the same system, make note of the system profile
2. Install the latest Firefox Nightly 32-bit build and start with a new profile
3. Load http://kripken.github.io/Massive/ and click "Run the benchmark now!"
4. Let the benchmark run
5. Record the results below when the benchmark completes
6. Include a screenshot of the result
7. Repeat the benchmark 5 times on the same system and record the result separately
8. Install the latest Firefox Nightly 64-bit and repeat the tests with a new profile

System Profile

Operating System	Windows 7 64-bit
Firefox	(user agent string with build ID) Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202 Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202
Processor	(make, model, speed) AMD FX(tm)-8320 Eight-Core Processor 3.50 Ghz
Memory	(type, speed, amount) 8 GB DDR3 1333 Mhz
GPU	(make, model, speed, memory, driver version) AMD Radeon R7 200 Series, Core Clock 780 Mhz, 2GB DDR3 900 Mhz, Catalyst version 13.12

Results

Testrun #1

Benchmark	Firefox 32-bit	Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold	0.452	0.348
main-thread-poppler-warm	0.428	0.372
main-thread-sqlite-cold	0.053 (±26%!)	0.062 (±25%!)
main-thread-sqlite-warm	0.033 (±24%!)	0.048 (±29%!)
Throughput
box2d-throughput	6.838	4.882
box2d-throughput-f32	6.316	4.567
lua-binarytrees	8.067	7.447
lua-scimark	10.545	11.730
poppler-throughput	7.008	6.445
sqlite-throughput	10.891	8.773
Preparation
poppler-cold-preparation	0.939	0.926
poppler-warm-preparation	0.185	0.185
sqlite-cold-preparation	1.084	1.347
sqlite-warm-preparation	0.098	0.123
Variance
box2d-variance	1.466	1.261
poppler-variance	4.843	4.395
Overall Score	3,967	4,120

Testrun #2

Benchmark	Firefox 32-bit	Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold	0.461	0.367
main-thread-poppler-warm	0.427	0.361
main-thread-sqlite-cold	0.065	0.064
main-thread-sqlite-warm	0.040 (±30%!)	0.043 (±28%!)
Throughput
box2d-throughput	6.987	4.666
box2d-throughput-f32	6.317	4.277
lua-binarytrees	7.899	7.182
lua-scimark	10.555	12.055
poppler-throughput	7.023	5.959
sqlite-throughput	10.733	8.485
Preparation
poppler-cold-preparation	0.953	0.846
poppler-warm-preparation	0.186	0.180
sqlite-cold-preparation	1.103	1.209
sqlite-warm-preparation	0.109	0.115
Variance
box2d-variance	1.758	1.290
poppler-variance	4.830	4.036
Overall Score	3,839	4,287

Testrun #3

Benchmark	Firefox 32-bit	Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold	0.456	0.338
main-thread-poppler-warm	0.445	0.356
main-thread-sqlite-cold	0.050	0.063
main-thread-sqlite-warm	0.034 (±24%!)	0.042 (±28%!)
Throughput
box2d-throughput	6.806	4.643
box2d-throughput-f32	6.316	4.284
lua-binarytrees	8.002	7.239
lua-scimark	10.250	12.185
poppler-throughput	7.034	6.003
sqlite-throughput	10.759	8.366
Preparation
poppler-cold-preparation	0.937	0.842
poppler-warm-preparation	0.183	0.173
sqlite-cold-preparation	1.071	1.203
sqlite-warm-preparation	0.110	0.121
Variance
box2d-variance	1.467	1.210
poppler-variance	4.836	4.083
Overall Score	3,942	4,323

Testrun #4

Benchmark	Firefox 32-bit	Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold	0.448	0.359
main-thread-poppler-warm	0.451	0.362
main-thread-sqlite-cold	0.056 (±22%!)	0.071 (±31%!)
main-thread-sqlite-warm	0.033 (±22%!)	0.042 (±26%!)
Throughput
box2d-throughput	6.834	4.674
box2d-throughput-f32	6.317	4.304
lua-binarytrees	8.219	7.226
lua-scimark	9.925	12.005
poppler-throughput	7.021	5.951
sqlite-throughput	10.768	8.414
Preparation
poppler-cold-preparation	0.953	0.840
poppler-warm-preparation	0.184	0.173
sqlite-cold-preparation	1.091	1.212
sqlite-warm-preparation	0.120	0.140
Variance
box2d-variance	1.546	2.258
poppler-variance	4.832	4.067
Overall Score	3,869	4,225

Testrun #5

Benchmark	Firefox 32-bit	Firefox 64-bit
Main-thread Responsiveness
main-thread-poppler-cold	0.453	0.360
main-thread-poppler-warm	0.453	0.351
main-thread-sqlite-cold	0.047 (±21%!)	0.060 (±27%!)
main-thread-sqlite-warm	0.038 (±26%!)	0.052 (±40%!)
Throughput
box2d-throughput	7.161	4.666
box2d-throughput-f32	6.589	4.292
lua-binarytrees	8.364	7.230
lua-scimark	9.820	12.195
poppler-throughput	7.596	5.989
sqlite-throughput	11.282	8.583
Preparation
poppler-cold-preparation	1.035	0.844
poppler-warm-preparation	0.199	0.177
sqlite-cold-preparation	1.232	1.236
sqlite-warm-preparation	0.112	0.137
Variance
box2d-variance	1.967	1.297
poppler-variance	5.149	4.086
Overall Score	3,762	4,211

Overall

Build	Lowest Score	Highest Score	Median Score	Average Score
Firefox 32.0a1 32-bit	3,762	3,967	3,869	3875.8
Firefox 32.0a1 64-bit	4,120	4,323	4,225	4233.2

Notes

The benchmark freezes very often at the "sqlite-warm-preparation" test. I had to right click on the page, or to select a word, open/close a new tab, or stuff like that to make it "refresh" and continue

Conclusions

• Overall, 64-bit builds seem to perform about 9% better than 32-bit builds