Summary
The following documents A-B testing comparing performance of 32-bit Firefox builds to 64-bit Firefox builds, utilizing the Massive benchmark.
Methodology
- Test using the same system, make note of the system profile
- Install the latest Firefox Nightly 32-bit build and start with a new profile
- Load http://kripken.github.io/Massive/ and click "Run the benchmark now!"
- Let the benchmark run
- Record the results below when the benchmark completes
- Include a screenshot of the result
- Repeat the benchmark 5 times on the same system and record the result separately
- Install the latest Firefox Nightly 64-bit and repeat the tests with a new profile
System Profile
| Operating System | Windows 8.1 64-bit |
| Firefox | (user agent string with build ID) Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:32.0) Gecko/20100101 Firefox/32.0, 20140604030202 |
| Processor | (make, model, speed) AMD FX(tm)-8320 Eight-Core Processor 3.50 Ghz |
| Memory | (type, speed, amount) 8 GB DDR3 1033 Mhz |
| GPU | (make, model, speed, memory, driver version) AMD Radeon R7 200 Series, Core Clock 780 Mhz, 2GB DDR3 900 Mhz, Catalyst version 13.12 |
Results
Testrun #1
| Benchmark | Firefox 32-bit | Firefox 64-bit |
| Main-thread Responsiveness | ||
| main-thread-poppler-cold | 0.729 (±22%!) | |
| main-thread-poppler-warm | 0.523 | |
| main-thread-sqlite-cold | 0.872 | |
| main-thread-sqlite-warm | 0.043 (±27%!) | |
| Throughput | ||
| box2d-throughput | 4.623 | |
| box2d-throughput-f32 | 4.258 | |
| lua-binarytrees | 7.271 | |
| lua-scimark | 12.160 | |
| poppler-throughput | 5.988 | |
| sqlite-throughput | 8.404 | |
| Preparation | ||
| poppler-cold-preparation | 0.864 | |
| poppler-warm-preparation | 0.183 | |
| sqlite-cold-preparation | 1.260 | |
| sqlite-warm-preparation | 0.123 | |
| Variance | ||
| box2d-variance | 1.216 | |
| poppler-variance | 4.074 | |
| Overall Score | result | 3.377 |
Testrun #2
| Benchmark | Firefox 32-bit | Firefox 64-bit |
| Main-thread Responsiveness | ||
| main-thread-poppler-cold | 0.367 | |
| main-thread-poppler-warm | 0.361 | |
| main-thread-sqlite-cold | 0.064 | |
| main-thread-sqlite-warm | 0.043 (±27%!) | |
| Throughput | ||
| box2d-throughput | 4.666 | |
| box2d-throughput-f32 | 4.277 | |
| lua-binarytrees | 7.182 | |
| lua-scimark | 12.055 | |
| poppler-throughput | 5.959 | |
| sqlite-throughput | 8.485 | |
| Preparation | ||
| poppler-cold-preparation | 0.846 | |
| poppler-warm-preparation | 0.180 | |
| sqlite-cold-preparation | 1.209 | |
| sqlite-warm-preparation | 0.115 | |
| Variance | ||
| box2d-variance | 1.290 | |
| poppler-variance | 4.036 | |
| Overall Score | result | 4.287 |
Testrun #3
| Benchmark | Firefox 32-bit | Firefox 64-bit |
| Main-thread Responsiveness | ||
| main-thread-poppler-cold | ||
| main-thread-poppler-warm | ||
| main-thread-sqlite-cold | ||
| main-thread-sqlite-warm | ||
| Throughput | ||
| box2d-throughput | ||
| box2d-throughput-f32 | ||
| lua-binarytrees | ||
| lua-scimark | ||
| poppler-throughput | ||
| sqlite-throughput | ||
| Preparation | ||
| poppler-cold-preparation | ||
| poppler-warm-preparation | ||
| sqlite-cold-preparation | ||
| sqlite-warm-preparation | ||
| Variance | ||
| box2d-variance | ||
| poppler-variance | ||
| Overall Score | result | 4.323 |
Testrun #4
| Benchmark | Firefox 32-bit | Firefox 64-bit |
| Main-thread Responsiveness | ||
| main-thread-poppler-cold | ||
| main-thread-poppler-warm | ||
| main-thread-sqlite-cold | ||
| main-thread-sqlite-warm | ||
| Throughput | ||
| box2d-throughput | ||
| box2d-throughput-f32 | ||
| lua-binarytrees | ||
| lua-scimark | ||
| poppler-throughput | ||
| sqlite-throughput | ||
| Preparation | ||
| poppler-cold-preparation | ||
| poppler-warm-preparation | ||
| sqlite-cold-preparation | ||
| sqlite-warm-preparation | ||
| Variance | ||
| box2d-variance | ||
| poppler-variance | ||
| Overall Score | result | 4.225 |
Testrun #5
| Benchmark | Firefox 32-bit | Firefox 64-bit |
| Main-thread Responsiveness | ||
| main-thread-poppler-cold | ||
| main-thread-poppler-warm | ||
| main-thread-sqlite-cold | ||
| main-thread-sqlite-warm | ||
| Throughput | ||
| box2d-throughput | ||
| box2d-throughput-f32 | ||
| lua-binarytrees | ||
| lua-scimark | ||
| poppler-throughput | ||
| sqlite-throughput | ||
| Preparation | ||
| poppler-cold-preparation | ||
| poppler-warm-preparation | ||
| sqlite-cold-preparation | ||
| sqlite-warm-preparation | ||
| Variance | ||
| box2d-variance | ||
| poppler-variance | ||
| Overall Score | result | result |
Overall
| Build | Lowest Score | Highest Score | Median Score | Average Score |
| Firefox 32.0a1 32-bit | ||||
| Firefox 32.0a1 64-bit |
Conclusions
Anthony will fill this in later