Identity/AttachedServices/Key Stretching Performance Tests: Difference between revisions
< Identity | AttachedServices
Jump to navigation
Jump to search
No edit summary |
No edit summary |
||
| (11 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
{{LastUpdated}} | |||
{| border="1" cellpadding="2" | {| border="1" cellpadding="2" | ||
|+ Test Results (in | |+ Test Results (in seconds per operation) | ||
|- | |- | ||
! Browser / Test !! Firefox 23 Desktop !! Google Chrome 28 !! Firefox Android Beta 24 !! Firefox OS !! Android WebView !! Android + OpenSSL | ! Browser / Test !! Firefox 23 Desktop (MBP) !! Google Chrome 28 (MBP) !! Firefox Android Beta 24 (Nexus4) !! Firefox OS (Keon) !! Android WebView (Nexus4) !! Android + OpenSSL | ||
|- | |- | ||
! [1] 20k PBKDF2-SHA256 | ! [1] 20k PBKDF2-SHA256 | ||
| 5.39 ops/sec ±4.20% | | 186ms (5.39 ops/sec ±4.20%, 30 runs sampled) || 123ms (8.12 ops/sec ±1.15%, 43 runs sampled) || 2.13s (0.47 ops/sec ±7.73%, 7 runs sampled) || 4.17s (0.24 ops/sec ±7.06%, 6 runs sampled) || 1.52s (0.66 ops/sec ±4.35%, 8 runs sampled) || 351ms (2.85 ops/sec, **1 run sampled**) | ||
|- | |- | ||
! [2] 20k*PBKDF + scrypt(64k,8,1) + 20k*PBKDF | ! [2] 20k*PBKDF + remote(scrypt(64k,8,1)) + 20k*PBKDF | ||
| 0.66 ops/sec ±10.70% | | 1.52s (0.66 ops/sec ±10.70%, 8 runs sampled) || 1.27s (0.79 ops/sec ±1.98%, 8 runs sampled) || 6.25s (0.16 ops/sec ±9.02%, 5 runs sampled) || N/A || 5.56s (0.18 ops/sec ±21.96%, 6 runs sampled) || -- | ||
|- | |||
|- | |||
! [2.1] running on m1.large | |||
| 1.05s (0.94 ops/sec ±7.06% (9 runs sampled)) || 846ms (1.18 ops/sec ±0.55% (10 runs sampled)) || 5.25s (0.19 opts/sec ±8.91% (5 runs sampled) || 11.79s (0.08 opts/sec ±1.98% (5 runs sampled) || 3.94s (0.25 opts/sec ±5.78% (6 runs sampled) || -- | |||
|- | |- | ||
|- | |- | ||
! [3] 20kPBKDF + local(scrypt(64k,8,1)) + 20kPBKDF | ! [3] 20kPBKDF + local(scrypt(64k,8,1)) + 20kPBKDF | ||
| -- || -- || -- || -- || -- || 0.1369 ops/sec | | -- || -- || -- || -- || -- || 7.3s (0.1369 ops/sec, 10 runs sampled, extremes discarded -- 7362-7505msec) | ||
|- | |||
|- | |||
! [4] 20kPBKDF + emscripten(scrypt(64k,8,1)) + 20kPBKDF | |||
| 3.6s (0.28 ops/sec ±26.77%, 6 runs sampled) || 3.2s (0.31 ops/sec ±1.46%, 6 runs sampled) || 20s (0.05 ops/sec ±25.20%, 5 runs sampled) || Crash. || 33s (0.03 ops/sec ±2.86%, 5 runs sampled) || -- | |||
|- | |- | ||
|} | |} | ||
| Line 19: | Line 29: | ||
'''Tests''' | '''Tests''' | ||
* [1] '''20k PBKDF2-SHA256''', pdbkdf2.derive(...) - Client pdbkdf2 derivation [https://wiki.mozilla.org/Identity/AttachedServices/KeyServerProtocol#stretch-KDF #stretch-KDF] | * [1] '''20k PBKDF2-SHA256''', pdbkdf2.derive(...) - Client pdbkdf2 derivation [https://wiki.mozilla.org/Identity/AttachedServices/KeyServerProtocol#stretch-KDF #stretch-KDF] | ||
* [2] '''20k*PBKDF+scrypt(64k,8,1)+20k*PBKDF''' keyStretch.derive(email, password) - Full Key Stretch with a remote helper at "http://scrypt.dev.lcip.org/". [https://wiki.mozilla.org/Identity/AttachedServices/KeyServerProtocol#stretch-KDF #stretch-KDF + #main-KDF] | * [2] '''20k*PBKDF+scrypt(64k,8,1)+20k*PBKDF''' keyStretch.derive(email, password) - Full Key Stretch with a remote scrypt helper at "http://scrypt.dev.lcip.org/" (EC2 m1.small, est scrypt(64k,8,1) time = 2.2s). [https://wiki.mozilla.org/Identity/AttachedServices/KeyServerProtocol#stretch-KDF #stretch-KDF + #main-KDF]. m1.small instance runs scrypt in 2.2s, whereas the more cost-effective (given serious load) c1.medium or c1.xlarge instances will run it in 1.0s . | ||
* [2.1] running on m1.large remote instance | |||
* [3] '''20kPBKDF + local(scrypt(64k,8,1)) + 20kPBKDF''' | * [3] '''20kPBKDF + local(scrypt(64k,8,1)) + 20kPBKDF''' | ||
* [4] '''20kPBKDF + emscripten(scrypt(64k,8,1)) + 20kPBKDF''' Using the [https://github.com/tonyg/js-scrypt Pure-Javascript Emscripten-compiled scrypt routine]. Modified 'TOTAL_MEMORY' to 'scrypt_module_factory(134217728);' | |||
'''Test Framework''' | '''Test Framework''' | ||
| Line 28: | Line 42: | ||
* "ops/sec" stands for operations per second. That is how many times a test is projected to execute in a second. | * "ops/sec" stands for operations per second. That is how many times a test is projected to execute in a second. | ||
* A test is repeatedly executed until it reaches the minimum time needed to get a percentage uncertainty for the measurement of less than or equal to 1%. The number of iterations will vary depending on the resolution of the environment’s timer and how many times a test can execute in the minimum run time. We collect completed test runs for 5 seconds (configurable), or at least 5 runs (also configurable), and then perform statistical analysis on the sample. So, a test may be repeated 100,000 times in 50 ms (the minimum run time for most environments), and then repeated 100 times more (5 seconds). A larger sample size (in this example, 100), leads to a smaller margin of error. | * A test is repeatedly executed until it reaches the minimum time needed to get a percentage uncertainty for the measurement of less than or equal to 1%. The number of iterations will vary depending on the resolution of the environment’s timer and how many times a test can execute in the minimum run time. We collect completed test runs for 5 seconds (configurable), or at least 5 runs (also configurable), and then perform statistical analysis on the sample. So, a test may be repeated 100,000 times in 50 ms (the minimum run time for most environments), and then repeated 100 times more (5 seconds). A larger sample size (in this example, 100), leads to a smaller margin of error. | ||
* benchmark.js returns ops/sec, rounded to hundredths. This number was inverted to obtain secs/op, losing precision in the process. | |||
'''Hardware Used''' | |||
* Firefox Desktop, Google Chrome - MacBook Pro 2.7 GHz i7, 16 GB 1600MHz DDR3 | |||
* Firefox OS - Keon (Snapdragon S1 7225AB 1Ghz, 512 RAM) | |||
* Firefox Android, Android Webview - Nexus 4 (1.512 GHz quad-core, 2GB RAM) | |||
Test Suite Page: [http:// | Test Suite Page: [http://vladikoff.github.io/fxa-js-client-old/web/benchmark.html http://vladikoff.github.io/fxa-js-client-old/web/benchmark.html] | ||
Latest revision as of 01:27, 23 November 2013
Last updated: 2013/11/23
| Browser / Test | Firefox 23 Desktop (MBP) | Google Chrome 28 (MBP) | Firefox Android Beta 24 (Nexus4) | Firefox OS (Keon) | Android WebView (Nexus4) | Android + OpenSSL |
|---|---|---|---|---|---|---|
| [1] 20k PBKDF2-SHA256 | 186ms (5.39 ops/sec ±4.20%, 30 runs sampled) | 123ms (8.12 ops/sec ±1.15%, 43 runs sampled) | 2.13s (0.47 ops/sec ±7.73%, 7 runs sampled) | 4.17s (0.24 ops/sec ±7.06%, 6 runs sampled) | 1.52s (0.66 ops/sec ±4.35%, 8 runs sampled) | 351ms (2.85 ops/sec, **1 run sampled**) |
| [2] 20k*PBKDF + remote(scrypt(64k,8,1)) + 20k*PBKDF | 1.52s (0.66 ops/sec ±10.70%, 8 runs sampled) | 1.27s (0.79 ops/sec ±1.98%, 8 runs sampled) | 6.25s (0.16 ops/sec ±9.02%, 5 runs sampled) | N/A | 5.56s (0.18 ops/sec ±21.96%, 6 runs sampled) | -- |
| [2.1] running on m1.large | 1.05s (0.94 ops/sec ±7.06% (9 runs sampled)) | 846ms (1.18 ops/sec ±0.55% (10 runs sampled)) | 5.25s (0.19 opts/sec ±8.91% (5 runs sampled) | 11.79s (0.08 opts/sec ±1.98% (5 runs sampled) | 3.94s (0.25 opts/sec ±5.78% (6 runs sampled) | -- |
| [3] 20kPBKDF + local(scrypt(64k,8,1)) + 20kPBKDF | -- | -- | -- | -- | -- | 7.3s (0.1369 ops/sec, 10 runs sampled, extremes discarded -- 7362-7505msec) |
| [4] 20kPBKDF + emscripten(scrypt(64k,8,1)) + 20kPBKDF | 3.6s (0.28 ops/sec ±26.77%, 6 runs sampled) | 3.2s (0.31 ops/sec ±1.46%, 6 runs sampled) | 20s (0.05 ops/sec ±25.20%, 5 runs sampled) | Crash. | 33s (0.03 ops/sec ±2.86%, 5 runs sampled) | -- |
Tests
- [1] 20k PBKDF2-SHA256, pdbkdf2.derive(...) - Client pdbkdf2 derivation #stretch-KDF
- [2] 20k*PBKDF+scrypt(64k,8,1)+20k*PBKDF keyStretch.derive(email, password) - Full Key Stretch with a remote scrypt helper at "http://scrypt.dev.lcip.org/" (EC2 m1.small, est scrypt(64k,8,1) time = 2.2s). #stretch-KDF + #main-KDF. m1.small instance runs scrypt in 2.2s, whereas the more cost-effective (given serious load) c1.medium or c1.xlarge instances will run it in 1.0s .
- [2.1] running on m1.large remote instance
- [3] 20kPBKDF + local(scrypt(64k,8,1)) + 20kPBKDF
- [4] 20kPBKDF + emscripten(scrypt(64k,8,1)) + 20kPBKDF Using the Pure-Javascript Emscripten-compiled scrypt routine. Modified 'TOTAL_MEMORY' to 'scrypt_module_factory(134217728);'
Test Framework
These performance tests use Benchmark.js
- "ops/sec" stands for operations per second. That is how many times a test is projected to execute in a second.
- A test is repeatedly executed until it reaches the minimum time needed to get a percentage uncertainty for the measurement of less than or equal to 1%. The number of iterations will vary depending on the resolution of the environment’s timer and how many times a test can execute in the minimum run time. We collect completed test runs for 5 seconds (configurable), or at least 5 runs (also configurable), and then perform statistical analysis on the sample. So, a test may be repeated 100,000 times in 50 ms (the minimum run time for most environments), and then repeated 100 times more (5 seconds). A larger sample size (in this example, 100), leads to a smaller margin of error.
- benchmark.js returns ops/sec, rounded to hundredths. This number was inverted to obtain secs/op, losing precision in the process.
Hardware Used
- Firefox Desktop, Google Chrome - MacBook Pro 2.7 GHz i7, 16 GB 1600MHz DDR3
- Firefox OS - Keon (Snapdragon S1 7225AB 1Ghz, 512 RAM)
- Firefox Android, Android Webview - Nexus 4 (1.512 GHz quad-core, 2GB RAM)
Test Suite Page: http://vladikoff.github.io/fxa-js-client-old/web/benchmark.html