TestEngineering/Performance/NimbleDroid: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎Automated: add link to the upload code)
(Remove content because we stopped using ND)
Line 1: Line 1:
= At a glance =
As of around May 2020, we no longer use Nimbledroid: https://github.com/mozilla-mobile/fenix/issues/11050
[https://nimbledroid.com/ NimbleDroid] provides functional performance testing for Android and iOS via a service. We currently use NimbleDroid exclusively for testing the performance of our Android applications on real devices.


{| class="wikitable"
We've switched to an in-house system instead (FNPRMS, to be replaced by mach perftest) because:
|-
- it's difficult to debug performance when working with a black box
! Application !! Upload !! [[#Cold startup]] !! [[#Page load]]
- Nimbledroid tests run with <pre>debuggable=true</pre> and that provides unreliable performance characteristics
|-
| Fennec || None || ? || ?
|-
| GeckoView Example || [[#Manual]] || ✅ || ✅
|-
| Reference Browser || [[#Automated]] (push to master) || ✅ || ?
|-
| Fenix || [[#Automated]] (push to master) || ✅ || ✅
|-
| Chrome || [[#Manual]] || ✅ || ✅
|}
 
= Test scenarios =
 
== Cold startup ==
The cold startup scenario is profiled for every app. See the [http://docs.nimbledroid.com/#cold-startup NimbleDroid documentation] for details on what and how they measure for cold startup.
 
== Page load ==
Custom scenarios are used to test page load for the following sites:
# https://abcnews.go.com
# https://amazon.com
# https://amazon.com/s/ref=nb_sb_noss_2/139-6317191-5622045?url=search-alias%3Daps&field-keywords=mobile+phone
# https://blogger.com/about
# https://booking.com
# https://booking.com/searchresults.en-gb.html?city=20088325
# https://buzzfeed.com/daves4/the-new-york-times-makes-the-nerdiest-correction-e?utm_term=.tlL2KE08L6#.hwrBglaV2A
# https://cnn.com
# https://cnn.com/ampstories/us/why-hurricane-michael-is-a-monster-unlike-any-other
# https://edition.cnn.com/ampstories/us/still-missing-cassini-heres-what-else-is-out-there
# https://en.m.wikipedia.org/wiki/Main_Page
# https://en.m.wikipedia.org/wiki/Mozilla_Corporation
# https://eurosport.de
# https://eurosport.de/fussball/fifa-wm/2018/wm-2018-spielplan-russland-alle-dfb-spiele-wm-2018-gruppen-wm-2018-termine_sto6691356/story.shtml
# https://expedia.com/Hotel-Search?destination=New+York%2C+New+York&latLong=40.756680%2C-73.986470&regionId=178293&startDate=&endDate=&rooms=1&_xpid=11905%7C1&adults=2
# https://fashionbeans.com
# https://fashionbeans.com/article/coolest-menswear-stores-in-the-world
# https://glassdoor.com/index.htm
# https://glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=true&clickSource=searchBtn&typedKeyword=Mozilla&sc.keyword=Mozilla&locT=C&locId=1147431&jobType=
# https://google.com
# https://google.com/maps?force=pwa
# https://google.com/search?q=restaurants+near+me
# https://guardian.co.uk
# https://healthyway.com
# https://instagram.com
# https://m.360.cn
# https://m.ebay-kleinanzeigen.de
# https://m.ebay-kleinanzeigen.de/s-anzeigen/auf-zeit-wg-berlin/zimmer/c199-l3331
# https://m.facebook.com
# https://m.facebook.com/Cristiano
# https://m.ranker.com
# https://m.ranker.com/list/hunger-games-book-vs-movie-comparisons/lisa-waugh?ref=browse_list_5&l=2&pos=1
# https://m.vk.com
# https://m.weibo.cn
# https://m.yelp.com
# https://m.yelp.de
# https://m.yelp.de/biz/burgeramt-berlin-2
# https://marvel.wikia.com/wiki/Black_Panther
# https://medium.com
# https://medium.com/s/coincidences-are-a-lie/could-america-have-also-been-the-birthplace-of-impressionism-cb3d31a2e22d
# https://people.com/amp-stories/royal-a-to-z
# https://rumble.com/v3c44t-foxes-jumping-on-my-trampoline.html
# https://sbnation.com/2018/2/12/16719476/college-football-playoff-rankings-explained
# https://sima.com.cn
# https://tripadvisor.com
# https://tripadvisor.com/Hotels-g187147-Paris_Ile_de_France-Hotels.html
# https://urbandictionary.com
# https://urbandictionary.com/define.php?term=awesome%20sauce
# https://washingtonpost.com
# https://washingtonpost.com/news/the-switch/wp/2017/06/06/amazon-kickstarter-reddit-and-mozilla-are-staging-a-net-neutrality-online-protest/?noredirect=on&utm_term=.c2cc1fa4862a
# https://wired.com
# https://wired.com/story/vehicle-future-bike
# https://youtube.com
# https://zalando.de
# https://zalando.de/herrenschuhe/?exclusive=true
 
= Uploading apps =
 
== Automated ==
 
This is how we upload apks to Nimbledroid for our nightly runs.
 
<nowiki>
#!/bin/bash
 
APK="target.apk"
 
cd /tmp
rm -f "${APK}"
 
curl -L -O "https://index.taskcluster.net/v1/task/project.mobile.fenix.v2.nightly.latest/artifacts/public/build/arm/${APK}" || exit 1
 
curl -F apk="@${APK}" -F device_config="android5" -v -u SECRETHEREXXX: https://nimble.app/api/v2/apks || exit 1
</nowiki>
 
For Reference Browser and Geckoview, see [https://github.com/mozilla-mobile/reference-browser/blob/5752fef68dafb8218431edc77850a859e1bfb889/automation/taskcluster/upload_apk_nimbledroid.py here]
 
For Fenix, see [https://github.com/mozilla-mobile/fenix/blob/5a4b7aec005ab549a2c23acd1930abe6b62cd900/automation/taskcluster/upload_apk_nimbledroid.py here]
 
== Manual ==
[http://docs.nimbledroid.com/androidUserGuide.html#advanced-usage Documentation]
 
= Results =
The test results can be seen in [[../Results#Firefox_Health|Firefox Health]] and [[../Results#Nimbledroid|NimbleDroid]]'s dashboard.
 
= Alerting =
{{todo|}}

Revision as of 19:38, 17 September 2020

As of around May 2020, we no longer use Nimbledroid: https://github.com/mozilla-mobile/fenix/issues/11050

We've switched to an in-house system instead (FNPRMS, to be replaced by mach perftest) because: - it's difficult to debug performance when working with a black box

- Nimbledroid tests run with 

debuggable=true

and that provides unreliable performance characteristics

Retrieved from "https://wiki.allizom.org/index.php?title=TestEngineering/Performance/NimbleDroid&oldid=1230872"