TestEngineering/Performance/NimbleDroid: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
(Adding the script that does the nightly uploads.) |
||
| Line 83: | Line 83: | ||
== Automated == | == Automated == | ||
{{ | |||
This is how we upload apks to Nimbledroid for our nightly runs. | |||
<nowiki> | |||
#!/bin/bash | |||
APK="target.apk" | |||
cd /tmp | |||
rm -f "${APK}" | |||
curl -L -O "https://index.taskcluster.net/v1/task/project.mobile.fenix.v2.nightly.latest/artifacts/public/build/arm/${APK}" || exit 1 | |||
curl -F apk="@${APK}" -F device_config="android5" -v -u SECRETHEREXXX: https://nimble.app/api/v2/apks || exit 1 | |||
</nowiki> | |||
== Manual == | == Manual == | ||
Revision as of 01:41, 19 September 2019
At a glance
NimbleDroid provides functional performance testing for Android and iOS via a service. We currently use NimbleDroid exclusively for testing the performance of our Android applications on real devices.
| Application | Upload | #Cold startup | #Page load |
|---|---|---|---|
| Fennec | None | ? | ? |
| GeckoView Example | #Manual | ✅ | ✅ |
| Reference Browser | #Automated (push to master) | ✅ | ? |
| Fenix | #Automated (push to master) | ✅ | ✅ |
| Chrome | #Manual | ✅ | ✅ |
Test scenarios
Cold startup
The cold startup scenario is profiled for every app. See the NimbleDroid documentation for details on what and how they measure for cold startup.
Page load
Custom scenarios are used to test page load for the following sites:
- https://abcnews.go.com
- https://amazon.com
- https://amazon.com/s/ref=nb_sb_noss_2/139-6317191-5622045?url=search-alias%3Daps&field-keywords=mobile+phone
- https://blogger.com/about
- https://booking.com
- https://booking.com/searchresults.en-gb.html?city=20088325
- https://buzzfeed.com/daves4/the-new-york-times-makes-the-nerdiest-correction-e?utm_term=.tlL2KE08L6#.hwrBglaV2A
- https://cnn.com
- https://cnn.com/ampstories/us/why-hurricane-michael-is-a-monster-unlike-any-other
- https://edition.cnn.com/ampstories/us/still-missing-cassini-heres-what-else-is-out-there
- https://en.m.wikipedia.org/wiki/Main_Page
- https://en.m.wikipedia.org/wiki/Mozilla_Corporation
- https://eurosport.de
- https://eurosport.de/fussball/fifa-wm/2018/wm-2018-spielplan-russland-alle-dfb-spiele-wm-2018-gruppen-wm-2018-termine_sto6691356/story.shtml
- https://expedia.com/Hotel-Search?destination=New+York%2C+New+York&latLong=40.756680%2C-73.986470®ionId=178293&startDate=&endDate=&rooms=1&_xpid=11905%7C1&adults=2
- https://fashionbeans.com
- https://fashionbeans.com/article/coolest-menswear-stores-in-the-world
- https://glassdoor.com/index.htm
- https://glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=true&clickSource=searchBtn&typedKeyword=Mozilla&sc.keyword=Mozilla&locT=C&locId=1147431&jobType=
- https://google.com
- https://google.com/maps?force=pwa
- https://google.com/search?q=restaurants+near+me
- https://guardian.co.uk
- https://healthyway.com
- https://instagram.com
- https://m.360.cn
- https://m.ebay-kleinanzeigen.de
- https://m.ebay-kleinanzeigen.de/s-anzeigen/auf-zeit-wg-berlin/zimmer/c199-l3331
- https://m.facebook.com
- https://m.facebook.com/Cristiano
- https://m.ranker.com
- https://m.ranker.com/list/hunger-games-book-vs-movie-comparisons/lisa-waugh?ref=browse_list_5&l=2&pos=1
- https://m.vk.com
- https://m.weibo.cn
- https://m.yelp.com
- https://m.yelp.de
- https://m.yelp.de/biz/burgeramt-berlin-2
- https://marvel.wikia.com/wiki/Black_Panther
- https://medium.com
- https://medium.com/s/coincidences-are-a-lie/could-america-have-also-been-the-birthplace-of-impressionism-cb3d31a2e22d
- https://people.com/amp-stories/royal-a-to-z
- https://rumble.com/v3c44t-foxes-jumping-on-my-trampoline.html
- https://sbnation.com/2018/2/12/16719476/college-football-playoff-rankings-explained
- https://sima.com.cn
- https://tripadvisor.com
- https://tripadvisor.com/Hotels-g187147-Paris_Ile_de_France-Hotels.html
- https://urbandictionary.com
- https://urbandictionary.com/define.php?term=awesome%20sauce
- https://washingtonpost.com
- https://washingtonpost.com/news/the-switch/wp/2017/06/06/amazon-kickstarter-reddit-and-mozilla-are-staging-a-net-neutrality-online-protest/?noredirect=on&utm_term=.c2cc1fa4862a
- https://wired.com
- https://wired.com/story/vehicle-future-bike
- https://youtube.com
- https://zalando.de
- https://zalando.de/herrenschuhe/?exclusive=true
Uploading apps
Automated
This is how we upload apks to Nimbledroid for our nightly runs.
#!/bin/bash
APK="target.apk"
cd /tmp
rm -f "${APK}"
curl -L -O "https://index.taskcluster.net/v1/task/project.mobile.fenix.v2.nightly.latest/artifacts/public/build/arm/${APK}" || exit 1
curl -F apk="@${APK}" -F device_config="android5" -v -u SECRETHEREXXX: https://nimble.app/api/v2/apks || exit 1
Manual
[TODO]
Results
The test results can be seen in Firefox Health and NimbleDroid's dashboard.
Alerting
[TODO]