Hasal: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Jenkins try build information)
 
(16 intermediate revisions by 3 users not shown)
Line 1: Line 1:
A performance measurement project for Google Doc, Google Sheet, and Google Slide.
== About Hasal ==
Hasal /ha'sɑlu/


== State of this project ==
This word is came from Indigenous Taiwanese "Bunun", and it means hail. Why we choose this word is because we expect our testing could like the hail, fast and weighty.
And through our testing could bring more improvement on our performance or quality.


We are currently in the middle of phase 2. Execution summary as below:
This is a performance measurement project for web apps.


=== Execution Summary ===
== Contacts ==
* Priority: The priority of test cases
* Script Complete: test script is implemented and tested
* Execution Complete: test script run over 30 times and running time is recorded
* Profiler Data Complete: Gecko profiler, har profiler and performance timing object upload to bugzilla
* Analyze Data Complete: Analyze the gecko profiler data and find the most running time of different time unit (between navigationStart and domLoading)
{| class="wikitable"
{| class="wikitable"
! style="text-align: center;" | Name
! style="text-align: center;" | Email address
! style="text-align: center;" |
|-
| rowspan="1" | Google Group - [https://groups.google.com/a/mozilla.com/forum/#!forum/hasal-project Hasal Project]
| [mailto:hasal-project@mozilla.com  hasal-project@mozilla.com]
| Send email if you need any support
|-
| rowspan="1" | Shako Ho
| [mailto:sho@mozilla.com sho@mozilla.com]
| You can contact him to get more project detail.
|-
| rowspan="1" | Askeing Yen
| [mailto:askeing@mozilla.com askeing@mozilla.com]
|
|-
| rowspan="1" | Mike Lien
| [mailto:mlien@mozilla.com mlien@mozilla.com]
|
|-
|-
! Priority !! Script Complete !! Execution Complete !! Profiler Data Complete !! Analyze Data Complete
| rowspan="1" | Walter Chen
| [mailto:wachen@mozilla.com wachen@mozilla.com]
|
|-
|-
| High || 27/27 || 27/27 || 15/24 || 15/24
| rowspan="1" | Bobby Chien
| [mailto:bchien@mozilla.com bchien@mozilla.com]
| Program Management
|-
|-
| Medium || 0/25 || 0/25 || 0/25 || 0/25
| rowspan="1" | Ken Chang
| [mailto:kchang@mozilla.com kchang@mozilla.com]
| Point of Contact for Engineering
|-
|-
| Low || 0/14 || 0/14 || 0/14 || 0/14
|}
|}
== State of this project ==
We are finish the testing for google doc in July. Now, we are shifting the focus to Facebook and Gmail.
=== Execution Summary ===
* Google doc: Total we have 39 cases tested, 34 test cases indicate Firefox slower than Chrome. 6 cases how 30% gap between Firefox and Chrome.
* Facebook: Pilot test result will come out before end of July.
* Gmail: Pilot test result will come out before end of July.


=== Testing Result ===
=== Testing Result ===
You can find all testing result here [https://docs.google.com/spreadsheets/d/1XDZr46rR8CcDLhyqn_RjOZx0NdeP1AfVhcsRIcy1BGI/edit#gid=1256012575 Testing Result]
* Google doc testing result:
And the analysis diagram for module consume the most running time in each test cases could generate automatically like below:
  You can find all testing result here [https://docs.google.com/spreadsheets/d/1XDZr46rR8CcDLhyqn_RjOZx0NdeP1AfVhcsRIcy1BGI/edit#gid=1256012575 Testing Result]
 
  And the analysis diagram for module consume the most running time in each test cases could generate automatically like below:
[[File:Summary 0518.png|1024x768px]]
  [[File:Summary 0518.png|1024x768px]]


Or you can download the XMind file [https://github.com/Mozilla-TWQA/Hasal_analysis/blob/master/output/summary.xmind?raw=true here].
Or you can download the XMind file [https://github.com/Mozilla-TWQA/Hasal_analysis/blob/master/output/summary.xmind?raw=true here].
Line 35: Line 65:
We use the video recording to record the user actions and extract the frames to get the running time from user perspective. Based on same user actions, we measure the difference between Firefox and Chrome.
We use the video recording to record the user actions and extract the frames to get the running time from user perspective. Based on same user actions, we measure the difference between Firefox and Chrome.


== Project Plan ==
== How to Trigger Try Build==
 
** [https://goo.gl/umtHvq Resource Plan]
** [https://goo.gl/n4TnoL Test Plan]
** [https://docs.google.com/spreadsheets/d/1rU6cMxAtkucD-6PrpeWcGOf_6aplMCngS3ecSXgeHoo/edit#gid=1119453330 Meeting minutes]
 
=== Google Doc ===
 
==== Phase 1 (Done) ====
 
* Build up a workable framework that can run some manual created test cases
** Trigger actions on browser
** Record video on the background
** Enable profiler on firefox automatically
** Convert the video to image automatically
** Compare images with sample images
* Measure the difference between Firefox and Chrome for sample user actions (40% found between Firefox and Chrome)


==== Phase 2 ====
* Go to Jenkins server site using your browser
** You may request the ip of server from [mailto:hasal-project@mozilla.com  hasal-project@mozilla.com] or anyone in Hasal dev team
* Login (top-right corner) using your LDAP
* Trigger Try_Ubuntu and Try_Portal
        [[File:Jenkins1.png|1024px]]
* Enter necessary information as requested
        [[File:Jenkins2.png|1024px]]
* You can see the running progress and result artifacts after test finished


* Create test cases based on test matrices
* Create test scripts for high priority cases and execute them
* Generate profiler data for each testing result
* Find out the module and method which cost the most running time
* Go through the whole process from analyzing to provide information
* Framework improvement
** Replace Selenium with Sikuli
** Get the medium value and other statistics
* Support windows and mac


==== Phase 3 ====
== Working Measurements ==


* Create scripts for medium and low priority cases and execute them
=== Measurements ===
* Find out the module and method which cost the most running time


== Team ==
{| class="wikitable"
{| class="wikitable"
! style="text-align: center;" | Name
! style="text-align: center;" | App Name
! style="text-align: center;" | Email address
! style="text-align: center;" | Meta Bugs
! style="text-align: center;" |  
! style="text-align: center;" | Reports
! style="text-align: center;" | Status
|-
| rowspan="1" | Google Docs
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1260981 1260981]
| Pilot, [https://drive.google.com/open?id=1XDZr46rR8CcDLhyqn_RjOZx0NdeP1AfVhcsRIcy1BGI Full]
| User would aware performance issues on Firefox
|-
|-
| rowspan="1" | Shako Ho
| rowspan="1" | Outlook Web Access (Office 365)
| [http://mailto:sho@mozilla.com sho@mozilla.com]
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1275000 1275000]
| You can contact him to get more project detail.
| [https://drive.google.com/open?id=1EFLyUUFHdktOVyEz8zQACY_lRfz69_hwBiCeetQe9iE Pilot], Full
| No big issue per pilot test
|-
|-
| rowspan="1" | Askeing Yen
| rowspan="1" | Outlook Web Access (Exchange)
| [http://mailto:askeing@mozilla.com askeing@mozilla.com]
|  
|  
| Pilot, Full
| Suspended, awaiting measurable builds
|-
| rowspan="1" | [https://www.facebook.com/ Facebook]
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1288602 1288602]
| [https://drive.google.com/open?id=1O6Y0coEzWg6YMhe6ecsB-4Z6ABN_S18KIm87hPGjt2A Pilot], Full
| Found issues, working on full test
|-
|-
| rowspan="1" | Cynthia Tang
| rowspan="1" | [https://mail.google.com/ Gmail]
| [http://mailto:ctang@mozilla.com ctang@mozilla.com]
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1288604 1288604]
|  
| [https://drive.google.com/open?id=1UdEeXtwFbh_ODtzxIPHT7E1xeuwxSaRBFO7mhW1pxOA Pilot], Full
| No big issue per pilot test
|-
|-
| rowspan="1" | Farmer Tseng
| rowspan="1" | [http://www.bbc.com/ BBC]
| [http://mailto:fatseng@mozilla.com fatseng@mozilla.com]
|  
|  
| Pilot, Full
| 
|-
|-
| rowspan="1" | Walter Chen
| rowspan="1" | [https://www.youtube.com/ Youtube]
| [http://mailto:wachen@mozilla.com wachen@mozilla.com]
|  
|  
| Pilot, Full
| 
|-
|-
|}
|}


== [https://bugzilla.mozilla.org/show_bug.cgi?id=1260981 Working Items] ==
 
 
=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1260981 Google Docs] ===
<bugzilla>
<bugzilla>
{
{
     "status": ["NEW", "ASSIGNED", "REOPENED"],
     "status": ["NEW", "ASSIGNED", "REOPENED"],
     "blocks": "1260981",
     "blocks": "1260981",
    "include_fields": "id, component, summary, status, assigned_to, target_milestone, depends_on",
    "order": "target_milestone"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1275000 Outlook Web Access (Exchange)] ===
<bugzilla>
{
    "status": ["NEW", "ASSIGNED", "REOPENED"],
    "blocks": "1275000",
    "include_fields": "id, component, summary, status, assigned_to, target_milestone, depends_on",
    "order": "target_milestone"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1288602 Facebook] ===
<bugzilla>
{
    "status": ["NEW", "ASSIGNED", "REOPENED"],
    "blocks": "1288602",
    "include_fields": "id, component, summary, status, assigned_to, target_milestone, depends_on",
    "order": "target_milestone"
}
</bugzilla>
=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1288604 GMail] ===
<bugzilla>
{
    "status": ["NEW", "ASSIGNED", "REOPENED"],
    "blocks": "1288604",
     "include_fields": "id, component, summary, status, assigned_to, target_milestone, depends_on",
     "include_fields": "id, component, summary, status, assigned_to, target_milestone, depends_on",
     "order": "target_milestone"
     "order": "target_milestone"
Line 113: Line 171:


Following the steps [https://github.com/Mozilla-TWQA/Hasal/blob/master/README.md here] to setup the framework. For the time being, only Ubuntu is officially supported as a testing environment; you can use OS X but you'll have to manually solve some package dependency issue. Note that Ubuntu 12.04 and 14.04 is the best-supported testing OS.   
Following the steps [https://github.com/Mozilla-TWQA/Hasal/blob/master/README.md here] to setup the framework. For the time being, only Ubuntu is officially supported as a testing environment; you can use OS X but you'll have to manually solve some package dependency issue. Note that Ubuntu 12.04 and 14.04 is the best-supported testing OS.   
=== Template VM ===
You can use vagrant to download the template VM with Hasal's automation framework.
Please follow the instruction below:
* vagrant init shako/hasal
* vagrant up --provider virtualbox


=== Testing Data Summarize Tool ===
=== Testing Data Summarize Tool ===

Latest revision as of 09:35, 10 February 2017

About Hasal

Hasal /ha'sɑlu/

This word is came from Indigenous Taiwanese "Bunun", and it means hail. Why we choose this word is because we expect our testing could like the hail, fast and weighty. And through our testing could bring more improvement on our performance or quality.

This is a performance measurement project for web apps.

Contacts

Name Email address
Google Group - Hasal Project hasal-project@mozilla.com Send email if you need any support
Shako Ho sho@mozilla.com You can contact him to get more project detail.
Askeing Yen askeing@mozilla.com
Mike Lien mlien@mozilla.com
Walter Chen wachen@mozilla.com
Bobby Chien bchien@mozilla.com Program Management
Ken Chang kchang@mozilla.com Point of Contact for Engineering

State of this project

We are finish the testing for google doc in July. Now, we are shifting the focus to Facebook and Gmail.

Execution Summary

  • Google doc: Total we have 39 cases tested, 34 test cases indicate Firefox slower than Chrome. 6 cases how 30% gap between Firefox and Chrome.
  • Facebook: Pilot test result will come out before end of July.
  • Gmail: Pilot test result will come out before end of July.

Testing Result

  • Google doc testing result:
 You can find all testing result here Testing Result
 And the analysis diagram for module consume the most running time in each test cases could generate automatically like below:
 Summary 0518.png

Or you can download the XMind file here.

About this project

We are trying to measure the difference between Firefox and Chrome on high visibility web apps (Google Doc, Google Sheet, Google Slide, etc.)

We use the video recording to record the user actions and extract the frames to get the running time from user perspective. Based on same user actions, we measure the difference between Firefox and Chrome.

How to Trigger Try Build

  • Go to Jenkins server site using your browser
  • Login (top-right corner) using your LDAP
  • Trigger Try_Ubuntu and Try_Portal
       Jenkins1.png
  • Enter necessary information as requested
       Jenkins2.png
  • You can see the running progress and result artifacts after test finished


Working Measurements

Measurements

App Name Meta Bugs Reports Status
Google Docs 1260981 Pilot, Full User would aware performance issues on Firefox
Outlook Web Access (Office 365) 1275000 Pilot, Full No big issue per pilot test
Outlook Web Access (Exchange) Pilot, Full Suspended, awaiting measurable builds
Facebook 1288602 Pilot, Full Found issues, working on full test
Gmail 1288604 Pilot, Full No big issue per pilot test
BBC Pilot, Full
Youtube Pilot, Full


Google Docs

Full Query
ID Component Summary Status Assigned to Target milestone Depends on
1278554 Graphics: Canvas2D CanvasRenderingContext2D::DrawImage takes lots of time when scrolling Google Spreadsheet NEW --- 1237058
1307024 General [Perf] [google docs] Firefox will get 3~9% faster when running gdoc test cases with chrome user agent NEW --- 1321623

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Outlook Web Access (Exchange)

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Facebook

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


GMail

Bugzilla query error

Bugzilla API returned an error: 1


Testing Tool

Testing Framework

Following the steps here to setup the framework. For the time being, only Ubuntu is officially supported as a testing environment; you can use OS X but you'll have to manually solve some package dependency issue. Note that Ubuntu 12.04 and 14.04 is the best-supported testing OS.

Template VM

You can use vagrant to download the template VM with Hasal's automation framework. Please follow the instruction below:

  • vagrant init shako/hasal
  • vagrant up --provider virtualbox

Testing Data Summarize Tool

Following the steps here to setup the tool. For the time being, only Ubuntu is officially supported as a testing environment; you can use OS X but you'll have to manually solve some package dependency issue. Note that Ubuntu 12.04 and 14.04 is the best-supported testing OS.

Add-on

Performance mark on profiling results at pageMod start and end

You can install this add-on from Here

Sending Performance Mark by Hotkeys

You can install this add-on from Here

External Links

Retrieved from "https://wiki.allizom.org/index.php?title=Hasal&oldid=1162568"