QA/Fennec/Play HLS: Difference between revisions

From MozillaWiki
< QA‎ | Fennec
Jump to navigation Jump to search
 
(38 intermediate revisions by 3 users not shown)
Line 6: Line 6:
Developer contact: [mailto:x@y.com X]  
Developer contact: [mailto:x@y.com X]  


QA: [mailto:mihai.ninu@softvision.com :Ninu]
QA: [mailto:mihai.ninu@softvision.ro :Ninu]
*From 2017-06 QA: [mailto:sorina.florean@softvision.ro Sorina]
*From 2017-11 QA: [mailto:oana.horvath@softvision.ro Oana]


== Overall Status ==
== Overall Status ==


<small>'''<font color="green">[LANDED]</font>'''</small> Nightly 50.0a1: first landed on - '''30-07-2016 '''
<small>'''<font color="green">[LANDED]</font>'''</small> Nightly 56.0a1


<small>'''<font color="green">[ LANDED]</font>'''</small> Aurora 50.a2: first landed on - '''02-08-2016 '''
<small>'''<font color="green">[LANDED]</font>'''</small> Nightly 59.0a1


<small>'''<font color="red">[NOT LANDED]</font>'''</small> Beta 50: NA
<small>'''<font color="green">[LANDED]</font>'''</small> Beta 59: 1/23/2018


= Testing Summary =  
= Testing Summary =  
Line 26: Line 28:
Testing will be performed on following devices:  
Testing will be performed on following devices:  
* Phones
* Phones
** 4.x
** From Android 4.4 to 7.1
** 5.x
** 6.0
* Tablet
* Tablet
** 4.x
** From Android 4.4 to 7.1
** 5.x
** 6.0


== Specifications ==
== Specifications ==
*information about the feature
**how to use it
*[[File:HLS Player Portrait.png|600px|thumb|center| HLS Player Landscape]]
**turn on/off feature
*[[File:HLS Player Landscape.png|600px|thumb|center| HLS Player Portrait]]
*screenshots


== User stories ==
== User stories ==
*As a user, I want to
* As a user, I want to be able to play HLS videos without glitches and streaming problems, while the other app features are not impacted by a playing video.
*As a user, I want to
*As a user, I want to


== Acceptance Criteria==
== Acceptance Criteria==
Line 50: Line 45:
= Testing details =
= Testing details =
== Test Cases==
== Test Cases==
 
* https://docs.google.com/spreadsheets/d/1tZaPIq9kZ9PPEFO_8O3EcHGsxl0b3H3hYUqpgXJJVQY/edit#gid=0


Step  
Step  
Line 85: Line 80:


== Testing days==
== Testing days==
===Date:07-09-2016===


===Date:xx-yy-zz===
===Date:05-09-2016===
===Date:xx-yy-zz===
* Exploratory testing with focus on the Aurora build and creating a test plan for the feature


===Date:07-09-2016===
* Functional testing with focus on both Aurora and Nightly builds
Logged issues:
*  HLS video player overlaps other tabs once played - {{bug|1301043}}
*  A misleading error is received before playing an HLS video - {{bug|1301053}}
*  A HLS video cannot be played twice without reloading the whole page - {{bug|1301055}}


===Date:08-09-2016===
*  Creating test cases and exploratory testing on both Aurora and Nightly builds
Logged issues:
*  HLS videos automatically start playing when auto-play is OFF - {{bug|1301326}}
===Date:03-10-2016===
* Functional testing with focus on all: Beta Aurora and Nightly builds
Logged issues:
* Switching tabs will stop the HLS player - {{bug|1307074}}
* HLS Player breaks other video types in other tabs - {{bug|1307076}}
* Interrupting an HLS video with Task manager will lead to incorrect video resume - {{bug|1307078}}
===Date:22-06-2017===
* Exploratory testing with focus on links from duplicated bugs and GitHub issues with devices Asus ZenPad 8 (Android 6.0.1) and LG G4 (Android 5.1), build Nightly 56.0a1 from 2017/06/21
* New bugs found during testing
**{{Bug|1375478}} - HLS: "Full-screen" button not displayed in the right position
**{{Bug|1375474}} - Video doesn't play on observator.tv
===Date:03-07-2017===
* TCs and exploratory testing with devices Asus ZenPad 8 (Android 6.0.1) on build Nightly 56.0a1 from 2017/07/03
** Prerequisite: "media.mediasource.enabled" set to false
* New bugs found during testing
**{{Bug|1377873}} - [Fennec][HLS] Issues when changing device orientation
**{{Bug|1377891}} - [Fennec][HLS] Issues with videos from twitch.tv
===Date:06-07-2017===
* TCs and exploratory testing for Mid-Nightly Signoff on latest Nightly build
** Prerequisite: "media.mediasource.enabled" set to false
===Date:31-07-2017===
* Pre-Beta SignOff: GREEN
** https://testrail.stage.mozaws.net/index.php?/reports/view/485
===Date:22-11-2017===
* Exploratory testing and bug investigation on latest Nightly build (59.0a1-21/11/2017)
**{{Bug|1307074}} - Switching tabs will stop the HLS player
**{{Bug|1307078}} - Interrupting an HLS video with Task manager will lead to incorrect video resume
**{{Bug|1308205}} - While in landscape, HLS videos are played in FullScreen
**{{Bug|1302729}} - Media control notification is not displayed when HLS video is playing
**{{Bug|1307076}} - HLS Player breaks other video types in other tabs
**{{Bug|1310899}} - HLS VideoView prevents interacting with page, requires back button to close
**{{Bug|1377891}} - [Fennec][HLS] Issues with videos from twitch.tv.
===Date:23-11-2017===
*Feature testing on Nightly 59:
**Huawei P9 Lite (Android 6.0): https://testrail.stage.mozaws.net/index.php?/runs/view/7316
**Lenovo Yoga Tablet 2 (Android 4.4.2): https://testrail.stage.mozaws.net/index.php?/runs/view/7317
*Logged bug:
**{{Bug|1420192}} - Youtube and twitch.com live videos don't work when auto-play is off
===Date:27-11-2017===
*Feature testing on Nightly 59:
**Google Pixel (Android 8.0): https://testrail.stage.mozaws.net/index.php?/runs/view/7336
**Huawei MediaPad M2 (Android 5.1.1): https://testrail.stage.mozaws.net/index.php?/runs/view/7339
===Date:5-12-2017===
*Feature testing on Nightly 59:
**Samsung Galaxy Note 4 (Android 5.0.1): https://testrail.stage.mozaws.net/index.php?/runs/view/7473
**Mid-Nigthly signoff: Green
===Date:20-12-2017===
*Pre-Beta signoff: Green
===Date:25-01-2018===
Feature full testing on Beta 59:
*Prestigio Grace X5 (Android 4.4.2)
*Huawei Nexus 6P (Android 8.0)
*HTC Desire 820 (Android 6.0.1)
===Date: 30-01-2018===
Feature full testing on Beta 59:
*LG G4 (Android 5.1)
*Samsung Galaxy S8 (Android 7.0)
=Bug Work=
=Bug Work=
=Bug Work=
==Tracking bugs==
* Meta bug
* bugs to track the feature
**{{Bug|1345752}} - (HLS_on_Fennec) [META][HLS] Make Gecko native support HLS on Fennec 
ex:
<onlyinclude>
<onlyinclude>
<bugzilla>
<bugzilla>
{
{
"id": "1286133, 1286898, 1295398, 1294823, 1243304, 1274864, 1301043, 1301053, 1301055, 1301050, 1301326 "
    "blocks": "1345752",
    "include_fields": "id, priority, summary, status, assigned_to",
    "order": "Status"
}
}
</bugzilla>
</bugzilla>
</onlyinclude>
</onlyinclude>
===Bug fixed ===


=Signoff=
=Signoff=
Line 111: Line 187:
== Results ==
== Results ==
'''Nightly testing'''
'''Nightly testing'''
 
*Done
 
'''Aurora testing'''
 
 
'''Beta testing'''
'''Beta testing'''
NA
*n/a

Latest revision as of 09:28, 14 February 2018

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that HLS video can be correctly played in any given condition.

Ownership

Developer contact: X

QA: :Ninu

Overall Status

[LANDED] Nightly 56.0a1

[LANDED] Nightly 59.0a1

[LANDED] Beta 59: 1/23/2018

Testing Summary

Scope of testing

The testing will be focus on

  • making the HLS video watching experience good
  • making sure that the implementation of HLS video controls don't impact other video playing type

Environments

Testing will be performed on following devices:

  • Phones
    • From Android 4.4 to 7.1
  • Tablet
    • From Android 4.4 to 7.1

Specifications

  • HLS Player Landscape
  • HLS Player Portrait

User stories

  • As a user, I want to be able to play HLS videos without glitches and streaming problems, while the other app features are not impacted by a playing video.

Acceptance Criteria

Testing details

Test Cases

Step 

1. Go to goo.gl/LLxEY8 and play the HLS video
R: The video plays correctly and the audio is correctly played

Step 

1. Go to goo.gl/LLxEY8 and play the HLS video in landscape/portrait
R: The video plays correctly and the audio is correctly played
2. Change the device's orientation 
R: The video keeps playing and no interruptions or glitches are triggered

Step 

1. Go to goo.gl/LLxEY8 and play the HLS video
R: The video plays correctly and the audio is correctly played
2. Try the video-player media controls
R: All buttons response is correctly applied in the video progress

Step 

1. Go to goo.gl/LLxEY8 and play the HLS video
R: The video plays correctly and the audio is correctly played
2. Close HLS video player and go to youtube, play a video
R: Any other video player is not impacted by the HLS video player previous use

Step 

Pre-Condition:  Make sure Auto-Play is ON
1. Go to goo.gl/LLxEY8 and play the HLS video
R: The video starts Auto-playing

Step 

Pre-Condition:  Make sure Auto-Play is OFF
1. Go to goo.gl/LLxEY8 and play the HLS video
R: The video should not start Auto-playing

Testing days

Date:05-09-2016

  • Exploratory testing with focus on the Aurora build and creating a test plan for the feature

Date:07-09-2016

  • Functional testing with focus on both Aurora and Nightly builds

Logged issues:

  • HLS video player overlaps other tabs once played - bug 1301043
  • A misleading error is received before playing an HLS video - bug 1301053
  • A HLS video cannot be played twice without reloading the whole page - bug 1301055

Date:08-09-2016

  • Creating test cases and exploratory testing on both Aurora and Nightly builds

Logged issues:

  • HLS videos automatically start playing when auto-play is OFF - bug 1301326

Date:03-10-2016

  • Functional testing with focus on all: Beta Aurora and Nightly builds

Logged issues:

  • Switching tabs will stop the HLS player - bug 1307074
  • HLS Player breaks other video types in other tabs - bug 1307076
  • Interrupting an HLS video with Task manager will lead to incorrect video resume - bug 1307078

Date:22-06-2017

  • Exploratory testing with focus on links from duplicated bugs and GitHub issues with devices Asus ZenPad 8 (Android 6.0.1) and LG G4 (Android 5.1), build Nightly 56.0a1 from 2017/06/21
  • New bugs found during testing
    • bug 1375478 - HLS: "Full-screen" button not displayed in the right position
    • bug 1375474 - Video doesn't play on observator.tv

Date:03-07-2017

  • TCs and exploratory testing with devices Asus ZenPad 8 (Android 6.0.1) on build Nightly 56.0a1 from 2017/07/03
    • Prerequisite: "media.mediasource.enabled" set to false
  • New bugs found during testing
    • bug 1377873 - [Fennec][HLS] Issues when changing device orientation
    • bug 1377891 - [Fennec][HLS] Issues with videos from twitch.tv

Date:06-07-2017

  • TCs and exploratory testing for Mid-Nightly Signoff on latest Nightly build
    • Prerequisite: "media.mediasource.enabled" set to false

Date:31-07-2017

Date:22-11-2017

  • Exploratory testing and bug investigation on latest Nightly build (59.0a1-21/11/2017)
    • bug 1307074 - Switching tabs will stop the HLS player
    • bug 1307078 - Interrupting an HLS video with Task manager will lead to incorrect video resume
    • bug 1308205 - While in landscape, HLS videos are played in FullScreen
    • bug 1302729 - Media control notification is not displayed when HLS video is playing
    • bug 1307076 - HLS Player breaks other video types in other tabs
    • bug 1310899 - HLS VideoView prevents interacting with page, requires back button to close
    • bug 1377891 - [Fennec][HLS] Issues with videos from twitch.tv.

Date:23-11-2017

Date:27-11-2017

Date:5-12-2017

Date:20-12-2017

  • Pre-Beta signoff: Green

Date:25-01-2018

Feature full testing on Beta 59:

  • Prestigio Grace X5 (Android 4.4.2)
  • Huawei Nexus 6P (Android 8.0)
  • HTC Desire 820 (Android 6.0.1)

Date: 30-01-2018

Feature full testing on Beta 59:

  • LG G4 (Android 5.1)
  • Samsung Galaxy S8 (Android 7.0)

Bug Work

Bug Work

  • Meta bug
    • bug 1345752 - (HLS_on_Fennec) [META][HLS] Make Gecko native support HLS on Fennec
Full Query
ID Priority Summary Status Assigned to
577084 -- Implement HTTP Live Streaming (HLS) for audio and video on Android RESOLVED
1341990 -- Introduce Exoplayer into build system RESOLVED Kilik Kuo [:kikuo] (inactive)
1350241 -- [Fennec][HLS] Create a demuxer-like component based on ExoPlayer. RESOLVED Kilik Kuo [:kikuo] (inactive)
1350246 -- [Fennec][HLS] Provide HLSDecoder / HLSDemuxer / HLSTrackDemuxer / HLSResouce for Gecko. RESOLVED James Cheng[:JamesCheng]
1350250 -- [Fennec][HLS] HLSResource should obtain buffer information from GeckoHlsPlayer.java RESOLVED James Cheng[:JamesCheng]
1350253 -- [Fennec][HLS] HLSDemuxer/HLSTrackDemuxer in Gecko should be able to get media & sample information from GeckoHlsPlayer.java RESOLVED Kilik Kuo [:kikuo] (inactive)
1350842 P3 Remove codes which are related with using external video player to play unsupported type media RESOLVED
1357984 P5 [META] Test case for HLS native supprts RESOLVED
1365227 -- AVCC NAL converter should generate extradata. RESOLVED Jean-Yves Avenard [:jya]
1365505 -- [Fennec][HLS] Turn on HLS native support not only for Nightly RESOLVED Kilik Kuo [:kikuo] (inactive)
1368904 P3 [Fennec][HLS] Handle unexpected OnError native callback from java GeckoHlsPlayer RESOLVED Kilik Kuo [:kikuo] (inactive)
1368907 -- [Fennec][HLS] Unify the naming from Hls into HLS for our Gecko HLS implementation RESOLVED James Cheng[:JamesCheng]
1368954 -- [Fennec][HLS] Separate source code which will affect generated JNI wrappers to avoid build bustage. RESOLVED Kilik Kuo [:kikuo] (inactive)
1368959 -- [Fennec][HLS] Set the MediaInfo duration to zero if the source is live streaming RESOLVED James Cheng[:JamesCheng]
1369598 -- [Fennec][HLS] Make HLSDemuxer initialized after underlying GekcoHlsPlayer is ready with data. RESOLVED Kilik Kuo [:kikuo] (inactive)
1371247 P5 [Fennec][HLS] Fix potential defects in exoplayer2. RESOLVED
1372501 -- [Fennec][HLS] Allow VideoDocument supporting HLS RESOLVED James Cheng[:JamesCheng]
1372841 -- Make HLSResource inherit from MediaResource instead of BaseMediaResource RESOLVED James Cheng[:JamesCheng]
1373119 -- [Fennec][HLS] Fix "explicit null dereferenced" case in GeckoHLSDemuxerWrapper.getSample RESOLVED Kilik Kuo [:kikuo] (inactive)
1373162 -- Crash in java.lang.NullPointerException: at org.mozilla.gecko.media.GeckoHlsPlayer$ComponentListener.onVideoInputFormatChanged(GeckoHlsPlayer.java) RESOLVED Kilik Kuo [:kikuo] (inactive)
1373377 -- Crash in java.lang.IllegalArgumentException: Unsupported type: 3 at org.mozilla.gecko.media.GeckoHlsPlayer.buildMediaSource(GeckoHlsPlayer.java) RESOLVED James Cheng[:JamesCheng]
1373500 -- [Fennec][HLS] Set DEBUG flag based on AppConstants.NIGHTLY_BUILD || AppConstants.DEBUG_BUILD RESOLVED James Cheng[:JamesCheng]
1374240 P1 [Fennec][HLS] Video sound stops working after a few seconds on Dailymotion RESOLVED James Cheng[:JamesCheng]
1374899 -- [Fennec][HLS] Handle InvalidResponseCodeException exception raised by Exoplayer with some url that may cause redirection. RESOLVED James Cheng[:JamesCheng]
1374953 P3 [Fennec][HLS] Take mimetype "audio/mpegurl" as HLS media type RESOLVED James Cheng[:JamesCheng]
1375474 P1 Video doesn't play on observator.tv VERIFIED James Cheng[:JamesCheng]
1375478 -- HLS: "Full-screen" button not displayed in the right position RESOLVED
1376966 P3 [Fennec][HLS] Fix defects in GeckoHlsPlayer.java RESOLVED James Cheng[:JamesCheng]
1376967 P3 [Fennec][HLS] Fix defects in GeckoHlsRendererBase.java RESOLVED Kilik Kuo [:kikuo] (inactive)
1377520 P3 [Fennec][HLS] Support encrypted HLS content playback RESOLVED
1377891 P5 [Fennec][HLS] twitch.tv does not play when media.mediasource.enabled is set to false. RESOLVED
1378852 -- [Fennec][HLS] getBufferedPosition in GeckoHlsPlayer should always return a value >= 0. RESOLVED Kilik Kuo [:kikuo] (inactive)
1379056 -- [Fennec][HLS] GeckoHlsPlayer should handle the case when obtain a negative duration from ExoPlayer. RESOLVED Kilik Kuo [:kikuo] (inactive)
1379856 P3 [Fennec][HLS] Move ExoPlayer main message handler to non-gecko main thread. RESOLVED Kilik Kuo [:kikuo] (inactive)
1379866 P1 [Fennec][HLS] App crashed a while later after stopping HLS playback. RESOLVED James Cheng[:JamesCheng]
1380201 P3 [Fennec][HLS] Make access to mIsPlayerInitDone & mResourceCallback guarded. RESOLVED Kilik Kuo [:kikuo] (inactive)
1380237 -- [Fennec][HLS] Handle audio format change by changing the stream id. RESOLVED James Cheng[:JamesCheng]
1381717 P3 [Fennec][HLS] Fix code defects found by Coverity in GeckoHlsPlayer.java RESOLVED James Cheng[:JamesCheng]
1382151 -- [Fennec][HLS] Pause ExoPlayer when player is paused to avoid OOM. RESOLVED James Cheng[:JamesCheng]
1384578 -- Crash in java.lang.NullPointerException: Null native pointer at org.mozilla.gecko.media.GeckoHLSDemuxerWrapper$Callbacks.onError(Native Method) RESOLVED James Cheng[:JamesCheng]
1387963 -- HLS tests crash on all versions Autophone RESOLVED
1418766 P1 Crash in java.lang.OutOfMemoryError: at com.google.android.exoplayer2.upstream.DefaultAllocator.allocate(DefaultAllocator.java) RESOLVED James Cheng[:JamesCheng]
1419249 P2 [Fennec][HLS] Audio only playback will not work correctly after seek. RESOLVED James Cheng[:JamesCheng]
1419669 P3 [Fennec][HLS] Refine switching Exoplayer to play/pause logic in GeckoHlsPlayer RESOLVED
1420030 -- [Fennec][HLS] Change HLS logging from BuildConfig.NIGHTLY_BUILD || BuildConfig.DEBUG_BUILD to !BuildConfig.MOZILLA_OFFICIAL RESOLVED James Cheng[:JamesCheng]
1420192 -- Youtube and twitch.com live videos don't work when auto-play is off RESOLVED Alastor Wu [:alwu]
1424168 P2 [Fennec][HLS] Exoplayer causes high memory usage on some websites. RESOLVED James Cheng[:JamesCheng]
1478843 P1 Cross-origin audio leak in HLS VERIFIED Paul Adenot (:padenot)

48 Total; 0 Open (0%); 46 Resolved (95.83%); 2 Verified (4.17%);


Bug fixed

Signoff

Criteria

  • All test cases should be executed
  • All blockers, critical issues must be fixed and verified or have an agreed-upon timeline for being fixed


Results

Nightly testing

  • Done

Beta testing

  • n/a
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Fennec/Play_HLS&oldid=1188999"