QA/Browser Technologies/2011-12-01

« previous mtg | Browser Technologies Home | next mtg »

Discussion Items

• BHAG discussion (Matt)

Project Status

Fennec

Execution (kevin, aaron)

• Mobile
  • Mobile Developers are looking for additional testing in regards to recently landed patches (bug 627842) re: improving font size readability on Firefox for Android. Please file bugs if pages you view have odd looking font inflation (size) issues.
  • Encourage and continue to audit and track testing of feature coverage here.

Automation (martijn, John)

• Functional Frameworks
  • Have plans to look into the following frameworks:
    • Marionette
    • Robocop
    • SL4A code.google.com/p/android-scripting (not a test framework as such but can be fitted - takes care of Android API side stuff I have been doing until now with native java apps)
    • MonkeyRunner/Testdroid recorder http://testdroid.com/product/testdroid-recorder

Specialized (naoki)

• CrashKill :
  • reporting using P1 ~ P5 status is over. Using Topcrash, reproducible and QA+ for tracking, not sure how Topcrash/reproducible is going to change anything since that's always the way that it's been done
  • going to start graphing the number of crasher bugs in component areas to raise awareness of crash bugs

• Performance
  • Automation seems about on par with manual testing for local pages, not sure about automation. Probably need to compute a graph with mean and variance.
  • QA to own testing for the video performance testing; need to talk to Erin about video equipment/current methods
  • initial query of where we can go with tpan/tzoom; possible hooks within Robotium; also need to see how Eideticker might help. Eideticker may be done towards the end of this year.

• Memshrink :
  • currently nothing to report; need to investigate RSS/Profiling

• Newsletter
  • Need to talk to Al about emailing through generic email account
  • this week's newsletter to be also tweeted about, posted on yammer, facebook, and several other places (including the normal channels of planet, qmo, btech, mobile)

Sync

Client (Tracy)

• no client trains
• been working on test cases and testing tips videos

Server (James/John)

• James
  • Fairly quiet two weeks with only a couple maintenance releases that needed to be tested in Production.
  • Got John ramped up on all things Sync Server

Test pilot (tracy)

• new study for broken/incompatible add-ons released.

BrowserID (james)

• One very large release in Beta over the last two weeks:
  • Train 14: bug 703596 - QA and deploy BrowserID train-2011.11.17 to production
  • (Yes, 14 releases already!)
• All QA, Dev, and OPs work now is focused on launching to OPs environments: Dev, Stage, Test/CI, and Production
• Details: https://wiki.mozilla.org/QA/BrowserID/OPsBuildOut

Pancake (Naoki)

• Project Status:
  • Discover tab implemented and pushed to staging; twitter /facebook login also pushed, but incomplete.
  • Addon implemented; blocking bug of initial bookmark+history being pushed to server bug 679133
  • Addon to get fixed, need server changes to support the change. Once it gets fixed, some retesting and release of the addon with the reset of server to take place.
  • Revamp of concept being worked upon still

• Testing:
  • Testing has been done on the addon and staging server: see bottom of QA notes
  • Testing environment : Need to setup VM testing environment

• Automation
  • Unit Test Automation demoed to tchung; need more tests added
  • Selenium test from AMO received, need to port to Selenium 2 and section off the BrowerID login portion for Browser ID (ETA end of next week?)
  • Instruments portion : delayed

WebAPI (John)

• Project Status:
  • Currently landed: Battery, Camera, IndexedDB, Vibrator, SMS
  • Demoed these to media, blogpost
  • Need to add EXIF support to camera app
  • Need to add Upgrade scenarios to IndexedDB
  • Need to create test pagesL Sensor/Accelerometer
  • Camera landing on Windows soon
• Testing
  • need to add coverage matrix/testplans/testcases to all apis except Battery
• App support
  • Have Battery APP
  • Have Info App
  • Need Geolocation/Accelerometer APP

Notes

Action Items

Action Items from This Meeting

• jbonacci and nhirata to discuss at least a sync newsletter if not a btech newsletter

Past Action Items

• [DONE] mobile team to meet at 1:00
• [DROPPED] b-tech team: please try out crowdbot for fennec xul; and give feedback
  • martijn : send out link to the crowdbot to the team
• tchung : talk about impact of other communities in the QA community meeting
• tchung : give bullets for browser tech on visibility for outreaching; show communities for each individual goals for people.
• tchung and mevans to talk about crowdbot off line
• [DONE] jbonacci : dashboard would be good idea. Spreadsheet is fine as well; relevant bugs, status, checklist items.
  • https://wiki.mozilla.org/QA/BrowserID/OPsBuildOut

Raw Notes

BHAG : getting out the door
- clarity of questions
- Big Hairy Audacious Goals
	- want to be able to define the direction
	- what are we trying to accomplish as a QA org
	- not just the weekly goals
	- How does a BHAG differ from team goal?
	- Let's group them up
		- Products
		- Automation
			- Sync services : automation
				- data to prove it's not ready
			- Tool for webkit CSS spidering
			- drive cross platform, cross browser automation testing on these products (Browser ID)
			- How can we make this a QA goal? - segrata(?)/pyrmamid
				- not going to write tests for every app, but maybe a template, or samples of tests
				- Need more information
				
		- Communication
			- Let's have communication with Dev
				- what's in our current process that can be improved?
					- Fennec : There's couple of broken things in the communication
						- triaging bugs
						- equal amount of say : go/no go , proving with data, etc.
								- communication w/ keeping up on test case, test case management
		- leading the edge, rather than trailing the edge
		- how is that different; how can we do better?
		- Bugzilla study work, Dev gets assigned with bugs, -> assigning bugs to QA until they are signed off
			- Improve by bug workflow and process communication : dev should pass the bugs to QA
		- better management of flags and follow up.
		- better recommendations (submit tests w/ patches); unit tests

					
		- Community
			- how can we get productivity, excitement, etc. from our community
				- test day : we don't want them to disappear after one day
				- how do we get it better?
				- localized test days around the region/world
				- Here's a schedule of test days around the world
				- moderate their own
				- huge goals
				- every quarter we can break it down based on regions and helping them out
			- quarterly goals will get broken down from the BHAGs
			- webkit CSS testing
			- we want to drive automated tools that people can easily contribute
				( http://testdroid.com/testdroid/547/testdroid-recorder-1-1-11-is-out )
			- define some sort of metric to raise awareness per product.  each team to define metrics
			- community dashboard
					- device anywhere, fennec work, etc. : limitting
		- continue to make what contributive to proactive members like gabriella
		- Target community members and mentor them into community leaders
			- motivation techniques

						
		- Manual testing
			- test plans and test days, but it's starting with blank page
				- maybe have templates or something
				- have global templates or duielines for creating testplans and testcases [tools]
			- how do we write a test plan
			- standardize metrics that we collect and results and what data needs to be...
				- metrics team already doing this
				- AMO, webteam doing metrics already
			- certain metrics for each of the products to display to the rest of the world?
			What are the number of bugs, what location of test cases, how many run, for each webapp?
				- dashboard
				- code coverage
			
	Cross team cross identify
	Firefox + Mobile now
	Next year is going to be different playing game:
		- OWAs
	- What are our goals?
		- OWAs is a glue to a lot of projects
	- Joint bug tracking systems
	- test environment 
	- how can we have a SLA?
		- we need QA : where do we start?  What's our service agreement
		- what framework, services, equipment, schedule, specs, plans, etc.
		- quality should be driven from QA not the other way around and an afterthought
		- should be testable before having the patch
		- some of it is changing the perception amongst the company

	- small business, and organizations that are picking up OWA and Browser ID
		- need to share resources with noncommunity, nonmozilla emp
		- this will change how we do QA; new audiences businesses
		- pick up a secondary
		- how do we coordinate testing efforts?
		- as other people pick up the product
		- web app, store.  the apps that go in the store
		- should be a similar model
		- landing amo, review process, etc. upgrades
		- browser ID?
		- cross browser will be a big difference
		- mozilla backed product has to be useful everywhere in web
		- webkit QA
		
- BHAG for mobile: website compatibility
	- webkit css 
	- structure doesn't work well on these sites; for cross browser
	- can we write some tools to spider the sites?
	- to throw them out to community and report bugs?
	- https://wiki.mozilla.org/Mobile/Evangelism
	- website compatibility
	- work with evangelism, SUMO, tool to crawl websites and detect webkit css
	- spread the testing to community
	- Marketting
	
- Automation : cross platform
	- browser id people are having some discussions; AMO, and other teams
	- need automation for cross product, cross browser
	- selenium 2
	- drive cross platform, cross browser automation testing
	- have some ideas on how we see ; sense server side testing
	- SL4A scripting language
	- get stuff from platform API (Android) to get information that we need, need to becareful of privacy
	- when we know what information that we want to get, we could potentially get it.
	- by working with an automation for frameworks
		- need to maintain the system
		- Robotium : SL4A python
		- http://testdroid.com/testdroid/547/testdroid-recorder-1-1-11-is-out
	- Tracy: concern for automation, keeping up with automation?  regression testing?  automation?  Sync?
		- mozmill + TPS should be able to automate manual testing that we come up with
		- if it's just on Tracy, it's a BHAG on top of everything else that Tracy 
			- push Tracy to do it for a Quarter or hire someone else
		- need framework, what has been done, etc. 
		- more automation for back features
		- w/ TPS it has a listener, so the test cases should be in continuous integration frameworks
	- Mobile : BHAG
		- automation that's a big outstanding goal
		- (moved stuff)
		- martijn : crowdbot - getting that for native ui
			- crowdsource tools that makes it simple for folks to run for cross-platform products
			- results/data/reporting driven
	- lower the barrier of entry, compelling and fun
	- give them a reason for them to join
		- a fun, interesting project
		- gaming, social interaction?
		- combine them?
	- automation of recording and pushing scripts to review for community
	- Performance process tools
		- eiderticker : recording : need to investigate this BHAG
	- Having a simplier for bug submissions on phones
	- something that captures the data, screenshots, bugs, etc.
	- how to submit bugs

- BHAG form and context underneath it
	- need the context for what areas that are being filled in
	- the quarterly goals would be the last step
	- large goals ideas
	- poin pointed specifics
	- crossbrowser specifcs
	- browser id : business QA as well