QA/StaffMeetings/QA-Staff-10-12-2011

« previous mtg | QA Staff Home | next mtg »

QA Staff Meeting Notes Oct 12, 2011

Meeting Details

 # Every other Weds @ 1:30p Pacific Time - Mozilla HQ, Warp Core 
 #
 # Vidyo:  https://v.mozilla.com/flex.html?roomdirect.html&key=UK1zyrd7Vhym
 # Phone:  +1 650 903 0800 or 650-215-1282 x92 Conf# 95312 (US/INTL)
 #         +1 800 707 2533 (pin 369) Conf# 95312 (US Toll Free)
 #         +1 416 848 3114 x92 Conf# 95312 (Canada)
 # IRC:    irc://irc.mozilla.org:6697/#qa

QA Org Items

• New Hires
  • John H, Mobile QA automatoin - Already here 4 weeks (say hi!)
  • John M, Sevices QA team - ETA 11/7
• Upcoming Events
  • Mobile summit Mon 10/17/11. Attendees: Aaron, Kevin, Naoki, Martijn
  • Ioana Chiorean and Vlad Maniac of Softvision will visit Mozilla Office between OCT 19-27
  • GTAC meet & greet mixer https://mozqa.sync.in/mozwebqa-GTAC-party-planning
  • [unconfirmed] Naoki and Tony to represent QA at MozCampAsia. Proposals still in flux. Nov 19-20

• PTOs
  • [juanb]: Oct 13, 14; wedding in Seattle (my brother's).
  • tchung: oct 16-28, pto, hawaii.
  • Dave Hunt: Any day now, for 2 weeks.
  • Henrik Skupin: Oct 13, 14
  • Tracy: afternoon of 10/13 and all day 10/14
  • Cameron: Oct 20, 21

Discussion Items

• overview of 7.0.1 postmortem - Anthony
  • general take-away:
    • QA needs to be more aggressive in helping to reproduce bugs, diagnose severity and prevalence
    • PM needs to be more aggressive in making sure issues are visible to QA
    • Update testing strategy cannot rely solely on automation
  • Updates strategy going forward -- see QA project page
    • will be two tiered: automation and manual testing with various rich profiles
    • time-to-market should be roughly the same given that builds now come out Wednesday morning and some of this testing can be done in parallel (manual on betatest, automation on releasetest)
• q3 goals overview https://intranet.mozilla.org/QA/Q32011_QAgoals
  • team leads any take aways from Q3?
• q4 goals review https://intranet.mozilla.org/QA/Q42011_QAgoals
  • matt added the top level org goals
  • add bhags to team goals (etherpad for each team)
• scrumpads discussion
  • move to new system
  • overview of usage
• pass down from directors meeting
  • native fennec
  • win8
• Progress on Team Slides?
  • Would like these in advance of some of the upcoming events.
  • 2-3 Slides that show easy paths to contribute to your particular team area

QA Team Updates

Desktop Firefox (Juan, Anthony)

• Firefox 8 (juanb): Testing in progress, to be released Friday morning.
• Firefox 9 (ashughes): No status update
• Firefox 10 (juanb): Heads up on probable compatibility issues due to two digit version. Web Compatibility test run in planning phase.
• Crash Status and Investigation
  • Weekly Stability Report
• Maintenance and Security ESR (Al)
  • No new updates at this time.
• New Lab Operating Systems Available
  • Windows 8 Developer Preview
  • Windows 2000

Browser Technologies (Tony)

• Issues
  • Native Fennec for Android project announced this week as a top Priority, starting Firefox 10. QA brainstorm here
  • The "503 meltdown of 2011" post mortem is scheduled 10/13
• Fennec (Aaron/Kevin)
  • Performance Benchmark baseline to be established
  • Crash reporting for Fennec working with Socorro team
  • biweekly newsletter being sent out
  • Toronto Mobile Workweek next week
• Sync (Tracy)
  • Sync fell over last week due to servers not being prepared for load increase of Instant Sync released in Fx7. Client was also not correctly respecting back-off requests from the server which made it difficult for OPs to repair the servers on the fly. Post-mortem scheduled for 10/13
  • Setup process has been somewhat simplified and Mobile devices can now initiate device pairing process before a sync account is even setup.
  • Mihaela from Softvision is getting up to speed with weekly client train cycles and is starting to take on some test case writing.
• Test pilot (tracy)
  • Test Pilot for Mobile was released last week as was updated version for desktop
• Sync Server (jbonacci)
  • Deployments have been delayed for a couple of weeks due to the sync/503 meltdown and other issues
  • Upcoming (planned) deployments:
    • Bug 693385 - deploy updated server-core (2.6.x) to syncstorage
    • server-node-assignment rpm-1.6-1
    • Something for Account Portal
    • Bug 686485 - roll out membase
• BrowserID (jbonacci)
  • BrowserID has taken a giant step forward with the current Beta train: 10/06/2011
    • New functionality, certs support, IE9 support, and, of course, the new UI
  • BrowserID Test Day 1 is coming up this Friday, 10/14/2011
  • ==> QA NEEDS MODERATORS! ... https://etherpad.mozilla.org/testday-20111014 <==
• Pancake (nhirata)
  • .17 Release
  • backend algorithm is being revamped
  • Places is being used

WebQA (Stephen)

• AMO
  • AMO 6.2.6 went live 10/06
  • goals are set
  • working on paypal testing for marketplace
  • refactoring page objects for themes
• Affiliates - on track to dark launch on 10.13.11
• Input - Q4 bug triaging in-progress
• Mozilla.com
  • working on 4.1 release
• MDN
  • 1.3 milestone shipped 10.11.2011
• Socorro
  • 2.3 milestone shipped on 10.10.11
  • Softvision is very active on the project
• SUMO
  • Released SUMO 10/11
  • In process SUMO 10/18
• Mozillians
  • Automation complete, can be seen in Jenkins: http://qa-selenium.mv.mozilla.com:8080/view/Mozillians/
  • Pushed Mozillians 1.0.1 last Thursday

Community (Marcia)

• Community invites submitted for both MozCamp Europe and MozCamp Asia.
• Second Draft of QA Sig - https://docs.google.com/document/d/1DZBKZl0mSXZjwcwFbnRmx1VfSZ8vcK2GMJbi9W-fwKA/edit?hl=en_US&pli=1
• Mozilla Festival - November 4-6, 2011, London.
  • marcia and henrik attending
• Mozilla Stewards Questions
  • If interested, please attend the Contribute Group meeting tomorrow at 10 AM Pacific as I have asked David Bowsell to clarify if they want to have stewards across various teams.

• Video Work
  • Revised nightly video has been completed and a bug has been filed to get it upon the Nightly First Run page.

• QMO.next (Al):
  • Should be updating Buddypress to version 1.5 with an updated theme on Tuesday or Thursday next week, depending on Craig Cook's time.
  • We're pulling from Github to our new Dev server every 10 minutes automatically now, making testing changes easier.

QA Automation Service (Henrik, David)

• General
  • Team was successful last quarter by accomplishing all listed team goals
  • Q4 goals have been finalized: https://intranet.mozilla.org/QA/Q42011_QAgoals#QA_Automation_Services_Goals
  • Henrik will participate in Mozilla Festival and MozCamp Europe in Berlin, proposals have been submitted
• Selenium
  • Selenium 2.8 has been released
• Mozmill
  • Mozmill 1.5.5 has been released
  • Mozmill 2.0rc1 has been released
  • A-team has started work on Mozmill 2.1 (e10s support)

Project updates

• Project pages for new projects have to be created and leads have to be assigned:
  • Fully automated Mozmill tests for releases
  • Prototype of Fennec driver
  • Community Documentation
  • CI system for daily and localized Firefox builds
  • Selenium Grid for Android and iOS
  • Long-term Goals for 2012

Case Conductor

• Marching forward to get our staging instance up and running with IT. Keep working through platform and logistical issues as they come up.
• Working toward the 0.6 milestone for the uTest Demo
• Demo'd the product to the uTest designer they just hired.
• Carl Meyer will visit Mt. View and be in-da-house Oct 17th & 18th.
• Q4 goals done, but haven't yet gone over them with Matt.

Notes, Takeaways and Action Items

Today

• please sign up to help with BrowserID Testday this friday, Oct 14

Action Items

• mevans to pool together ideas for next meeting about this quarter and future
• marcia to create a template for team slides
• bmoss to chase down who is going to mozcamp (w/ mary colvig)
• (everyone) Please sign up for test day on friday BrowserID

Last Meeting

• Action Items
  • draft of Q4 goals for teams by tuesday
  • draft of Q4 goals for individuals by end of week
  • add feedback to workweek for those that haven't commented yet
  • mevans: schedule of post mort
    • how to respond on emergency situation; checklist to be followed/consulted
  • marlenac to post as QMO event about E. Hendrickson's brown bag
  • all: overall presentation; each group does it's talk
  • Marlena/David : event planning for GTAC
  • Mevans : ask James Whittiker for brown bag
  • David C: Blog post about his unit test development framework
  • Juan : update tests for chemspill issue
    • Anthony taking ownership -- see tracking page

This Meeting Raw Notes

need to change the QA meeting to vidyo

- bullet points : Gtac mixer, email david burns, marlena, mattb, or mattevans
mbrandt|mtg: nhirata|meeting: here's the url to the mixer -> http://www.gtac.biz/home/announcements/meetgreetmixer
- PTO: look at wiki
- 7.0.1 Post morteum
- general take away : QA need to be more aggressive on triage
- PM needs to help to make this more visual, and make QA more aware of these issues
- our own part, update has been 100 % on automation
- need to adopt a similar model to functional testing
- automation + manual testing
- ashughes to take testing / strat for update testing
 - rich profiles testing
 - shoe horning for manual testing?
 - shouldn't effect that much
 - build gets 24 hrs earlier 
 - can do manual testing in parallel to automation
 - effects should be negligible
- why no automation?
 - under some testing such as addon testing?
 - testing with different plugin version
 - richer profile testing, is necessary 
 - they don't have to be linked to the hip
 - release testing can be automated
 - should investigate
 - updater code changes
 - some things matter on the updater, some don't
 - real time makes sense
 - should check, Rob Strong/Rel Eng
 - update testing has always been a concern
 - renewed 
 - also important with silent update
 - test plan concerns ?
 - Ashughes has taken over as a lead
 - automation strat
 - update testing requirements need to be nailed down
 - much richer profile testing
 
- Q3 goals wrapped up
 - action item from the last staff meeting for team leads to give update
 - from the team leads, wrap up/take aways from Q3 goals:
 	- Stephen : web apps, etc utilized dave hunt's plugins, helped out quite a bit
 	 - LDAP to jenkins hook up
 	 - open up that even further with IT
 	 - automation specific
 	- Juanb : Q3 goals met, relied to a large extent on Softvision
 	- going forward, we need to change things to scale for the future
 	- keeping up but will end up getting tougher
 	- Tony : mobile side, following Firefox lead
 	- did miss a goal, for crowd source tool
 	- did get the test working, but not logging
 	- need to get logging working and deploy to crowd
 	- downtime for features on sync
 	- plumbing on operation
 	- instant sync landed
 	 - weekly train model for client/server for sync
 	 - porting some of the functional load test
 	 - not integrated to production
 	 - support web apps team on the mobile side
 	- David/Henrik : we expect to get more things done
 	 - follow up : that we set goals that can be accomplished
 	 - new team, setup of the new team on wiki
 	 - get more community members
 	 - prototype for the fennec driver
 	 - endurance test
 	 - useful things to get in the test run
 	 - endurance test for memshrink
 	 - Case Conductor: 
 	 	- wrap up: dev project
 	 	- .4/.5 release shipped
 	 	- great progress
 	 	- 2 different items weren't hit
 	 	- making some features as goals is ill advised
 	 	- reprioritize features
 	 	- security issues
 	 	- internal server deployment
 	 	- user research done from rebecca that was great
 		- some community use it, and are getting feedback
 		- Q4 goal = CC 1.0
 - Q3 was very productive
  - we responded, we have good highlights, lots to learn, lots to carry over
  - top level QA goals are what we publish
  - team goals down to personal goals
  - altering the processes, a bit

  - top level QA goals
  - been a little tough because Eng focus has changed
  - what are the proper goals that align with Dev has been tough
  - Bob : direction from all hands/work week that has been completed
  - that gives a good guideline 
  - need to put goals together with what QA thinks is important
  - and then check alignment
  - we should have enough information to say what's important and what's not:
  		https://intranet.mozilla.org/QA/Q32011_QAgoals
  
  - we did publish draft goals: : https://intranet.mozilla.org/QA/Q42011_QAgoals

top level goals
- 1st on Tony's group :
	- mobile test harness for crowd sourcing : 
		- take reftest/crashtest => remotely put that on server, and use an addon and fire off and run the test on the device
		- report failures back to server
		- get a baseline on real failures on real devices
			- who on mobile team will be working on this when we go native
				- mw22 working with jmaher
			- if they change the framework w/native then we're going to have to go a different route
			- should we stop?  10 and on is the earliest
			- we're talking 12 weeks, and new front end: how much effort, and how much benefit, let's stop here.  let's get 8/9 out and then get ready from 10
			- 2 weeks away for addon working; talk offline; giving this goal last week, we didn't change it early enough
			- need to take a look at this for change in direction
	- Marcia : 
		- remote sig : draft link to draft doc (formerly embassador )
			https://docs.google.com/document/d/1DZBKZl0mSXZjwcwFbnRmx1VfSZ8vcK2GMJbi9W-fwKA/edit?hl=en_US&pli=1
		- develop community in different countries.  sig helps grow the community.
		- overall one or one for each team?
		- second draft circulated
	- update testing as a high priority
		- automate what we can do
			- automated
			- functional testing
			- been a phased in approach of what we do for update testing
				- one button initiative
				- controlled by jenkins
		- David Burns :
			- this could change for native fennec
			- will go to toronto for the tooling will work for fennec front end
	- case conductor : 1.0 ; set as basis of what we do for test /test execution

At the team level and personal goals:
- flexibility so that as we work on these goals that we have an element of agility
- top level org goals, team goals, personal goals can have adjustment
- Dave Hunt's goals are impressive : stretch and super stretch goals
- we look to him to complete the main goals,... stretch goals awesome, super stretch walk on water
	- backlog as we go forward if the super stretch aren't hit.
	- mevans likes that.
- Bob's expectation : QA team is head/shoulders ahead.  rest of Eng should follow suit
	- kudos.  level of thoughts are good
	- great opportunity to do a hard reset
	- is what we're doing really working, what can we do more of, what can we do better
	- it's obvious that we aren't big enough to handle this ourselves
	- need to work smarter, scalable, and more community
	- if we don't we're going to "die"
- process review and refinement?
	- doesn't necessarily have to be top level goals
	- could be about business
	- external facing as well
	- need to get things done
	- pushing the rope right now for QA, need to figure out how to do this better
	- let's for real do it now.
	- capture the idea of the hard reset?
	- look at what we're doing, plan our work and then work our plan
		- if we don't do that at some point, our day jobs will overcome us
- looking at each of the teams, and task what does 2012 look like?
	- what is going to give us the time to 
	- this quarter is a pivotal quarter
	- time frame is tight
	- fennec, open web apps, b2g, etc. is something that we're going to have to adjust to
	- given the team and the curtailing of hiring, we're going to have to make some hard choices
	- how do we handle expectations
	- what is the proper course of action : smarter, focus
	- most focused
	- prioritization on how much we can make an impact
- meeting next week : bring in ideas
	- mevans to pool together ideas
	- this quarter and future
	- there is a fair amount of planning that has been done : qa automation, infrastructure
		- need to have up on the board
		- complete reset ideas
	- dare to be great
		- find a way to find bugs and make the fixes happen
		- move forward and throw stuff on the table
		- the energy from the meeting might be worth the effort; mail could work, but meeting might be better
		- train coming/light of the end of the tunnel
	- chaotic : rapid release, we're doing it
		- define the scope of the problem, try to come up with solutiosn to the problem
		- solutions beyond that
		- understanding what the problem is
	- initiaves that the company doing is rapidly outstripping what we're doing
		- services
		- platform
			- desktop
			- fennec
	- more chaos coming our way
		- need to adopt testing in bigger chunks?
		- down to the hard choices, where do we focus
		- how do we make the services that we provide the most effective?
		- use that evaulation currently
		- given the context on what's being released, how do we handle it
	- what's on our plate currently
	- how big of a plate can we have
	- how ... etc.
- dilemma :
	- try to examine the old way of doing things is hard
	- mozmill, mochitest runnning on desktop
	- should we be looking at that?
	- where we want to go?
	- trying to look at the old way
	- new things are developing are based on the old ways of doing things
	- hit the reset, and look where we need to go
	- understanding the historical thing
	- mobile may be very right for a reset
	- not much on automation there comparitive
	- from traditional FF approach, whole new tools might need to be evaluated
	- time/effort/money already put in, ... finding a happy medium
	- how do we approach current things, what's coming down the pike, and does it map
	- this is going to be our new approach
	- current approach
		- how to evaluate it
	- what is the sensible approach
	- what is happening coming down the pike
- moving forward : 
	- advancement in what mozilla does, and is investing is ... is changing all the time
	- evaluation and moving forward
- where do we want to be excellent
	- services, browser id, app sync, sync service
	- OWA, fennec, desktop
	- where do we want to have excellent test framework and all around
	- we have to start somewhere.
	- where do we want to start and where do we want to put our efforts.
- align with company goals
	- we should be listening to the company goals
- paste link into the channel : http://www.t3.org/tangledwebs/07/tw0706.html
Expectations of what this team can be

Scrumpad: keep as is : looking for information that you want to relay to other team members ( not daily status ) 
- moving to new server
- native fennec : huge priority
- meeting with google and win 8 restrictions
- problems with team slides 
	- useful for mozcamp asia
	- take away : easiest way to get someone involved on your team
	- provide template please?  : marcia action item
	- events are coming up; can be repurposed, tweaked, modified
	- slides from vancouver?  some might, but need crisper slides
	
- post mortem for sync server meltdown
- community invites have gone out, no update for staff for mozcamps; hoping that comes through
- proposal : henrik, marcia for festival, but nothing from the mozcamps
	- request : mary colvig > bob to chase that down
- automation : 

Action Items:
- mevans to pool together ideas for next meeting about this quarter and future
- marcia to create a template for team slides
- Bob to chase down mozcamp (mary colvig)
- Please sign up for test day on friday  BrowserID