Marketplace/PushDuty

From MozillaWiki
< Marketplace
Revision as of 18:39, 13 February 2015 by Charmston (talk | contribs) (Adds marketplace-operator-dashboard)
Jump to navigation Jump to search

Marketplace Push Duty

So, you're going to update marketplace.firefox.com, eh? You've come to the right place. As release manager your responsibilities include:

  • Tagging releases when the milestone closes and updating stage
  • Evaluating and cherry picking requests for the tag after it closes
  • Ensuring the waffle flags on stage are set appropriately (if it's going out in the next push, it's enabled, otherwise it is equivalent to production)
  • Working with ops during our push window to make sure the release is smooth
  • Working with QA during our push window to make sure any concerns are addressed
  • Follow up with ops and QA to do repeat pushes to address any critical issues.
  • Noting any new major features going out on the etherpad
  • Telling the person after you that they are on for the next week

Calendar

If you'd like some calendar events to remind you when to do things, I've set one up: HTML or iCal.

You can subscribe to this for the week you are on push duty, then turn off when you are off it.

A walkthrough of a push

Tagging

You should be tagging Friday at 11am PST before you expect to push. It doesn't hurt to chat with folks ahead of time on IRC to remind people you're tagging and making sure they don't have half-finished patches.

Automatic tagging and staging

In the zamboni repository, you can run: 

   make release

Before you run that command make sure you are connected to the Mozilla VPN so that you can access the dreadnot API.

If this is your first time running it, it will prompt you for dreadnot URL (http://dreadnot-stage.addonsadm.private.phx1.mozilla.com/) and region (phx1), as well as username and password. If you have the "keyring" python module installed, it will securely remember your password. Remember to be connected to the VPN to run this.

The script will then tag all the appropriate repos, and push them to stage. If it encounters an error, you can re-run make release and it will probably pick up where you left off.

IMPORTANT: If any of the stage pushes fail make sure the code gets re-pushed successfully.

Manual instructions

Name your tag with the date of the push in the format YYYY.MM.DD.

The following repositories need tagging:

There is a script which can do all that for you. Try: 

 python tagz.py -r mozilla/solitude,mozilla/webpay,mozilla/commbadge,mozilla/fireplace,mozilla/marketplace-stats,mozilla/monolith-aggregator,mozilla/transonic,mozilla/zamboni,mozilla/spartacus,mozilla/marketplace-operator-fashboard -c create -t YYYY.MM.DD

Next add in the github compare URLs into the etherpad, so when the push comes people can easily see what is about to go out.

Next you'll need to update the staging servers:

  1. Load dreadnot (restricted, you'll need VPN+permissions to get here)
  2. Push items in the following order (push by clicking "phx1" and then typing in the name of the tag or hash and hitting deploy):
    1. payments.allizom.org-solitude
    2. payments-proxy.allizom.org-solitude
    3. marketplace.allizom.org-spartacus
    4. marketplace.allizom.org-webpay
    5. monolith.allizom.org-aggregator
    6. marketplace.allizom.org-marketplace-stats
    7. marketplace.allizom.org-commbadge
    8. marketplace.allizom.org-transonic
    9. marketplace.allizom.org-marketplace-operator-dashboard
    10. marketplace.allizom.org-fireplace
    11. marketplace.allizom.org-zamboni (would be good to wait for everything else to finish deploying before hitting this one)

There is a script which can also do all this for you. You'll need to set up a config file (mine is dreadnot.settings which will be used in the commands that follow). 

# I have this saved as dreadnot.settings
[dev]
username = <your dreadnot user name>
password = <your dreadnot password>
dreadnot = https://dreadnot-stage.addonsadm.private.phx1.mozilla.com
region = phx1

Once you have the settings file configured, you can run the deploy commands, monitoring the deploys on the dreadnot landing page. 

python dreadnot.deploy payments.allizom.org-solitude -r [TAG] --conf=dreadnot.settings -e dev 
python dreadnot.deploy payments-proxy.allizom.org-solitude  -r [TAG] --conf=dreadnot.settings -e dev
python dreadnot.deploy marketplace.allizom.org-spartacus -r [TAG] --conf=dreadnot.settings -e dev 
python dreadnot.deploy marketplace.allizom.org-webpay -r [TAG] --conf=dreadnot.settings -e dev 

python dreadnot.deploy marketplace.allizom.org-marketplace-stats  -r [TAG] --conf=dreadnot.settings -e dev 
python dreadnot.deploy marketplace.allizom.org-commbadge -r [TAG] --conf=dreadnot.settings -e dev 
python dreadnot.deploy marketplace.allizom.org-fireplace -r [TAG] --conf=dreadnot.settings -e dev 

python dreadnot.deploy marketplace.allizom.org-transonic -r [TAG] --conf=dreadnot.settings -e dev 
python dreadnot.deploy marketplace.allizom.org-marketplace-operator-dashboard -r [TAG] --conf=dreadnot.settings -e dev 

# wait for the rest to finish deploying, then: 
python dreadnot.deploy marketplace.allizom.org-zamboni -r [TAG] --conf=dreadnot.settings -e dev

Presumably we can script this too, but for now it can be useful to run each deploy separately to catch problems before they cascade.

Pushing

Pushes happen Tuesday at 11am. There is an etherpad made each week named mkt-YYYY-MM-DD. An example. The push will mostly follow this etherpad and any special notes should be in that pad.

You might want to add in meeting for yourself in Zimbra for the push time so that people won't try and schedule you for meetings.

To push:

  1. The release manager (you), QA (krupa), and Ops (Jason) should be in contact on IRC and in the Marketplace vidyo room
  2. Once everyone gives the thumbs up Ops will push the actual code using dreadnot. Ops will push the projects in order (same order you did for stage). Talk on vidyo if there are any questions.
  3. The IRC bots will say when the pushes are done
  4. Once the push is done, QA will verify any changes. Work with them to flip any waffle switches or tweak any adjustments
  5. Whilst QA is reviewing....
    1. Review sentry errors, there will often be errors during the push, but after there should be nothing to worry about
    2. Review the graphite graphs on the dashboard to see if anything looks amiss
    3. You can look at nagios as well if you like on the dashboard, but Ops will do that.
  6. If QA or Ops finds something that needs fixing immediately:
    1. Either write a patch or find someone who can
    2. Cherry-pick the patch onto the previous tag. (Example)
    3. Go back to step 2 until QA is (relatively ;) happy
  7. Once QA, Ops, and you all sign off the push is over. Record the time it took in the bottom of the etherpad

Post Push

  1. Create a new etherpad for the next week using the push template
  2. Edit the topic in the secret channel pointing to the new etherpad.
  3. Remind next week's release manager they are on the hook :)
  4. Send an email to the public mailing list (dev-marketplace@lists.mozilla.org) saying how the push went. If there was reason for multiple pushes, or anything that could be improved or fixed (eg: dodgy migration), let the team know. simple template

Known issues

There may be an issue with the build id, as described in bug 1083185. To check this once push has completed, load https://marketplace.firefox.com/, and compare the md5 of the include.js and a random value: 

expr $(curl https://marketplace.cdn.mozilla.net/media/fireplace/js/include.js\?b\=1416347049712 2>/dev/null | md5) = 
$(curl https://marketplace.cdn.mozilla.net/media/fireplace/js/include.js\?b\=123123123123123123 2>/dev/null | md5)

Release manager rotation

  • ashort
  • kumar
  • spasovski
  • chuck
  • ngoke
  • mstriemer
  • robhudson
  • ddurst
  • jared
  • andym

(there will be exceptions. No problem, we just need to be aware of them and plan for them)

Retrieved from "https://wiki.allizom.org/index.php?title=Marketplace/PushDuty&oldid=1056140"