Breakpad/Status Meetings/2017-05-17

« previous meeting — index – next week » ^create?

Meeting Info

Breakpad status meetings occur on Wed at 10:00am Pacific Time.

Conference numbers:

   Vidyo: Stability 
   650-903-0800 x92 conf 98200#
   800-707-2533 (pin 369) conf 98200# 

IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)

Operations Updates

• Miles wrote up a new migration document with different details than in past versions. https://docs.google.com/document/d/18xKCDeXQDpoPtVpxJ9BeisAMrxLXyKhFUIfJNWklaYM/edit
  • we're going to "tee" off the infrastructure at Antenna and then build two parallel Socorro universes which should let us figure out all the stuff and then cut over smoothly
• We saw a weird uptick in processor/elasticsearch save_raw_and_processed times and the symbols api hits.
  • RabbitMQ queues were backing up, so Miles threw more processors at the queue.
  • It's weird because it doesn't look like we're processing more crashes.
  • Adrian fixed some issues in the processor and maybe that'll make things better. We'll find out after -prod deploy and investigate further if need be.
• System test for Tecken is working again.
  • There are some issues with https urls, but Peter and Miles will take that offline.
• We have a crontabber thing that runs in SCL3 that needs to send data to postgres for ADI.
  • How to do this in the new upcoming infrastructure?
  • Switch it to push via HTTP to the webapp rather than send stuff to Postgres.
  • Also, that box is "interesting", so it's probably hard to update/deploy/etc.
  • We might be able to get rid of this altogether.
  • Peter will file a bug about this server.
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1365665
• Miles asked whether we can ditch crontabber on admin nodes. Probably not, yet.
• Official ES migration plan involves snapshots and that's the way it's going to go.

Security Updates

• Greg completed code review (security) for Antenna and pyup is now running against Antenna.
• Greg will talk with Peter re: Tecken risk assessment.
• risk assessment request: https://bugzilla.mozilla.org/show_bug.cgi?id=1365673

Project Updates

Deployment Triage

• Socorro: https://whatsdeployed.io/s-7M7
• Antenna: https://whatsdeployed.io/s-Up2

PR Triage

• http://prs.mozilla.io/mozilla/socorro

Major Projects

Deprecation rampage

• removed the lines
• removed obsolete cron job code

Processor rewrite

• (willkg): As part of dockerizing Socorro, I got stackwalker built and running in a python:3.6.1-slim container. That's related, but otherwise, no new progress.
• We talked about parallelizing code changes to the old processor (e.g. KeyErrors in transform rules)
  • willkg states...
    • don't worry about making changes to the old code
    • don't worry about formalizing annotation about changes to the code

Symbol Server 2017

• Celery is working in dev
• Needs to decide a date to go to prod and take over the domain symbols.mozilla.org
  • Requirements to go live
    • OK from Ted that we take over symbols.mozilla.org
    • Start a security review process
    • Start a load testing once it's got a stage or prod environment

Other Business

• Stickers? Will is working on stuff and will talk about it next week.
• We should add a QA section to the meeting agenda.

Travel, etc

• peterbe out May 18-19, May 25-26