Confirmed users
1,233
edits
Breakpad/Status Meetings/2016-01-06: Difference between revisions
Jump to navigation
Jump to search
Breakpad/Status Meetings/2016-01-06 (view source)
Revision as of 19:20, 6 January 2016
, 6 January 2016→Operations Updates
| Line 22: | Line 22: | ||
** problem 1: disks filled up due to no server recycling | ** problem 1: disks filled up due to no server recycling | ||
** problem 2: processed crashes going down. 80% difference, verified. Rabbit is silently losing 80%. 43 unable to send as many crashes (proxy ssl problems). Abandon our RabbitMQ. We're using cloudamqpapp.com. Lars unable to fully test this on staging. Once tested on staging, lars will push that prod. If continues to lose things, backup plan; Upgrading pika libs. We have preserved logs for failed UUIDs for failed ones from 1st Jan. Enumerating UUIDs on S3 takes very long time, thus a problem if we tried to reprocess all of December. "A lot of data pain the butt". | ** problem 2: processed crashes going down. 80% difference, verified. Rabbit is silently losing 80%. 43 unable to send as many crashes (proxy ssl problems). Abandon our RabbitMQ. We're using cloudamqpapp.com. Lars unable to fully test this on staging. Once tested on staging, lars will push that prod. If continues to lose things, backup plan; Upgrading pika libs. We have preserved logs for failed UUIDs for failed ones from 1st Jan. Enumerating UUIDs on S3 takes very long time, thus a problem if we tried to reprocess all of December. "A lot of data pain the butt". | ||
** problem 3: | ** problem 3: 28th Dec all processors died. ujson couldn't handle bad data. Breakpad bug? ujson bug? (we're on usjon 1.33, there is a 1.34 available). Lars has a patch that drops ujson for regular json. | ||
** (jp) How do we do monitoring of this going forward? | ** (jp) How do we do monitoring of this going forward? | ||
*** reaching out to experts for help. | *** reaching out to experts for help. | ||