CIDuty/Dealing With Outages

Communication

Communication is the most important thing during an outage.

• Talk to sheriffs in IRC to let them know what you're working on and what the status is.
• Mirror that status into the relevant bugs as appropriate.
• Coordinate with others who are helping to avoid duplicating work.
  • If debugging collaboratively with others, start a public etherpad to share state.

Problems with stage/ftp

Ganglia is useful for monitoring Mozilla-hosted services:

• https://ganglia-scl3.mozilla.org/ganglia/

For problems with stage/ftp, the productdelivery cluster is informative:

• https://ganglia-scl3.mozilla.org/ganglia/?c=productdelivery-ftp&m=load_one&r=hour&s=by%20name&hc=4&mc=2

Problems with networking

If you're experiencing network issues, particularly being colos (e.g. SCL3<->EC2), join the #netops-alerts IRC channel. Connection details are in https://mana.mozilla.org/wiki/display/SYSADMIN/IRC+use+within+IT

Smokeping can help you visualize outages in real-time:

• http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?target=Datacenters.RELENG-SCL3.core1-releng-scl3
• http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?target=Datacenters.RELENG-SCL3.nagios1-releng-use1
• http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?target=Datacenters.RELENG-SCL3.nagios1-releng-usw1
• http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?target=Datacenters.RELENG-SCL3.nagios1-releng-usw2

Investigate network packet loss between Mozilla and Amazon EC2 using mtr.