ReleaseEngineering/How To/Restart Redis

From MozillaWiki
< ReleaseEngineering‎ | How To
Revision as of 22:34, 18 October 2013 by Hwine (talk | contribs) (cronjobs to automate fix)
Jump to navigation Jump to search


Redis is used to cache information about builds, as well as support the signing servers. The general way of restarting Redis is

service redis restart

as root@redis01.build.mozilla.org.

Note that we write the in-memory database to disk frequently so there is a low risk of data loss in doing this.

Redis hangs

bug 905587 tracks an issue with the version we're running where it stops functioning after opening too many file descriptors. This results in builds-4hr.js.gz getting stale, and TBPL stops getting information about builds that are recently finished. This is a tree closing problem.

Note: as of 2013-10-18 there are cronjobs on redis01 that attempt to automatically do the fix below. See bug 905587 for details.

Fix

  • Connect:
ssh root@redis01.build.mozilla.org
  • Diagnosis - look for a result >= 1000
ls /proc/`pidof redis-server`/fd | wc -l
  • Resolution
service redis restart
  • Verification:
telnet localhost 6379
# say 'MONITOR'

You should see a lot of lines fly past once builds-4hrs generation gets going again. If it doesn't root@buildapi01.build.mozilla.org and look at buildapi processes