ReleaseEngineering/How To/Restart Redis

Redis is used to cache information about builds. The general way of restarting Redis is

service redis restart

as root@redis01.build.mozilla.org.

Note that we write the in-memory database to disk frequently so there is a low risk of data loss in doing this.

Redis hangs

bug 905587 tracks an issue with the version we're running where it stops functioning after opening too many file descriptors. This results in builds-4hr.js.gz getting stale, and TBPL stops getting information about builds that are recently finished. This is a tree closing problem.

NOTE: We disabled the cronjobs since the unattended restart could leave things in a bad state. Buildduty manually restarts the service and rail added some monit monitoring.
Note: as of 2013-10-18 there are cronjobs on redis01 that attempt to automatically do the fix below. See bug 905587 for details.

Fix

• Connect:

ssh root@redis01.build.mozilla.org

• Diagnosis - look for a result >= 1000

ls /proc/`pidof redis-server`/fd | wc -l

• Resolution

service redis restart

• Verification:

telnet localhost 6379
# say 'MONITOR'

You should see a lot of lines fly past once builds-4hrs generation gets going again. If it doesn't root@buildapi01.build.mozilla.org and look at buildapi processes