Snappy Symbolication Server

The Snappy Symbolication Server is a Web server for symbolicating Firefox stacks. It matches PC addresses to modules in memory and looks up the corresponding function names in server-side symbol files (.SYM files).

Running

The source code for Snappy lives in the Mozilla Github repository. Snappy runs at Python 2.7 and depends on Tornado and concurrent.futures packages. The best way to install them is through pip:

pip install tornado futures

To run snappy on your machine, just type:

python symbolicationWebService.py <configuration-file>

The snappy repository contains two sample configuration files for Linux and Windows. Here is a summary for the config fields:

Section	Fields
	Field	Description
General	hostname	Server address or name
	portNumber	Server port number
	remoteSymbolServer	The address of a secondary remote server to forward requests to
	maxCacheEntries	Number of entries for RAM symbol cache
	mruSymbolStateFile	Json file with RAM cache info
	maxMRUSymbolPersist	Maximum number of symbols for cache
DiskCache	cachePath	Path to cached symbol files
DiskCache	maxCacheFiles	Maximum number of files in the cache dir
Log	MaxFiles	Maximum number of log files
	maxFileSize	Maximum size for a log file
	logPath	Path to log files
	logLevel	NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL
SymbolPaths		Each entry represents a path to search for symbols in the local disk
SymbolURLs		Each entry represents a remote path to search for symbols

Contributing

It is a Github project, just fork it and send a PR. If you want to ask something, you can find people involved with Snappy Server in the #perf channel at irc.mozilla.org.

Project ideas

Snappy parses symbol files in another process (which we informally call "Symbolication Process") due to the well known GIL problems. We could have a Symbolication Process per CPU core, but we have a tricky problem here. The Symbolication Process maintains a in memory symbols cache, with the most recently used symbols. The problem is how we handle this, we could have a single shared cache among all Symbolication Processes, which would bring contention, hurting the code parallelism. Other approach is that each Symbolication Process could have its own memory cache, but we potentially could waste memory due to duplicated symbols among all processes, and we could duplicate work because all subprocesses would parse the same symbol file in case of several similar symbolication requests. One good solution is to maintain the memory cache in the parent process.

The Symbolication Process requests symbol files from S3. This is a I/O bound task, so this should happen in the main process, and then Snappy would use asynchronous I/O for that. The problem is that we have to send the symbol file to the Symbolication Process through IPC. The IPC overhead could kill the performance gain with asynchronous requests. We need performance numbers here to make a decision on what's the best approach.

Too bad we don't have unit tests.

Regressions tests

Some regressions tests to perform before sending a PR:

Perform a get request to the server
Check if the IP in X-Forward-For is logged
Test requests with /gecko-profiler/ path
Test for /debug and /nodebug special paths
Test with compressed symbols files
Exiting with Control-C should kill the server smoothly
Check if the host handles ill-formed requests
Check if symbol server can forward requests
Parse a symbol file, exit and start the server again. The server should fetch the sym file from cache
Check if the server writes logs to files
Test requests forward