Auto-tools/Projects/Structured Logging

This page is a high level account of the structured logging project providing links to further information about the project.

Background and Motivation

Goals:

• Support a range of output formats for test results
• Eliminate need for complex and brittle regex based log parsing
• Consolidate output processing and logging configuration

Approach:

• Establish a common data format for test results and test harness diagnostics
• Populate data format in-harness with a logging API designed to output this format
• Log data from test harnesses as JSON rather than formatted strings

Implementation

Participant Systems

• Producer test framework (Firefox/Fennec/B2G - JS/Java)
• Test harness (Python/JS)
• Mozharness/Buildbot
• Tbpl/Treeherder
• Blobber

Logging APIs

The primary logging API for our data format is mozlog's structured module (readthedocs). An extensive example of how to use the api is included in these docs.

Mozlog is a module in Mozbase. More information about contributing to Mozbase can be found on its wiki page: https://wiki.mozilla.org/Auto-tools/Projects/MozBase#Working_on_Mozbase_and_Contributing_Patches

Mozlog bugs:

Full Query

ID	Summary	Priority	Status
944177	[mozlog] mozlog's getLogger throws exception in what seems like a normal use case	--	REOPENED
1030228	[mozlog] Result.html doesn't contain the 'message' string for web-platform-test failures	--	NEW
1057210	[mozlog] Mozlog should have a thorough set of tests establishing desired properties for its formatters	--	NEW
1066161	[mozlog] Mozlog should be able to interpret test expectations from test manifests	--	NEW
1070669	[mozlog] Make test_end track the status of its subtests	--	NEW
1073140	[mozlog] commandline.py overwrites user specified 'mach' formatter if no sys.stdout loggers found	--	NEW
1119407	[mozlog] Allow setting a default stdout logger through an environment variable or other persistent config mechanism	--	NEW
1262147	[mozlog] Create rich html log format designed with live streaming in mind	--	NEW
1262412	[mozlog] Make output for disabled tests more obvious	--	NEW
1287026	[mozlog] Support pop-up reftest analyzer in html report	P3	NEW
1294652	[mozlog] HTMLFormmater does not handle for subtests(test_status) log	P3	NEW
1372565	[mozlog] When using log_raw, we don't ensure state for test_start, test_end or test_status actions	P3	NEW
1378174	[mozlog] Create a "sugar" formatter based on pytest-sugar	P3	NEW
1428363	[mozlog] Reduce number of unused dependencies for log-parsing-only consumers of mozlog	P3	NEW
1442645	[mozlog] mozlog.structuredlog: Failure calling log handler, due to KeyError: 'subtest'	P3	NEW
1446965	[mozlog] Buffer all test logs in 'mach' formatter and flush on failure	P3	NEW
1448532	[mozlog] Mach log formatter can cause log output to be really slow	P3	NEW
1472258	[mozlog] It would be nice if mozlog provided a decorator that mach commands could use to add mozlog options.	P3	NEW
1515327	[mozlog] register_message_handlers undocumented	P3	NEW
1515330	[mozlog] Unclear how to write a new mozlog handler	P3	NEW
1719677	[mozlog] Permanent Windows Marionette UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f36a' in position 194: character maps to <undefined> - DO NOT USE FOR CLASSIFICATION	P5	NEW

21 Total; 21 Open (100%); 0 Resolved (0%); 0 Verified (0%);

For data originating from JavaScript or Java, StructuredLog.jsm and StructuredLogger.java have been implemented to produce this format. Mozlog supports logging data directly from these producers.

Status

• The tracking bug for this effort is bug916295.

Structured Logging FAQ

What does this mean for you? It’s been a goal of the implementation effort to provide a foundation for future tools and infrastructure improvements without disrupting familiar workflows or breaking compatibility with existing tools, so the changes will be largely unobtrusive at this stage. However, there are two things to keep in mind:

• Support for failing a build/test job by logging “TEST-UNEXPECTED-“ directly will be diminished as we move away from regex parsing in our tools. Please do not add instances of this when something else is available.
• As we roll out this out to the various harnesses you may notice different output formats than you expected. Changes to formatting shouldn’t make logs any less useful — if they have, this is probably an unintended effect of the implementation and you should file a bug (Testing :: Mozbase for mozlog features/bugs). To get output closer to what you used to see, you should be able to pass “—log-tbpl -“ to mach to specify a formatter implementing the legacy log format.