MozillaWiki

Oxidation: Difference between revisions

(→‎Within Firefox: Linux audio remoting shipped in 60.)
No edit summary
 
(100 intermediate revisions by 23 users not shown)
Line 1: Line 1:
'''Oxidation''' is a project to integrate [https://www.rust-lang.org/ Rust] code in and around Firefox.  
'''Oxidation''' is a project to integrate [https://www.rust-lang.org/ Rust] code in and around Firefox.  


Rust support has been required on all platforms since Firefox 54, and the first major Rust components were shipped in Firefox 56 (encoding_rs) and 57 (Stylo). Moving forward, the goal of Oxidation is to make it easier and more pleasant to use Rust in Firefox, and correspondingly to increase the amount of Rust code in Firefox.
Rust support has been required on all platforms since Firefox 54, and the first major Rust components were shipped in Firefox 56 (encoding_rs) and 57 (Stylo). Moving forward, the goal of Oxidation is to make it easier and more productive to use Rust in Firefox, and correspondingly to increase the amount of Rust code in Firefox.


This page is intended to serve as the starting point for all matters relating to Rust code in Firefox: the what, the why, and the how.
This page is intended to serve as the starting point for all matters relating to Rust code in Firefox: the what, the why, and the how.
Line 18: Line 18:


Rust has the following strengths.
Rust has the following strengths.
* Memory safety, which prevents crashes and security vulnerabilities.
* Memory-safety and thread-safety. Crashes and security vulnerabilities are much less likely. (Roughly 70% of critical security vulnerabilities are due to memory safety bugs.)
* Thread safety, which enables improved performance via parallelism.
* High performance. In particular, the safety enables code that is designed more actively for performance, especially parallel performance.
* Nimbleness: the safety makes it easy to make significant changes quickly and with confidence.
* Nimbleness. The safety enables significant changes to existing code to be made quickly and with confidence.
* Pleasant to use, particularly once a moderate level of experience has been reached.
* Expressiveness. It is powerful and pleasant to use, particularly once a moderate level of experience has been reached.
* A great community.
* Excellent package management and an extensive ecosystem.
* Excellent compiler error messages.
* Excellent documentation.
* A friendly and helpful community.


== Rust Weaknesses ==
== Rust Weaknesses ==
Line 28: Line 31:
One major issue with Rust relates to personnel.
One major issue with Rust relates to personnel.
* There is a wide variety of experience levels within Mozilla, for both coding and reviewing.
* There is a wide variety of experience levels within Mozilla, for both coding and reviewing.
* Rust's learning curve is steep at the start, which can be intimidating.
* Rust's learning curve is steep at the start, which can be intimidating. (The new borrow checker released in Rust 1.31 helped greatly with this.)


There are also technical challenges.
There are also technical challenges.
* Compilation is slow.
* Compilation is slow.
* Crossing the C++/Rust boundary can be difficult
* [https://gist.github.com/zbraniecki/b251714d77ffebbc73c03447f2b2c69f Crossing the C++/Rust boundary can be difficult].


See "Blockers and obstacles" below for more details about work being done to remedy these weaknesses.
See "Blockers and obstacles" below for more details about work being done to remedy these weaknesses.
Line 49: Line 52:
* For components where Servo has demonstrated success.
* For components where Servo has demonstrated success.


In terms of where to keep Rust crates, there are three options, from most-coupled to least-coupled.
In terms of where to keep Rust crates, there are three options.
* Put the crate in mozilla-central or in Servo's repository.
* Put the crate in mozilla-central or in Servo's repository.
** For binding code, the decision to put it into Gecko or Servo can be difficult. The best choice depend on the details of the binding code in question.
** For binding code, the decision to put it into Gecko or Servo can be difficult. The best choice depends on the details of the binding code in question.
* Put the crate on [https://crates.io crates.io] and use Cargo to access it at build-time.
** This is only suitable for highly general-purpose crates, such as <tt>smallvec</tt>.
* Put the crate somewhere else (e.g. a separate GitHub repository), and regularly vendor it into mozilla-central.
* Put the crate somewhere else (e.g. a separate GitHub repository), and regularly vendor it into mozilla-central.
** This works well if relatively few people are working on the crate, and the API isn't too complex.
** This makes sense for pre-existing third-party crates that we choose to import.
** Can be useful if useful development can be done using just that crate, without the rest of Gecko or Servo.
** Otherwise, this option is not recommended, because vendoring is something of a hassle.
** This also makes sense for pre-existing third-party crates that we choose to import.
** This is done by WebRender, for example.
* Put the crate somewhere else (e.g. [https://crates.io crates.io]), and use Cargo to access it at build-time.
** This is only suitable for highly general-purpose crates.


In general, erring on the side of tighter coupling is advisable. For example, the <tt>heapsize</tt> crate used in memory reporting was moved to crates.io, and then other crates came to depend on it. Later on it needed major API changes, and we ended up replacing it with a new crate called <tt>malloc_size_of</tt> (stored in Servo's repository) because that was easier than modifying <tt>heapsize</tt>.
In general, erring on the side of tighter coupling is advisable. For example, the <tt>heapsize</tt> crate used in memory reporting was moved to crates.io, and then other crates came to depend on it. Later on it needed major API changes, and we ended up replacing it with a new crate called <tt>malloc_size_of</tt> (stored in Servo's repository) because that was easier than modifying <tt>heapsize</tt>.
Line 66: Line 67:
== Rust in general ==
== Rust in general ==


* The [https://www.rust-lang.org/en-US/documentation.html Rust Documentation] page is the best place to start. In particular, the [https://doc.rust-lang.org/book/ The Rust Programming Language] provides a good overview.
* The [https://www.rust-lang.org/learn Rust Documentation] page is the best place to start. In particular, the [https://doc.rust-lang.org/book/ The Rust Programming Language] provides a good overview.
* [https://www.rust-lang.org/en-US/community.html The Rust Community] page lists IRC channels, forums, and other places where Rust assistance can be obtained.
* [https://www.rust-lang.org/en-US/community.html The Rust Community] page lists IRC channels, forums, and other places where Rust assistance can be obtained.
* [http://shop.oreilly.com/product/0636920040385.do Programming Rust: Fast, Safe Systems Development], by Jim Blandy & Jason Orendorff, is a detailed guide to the language.
* [http://shop.oreilly.com/product/0636920040385.do Programming Rust: Fast, Safe Systems Development], by Jim Blandy & Jason Orendorff, is a detailed guide to the language.
Line 73: Line 74:
== Rust in Firefox ==
== Rust in Firefox ==


* [https://developer.mozilla.org/en-US/Firefox/Building_Firefox_with_Rust_code Developer Documentation]
Policy
* [https://firefox-source-docs.mozilla.org/build/buildsystem/rust.html Build System Documentation]
* [[Rust_Update_Policy_for_Firefox|Rust Update Policy for Firefox]]
* [https://public.etherpad-mozilla.org/p/stylo Stylo workflow] (applies to any changes that straddle Gecko and Servo)
 
* The #servo IRC channel contains lots of people who know about both Rust and Gecko.
Rust in Firefox docs
* [https://firefox-source-docs.mozilla.org/build/buildsystem/rust.html Adding Rust code]
* [https://firefox-source-docs.mozilla.org/writing-rust-code/index.html Writing Rust code]
* [https://firefox-source-docs.mozilla.org/testing-rust-code/index.html Testing & debugging Rust code]
 
Getting extra help
* The Rust channel on [https://wiki.mozilla.org/Matrix Mozilla's Matrix network] contains lots of people who know about both Rust and Gecko.
* Are you new to Rust and not sure if your Rust code could be improved? The following people can review Rust patches for Firefox from an "is this good Rust code?" point of view.
* Are you new to Rust and not sure if your Rust code could be improved? The following people can review Rust patches for Firefox from an "is this good Rust code?" point of view.
** Alexis Beingessner (:gankro)
** Alexis Beingessner (:Gankra)
** Josh Bowman-Matthews (:jdm)
** Josh Bowman-Matthews (:jdm)
** Emilio Cobos Alvarez (:emilio)
** Emilio Cobos Alvarez (:emilio)
** Manish Goregaokar (:manishearth)
** Manish Goregaokar (:manishearth)
** Nika Layzell (:mystor)
** Nika Layzell (:nika)
** Cameron McCormack (:heycam)
** Cameron McCormack (:heycam)


= Supported Rust versions for Firefox builds =
= Rust Components =


Below are [[Rust Update Policy for Firefox|our expectations]] if everything goes smoothly; we may need newer toolchains to address specific issues. Our ultimate policy is just that release Firefox will not require unstable or beta Rust to build.
== Within Firefox ==


* Firefox 56 (released 2017 September 26) requires Rust 1.17.0 or later.
=== Statistics ===
** Official builds will use Rust 1.19.0.
* Firefox 57 (released 2017 November 14) requires Rust 1.20.0.
** Mozilla-central can start using Rust 1.19.0 language features 2017 August 3.
** Mozilla-central can start using Rust 1.20.0 features 2017 September 14.
* Firefox 58 (released 2018 January 16) requires Rust 1.21.0.
** Mozilla-central can use Rust 1.21.0 language features 2017 October 26.
* Firefox 59 (released 2018 March 6) requires Rust 1.22.0.
** Mozilla-central can use Rust 1.22.0 features 2017 December 7.
** Firefox 59 ESR is expected to continue to work with Rust 1.22 throughout its support lifetime.
* Firefox 60 (released 2018 May 1) requires Rust 1.23.0.
** Mozilla-central can use Rust 1.23.0 features 2018 January 18.
* Firefox 61 (released 2018 June 26) requires Rust 1.25.0.
** Mozilla-central can use Rust 1.24.0 features 2018 March 6.
** Mozilla-central can use Rust 1.25.0 features 2018 April 12.
 
= Rust Components =


== Within Firefox ==
* [https://docs.google.com/spreadsheets/d/1flUGg6Ut4bjtyWdyH_9emD9EAN01ljTAVft2S4Dq620/edit#gid=885787479 Lines of compiled Rust code shipped in Firefox (over time)]
* [https://4e6.github.io/firefox-lang-stats/ Lines of Rust code in mozilla-central (current)]


=== Completed ===
=== Shipped ===


* MP4 metadata parser: {{bug|1161350}} (shipped for desktop in Firefox 48)
* MP4 metadata parser: {{bug|1161350}} (shipped for desktop in Firefox 48)
Line 117: Line 109:
** '''Why Rust?''' Code taken from Servo, uses parallel algorithms.
** '''Why Rust?''' Code taken from Servo, uses parallel algorithms.
* U2F HID backend: {{bug|1388843}} (shipped in Firefox 57)
* U2F HID backend: {{bug|1388843}} (shipped in Firefox 57)
* libcubeb Audio backend for Linux (PulseAudio): {{bug|1346665}} (shipped in Firefox 59)
* libcubeb Audio backend for macOS (CoreAudio): {{bug|1530713}} (shipped in Firefox 74)
** '''Why Rust?''' Safer implementation avoids data racing by leveraging lifetime, variable mutability check and better mutex pattern.
* XPIDL binding generator ({{bug|1293362}}) (shipped in Firefox 60)
* XPIDL binding generator ({{bug|1293362}}) (shipped in Firefox 60)
* New prefs parser: {{bug|1423840}} (shipped in Firefox 60)
* New prefs parser: {{bug|1423840}} (shipped in Firefox 60)
** '''Why Rust?''' Old parser needed replacing. Well-separated component, simple interface, parses untrusted input.
** '''Why Rust?''' Old parser needed replacing. Well-separated component, simple interface, parses untrusted input.
* Audio remoting for Linux: {{bug|1434156}}
* Audio remoting for Linux: {{bug|1434156}} (shipped in Firefox 60)
* WebRender: {{bug|webrender}} (shipped in Firefox 67, enabled for users with appropriate hardware)
** '''Why Rust?''' Code taken from Servo, has high performance; Rust's memory and thread safety assist productivity, and allow more aggressive optimizations.
* kvstore (key-value storage backed by LMDB): {{bug|1490496}} (shipped in Firefox 67)
** '''Why Rust?''' The rkv crate provides a safe, ergonomic wrapper around LMDB, our choice for simple key-value storage in Firefox. kvstore wraps rkv in an asynchronous XPCOM API for JS and C++ callers.
* Profiler symbolication: {{bug|1509549}} (shipped in Firefox 67)
** '''Why Rust?''' Makes use of existing crates that handle object file parsing and symbol iteration. Easy to compile to WebAssembly.
* XUL store, backed by rkv: {{bug|1460811}} (landed in Firefox 68, used in Nightly only)
* TLS certificate store, backed by rkv: {{bug|1429796}} (shipped in Firefox 68)
* Synced bookmark merger: {{bug|1482608}} (shipped in Firefox 68, on by default in Nightly and early Beta)
** '''Why Rust?''' Code sharing! The bookmark merging algorithm was factored out into a [https://crates.io/crates/dogear separate crate], and is shared between Desktop and [https://github.com/mozilla/application-services all our mobile products].
* Windows BITS interface: {{bug|1520321}}  (shipped in Firefox 68)
* Japanese encoding detector: {{bug|1543077}} (shipped in Firefox 69)
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer.
* [https://github.com/padenot/audio_thread_priority audio_thread_priority] {{bug|1429847}} (shipped in Firefox 69), allow promoting threads to a real-time scheduling class, on Windows/Linux/macOS.
** '''Why Rust?''' This crate is being used by C++ code and by Rust code (audioipc), Rust is nicer to write than C++ (especially for what is essentially just a series of system calls, the error checking style is nice), and cbindgen made it trivial to expose a C ABI.
* [https://github.com/mozilla/dogear/ Dogear] a bookmark merger for Sync. Shipped pref'd-off in {{bug|1482608}} (Firefox 68), enabled by default in {{bug|1588005}} (shipped in Firefox 72)
** '''Why Rust?''' A single performant and safe implementation shared between desktop and the bookmarks engine in [https://github.com/mozilla/application-services/tree/master/components/places application-services]
* Unicode Language Identifier: {{bug|1571915}} (shipped in Firefox 72)
** '''Why Rust?''' Much faster, parser-heavy, easier to handle low-memory footprint thanks to `tinystr`.
* Language Negotiation: {{bug|1581960}} (shipped in Firefox 72)
** '''Why Rust?''' Ties into `unic-langid`, easier to handle list filtering and ordering.
* Replace libhyphen with mapped_hyph, {{bug|1590167}} (shipped in Firefox 72).
* Encoding detector: {{bug|1551276}} (shipped in Firefox 73)
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer, potentially parallelizable with Rayon.
* Integrate [https://github.com/projectfluent/fluent-rs fluent-rs], a localization system: {{bug|1560038}} (shipped in Firefox 76)
** '''Why Rust?''' Performance and memory wins are substantial over previous JS implementation. It brings zero-copy parsing, and memory savvy resolving of localization strings. It also paves the way for migrating the rest of the Fluent APIs away from JS which is required for Fission.
* qcms [https://searchfox.org/mozilla-central/source/gfx/qcms] ported from C to Rust
** '''Why Rust?''' Memory safety


=== In progress ===
=== In progress ===


* WebRender: {{bug|webrender}}
* Port [https://github.com/projectfluent/fluent-rs Localization API] to Rust: {{bug|1613705}}
** '''Why Rust?''' Code taken from Servo, has high performance; Rust's memory and thread safety provides protection against complexity.
* [https://github.com/mozilla/neqo neqo] A QUIC implentation.
* [https://github.com/CraneStation/cranelift/ cranelift, a low-level retargetable code generator]: {{bug|1469027}}
** '''Why Rust?''' It's a new, well-separated component with a clear interface. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
* [https://github.com/mozilla-spidermonkey/rust-frontend SmooshMonkey] A Rust front-end for SpiderMonkey (featuring [https://github.com/mozilla-spidermonkey/jsparagus jsparagus]).
** '''Why Rust?''' Parses untrusted input. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
* Audio remoting for Windows: {{bug|1432303}}
* Audio remoting for Windows: {{bug|1432303}}
* Audio remoting for Mac OS: {{bug|1425788}}
* Audio remoting for Mac OS: {{bug|1425788}}
* SDP parsing in WebRTC: {{bug|1365792}}
* SDP parsing in WebRTC: {{bug|1365792}}
** '''Why Rust?''' SDP is a complex text protocol and the existing parser in C has a history of security issues. This also allows us to tailor the SDP parser specifically for the subset used in WebRTC, further reducing its surface area. It is currently run in parallel with the C parser in Nightly.
* Linebreaking with xi-unicode: {{bug|1290022}} (last update late 2016)
* Linebreaking with xi-unicode: {{bug|1290022}} (last update late 2016)
* Optimizing WebVM compiler backend: [https://github.com/Cretonne/cretonne/ cretonne]
* Background Update Agent for Windows: {{bug|1343669}}
** '''Why Rust?''' New, well-separated component with a clear interface.
* [https://github.com/gfx-rs/wgpu wgpu], a [https://gpuweb.github.io/gpuweb/ WebGPU] API implementation: {{bug|webgpu-mvp}} (in Nightly since 72)
* Share [https://github.com/servo/gecko-media media playback stack] with Servo.
** '''Why Rust?''' Complex tracking logic, wide attack area. Also, leverages Rust ecosystem for building libraries on top of our native implementation and the API that will target the Web.
** '''Why Rust?''' We're not rewriting this in Rust because the current code is well-tested and performant. The plan is to first port the C++ module from Firefox into Servo.
* Integrate the [https://github.com/mozilla/glean/ Glean SDK], a data collection library.
** Once the module is shared, work can begin on smoothing the interface and migrating piecemeal.


=== Proposed ===
=== Proposed ===


* Parallel layout
** '''Why Rust?''' Existing code from Servo, parallel performance.
* Replace the XML parser
** '''Why Rust?''' Parses untrusted input, replaces expat, a 3rd-party library with a history of frequent security vulnerabilities.
* WebMIDI: {{bug|1201593}}, {{bug|1201596}}, {{bug|1201598}}
* WebMIDI: {{bug|1201593}}, {{bug|1201596}}, {{bug|1201598}}
* Gamepad code: {{bug|1286699}}
* Gamepad code: {{bug|1286699}}
Line 148: Line 171:
* Replace DOM serializers (XML, HTML for Save As.., plain text)
* Replace DOM serializers (XML, HTML for Save As.., plain text)
** '''Why Rust?''' Need a rewrite anyway. Minor history of security vulnerabilities.
** '''Why Rust?''' Need a rewrite anyway. Minor history of security vulnerabilities.
* Image decoders?
** '''Why Rust?''' Parsing untrusted input, some history of security vulnerabilities.
* Expose Rust API to JS Debugger: {{bug|1263317}}
* Expose Rust API to JS Debugger: {{bug|1263317}}
* Generate Rust bindings for IPDL actors ({{bug|1379739}})
* Generate Rust bindings for IPDL actors ({{bug|1379739}})
* URL parser: [https://bugzilla.mozilla.org/show_bug.cgi?id=1318426#c13 WONTFIX]
* WebM demuxer: {{bug|1267492}}
* WebM demuxer: {{bug|1267492}}
* Parallel JS parsing: fast preparse to find function boundaries, parse non-overlapping functions in parallel with a unification step to handle free names and such (no bug on file yet)
** '''Why Rust?''' Parses untrusted input. Requires safe threading. And generally, Rust is a better language than C++ for parsers, due to strong typing, algebraic data types, and pattern matching.
* Crash reporter
** '''Why Rust?''' Code needs rewriting, useful Rust crates exist that could be used.
* [https://github.com/mozilla/application-services Sync/FxA components]
** '''Why Rust?''' Single safe and performant implementation which is shared across all our products.
* libcubeb Audio backend for Windows (WASAPI)
* Replace the XML parser, possibly via c2rust: {{bug|1611289}}
** '''Why Rust?''' Parses untrusted input, replaces expat, a 3rd-party library with a history of frequent security vulnerabilities.
* Rewrite the ICE stack used by WebRTC; {{bug|1616966}}
** '''Why Rust?''' Works with network data.


== Outside Firefox ==
== Outside Firefox ==
Line 159: Line 190:
=== Completed ===
=== Completed ===


* GeckoDriver, a WebDriver implementation for Firefox integrated via marionette protocol: {{bug|1340637}} ([https://github.com/mozilla/geckodriver/releases standalone releases])
* Testing
* [https://github.com/mozilla/sccache/ sccache], compiler cache with s3 storage. Caching C++ and Rust compilation, used in Firefox CI
** GeckoDriver, a WebDriver implementation for Firefox integrated via marionette protocol: {{bug|1340637}} ([https://github.com/mozilla/geckodriver/releases standalone releases])
* [https://github.com/luser/rust-makecab makecab], reimplementation of Microsoft's makecab tool. Used to compress PDB files before uploading to symbol server in Firefox CI.
** [https://github.com/mozilla/grcov grcov], a tool to collect and aggregate code coverage data for multiple source files, used in Firefox CI.
* Build system, etc.
** [https://github.com/mozilla/sccache/ sccache], compiler cache with s3 storage. Caching C++ and Rust compilation, used in Firefox CI.
** Parts of [https://github.com/mozsearch/mozsearch mozsearch], the backend for the [http://searchfox.org Searchfox] code indexing tool.
** [https://github.com/luser/rust-makecab makecab], a reimplementation of Microsoft's makecab tool. Used to compress PDB files before uploading to symbol server in Firefox CI.
** [https://github.com/mozilla/dump_syms/ dump_syms], a reimplementation of Google Breakpad's dump_syms tool for Windows. Used to parse PDB files and print out Breakpad-compatible symbol files.
*** '''Why Rust?''' Doesn't rely on Microsoft closed-source libraries anymore, can be cross-compiled and run on a non-Windows host, is an order of magnitude faster, takes less than a third of the memory, produces better symbols and relies on an active and friendly upstream [https://github.com/getsentry/symbolic/ symbolic]
* Application Services, server-side
** [https://github.com/mozilla-services/autopush-rs autopush-rs] Rust async based websocket server that implements Mozilla's push/webpush/broadcast protocols.
*** '''Why Rust?''' Concise code with the memory efficiency of C.
** [https://github.com/mozilla-services/megaphone/ Megaphone], a real-time update broadcast server for Firefox.
** [https://github.com/mozilla/fxa-email-service/ fxa_email_service], a service for sending email to Firefox Accounts.
** [https://github.com/mozilla-services/pairsona/ pairsona], a tool to associate instances of firefox.
* Application Services, client-side
** [https://github.com/mozilla/application-services/tree/master/components various sync-related components used on iOS and Fenix], includes a cross-compiled FxA Rust client, and storage/syncing of bookmarks, history, logins, tabs and webextensions data.
* [https://github.com/mozilla/fix-stacks/ fix-stacks], a stack frame symbolizer: {{bug|1596292}}
** '''Why Rust?''' High performance needed, a single implementation can replace multiple platform-specific scripts, and we can use the [https://github.com/getsentry/symbolic/ symbolic] crate to do all the hard parts.


=== In Progress ===
=== In Progress ===


* IPDL Parser: {{bug|1316754}}
* IPDL Parser: {{bug|1316754}}
** '''Why Rust?''' Rust is a much better language than Python for writing compilers, due to strong typing, algebraic data types, and pattern matching.


= Blockers and obstacles =
= Blockers and obstacles =
Line 171: Line 219:
This section lists areas where Rust integration could be improved.
This section lists areas where Rust integration could be improved.
* Tracking bug: Make the developer experience for Firefox + Rust great: {{Bug|rust-great}}
* Tracking bug: Make the developer experience for Firefox + Rust great: {{Bug|rust-great}}
* Compile speed and memory usage
* [https://docs.google.com/document/d/16FgQPRxNb-Z6sfJy_P7edXzMVWE86jJEZVbmV0oC-m0/ 2020 Questionnaire results]
** Incremental compilation ([https://github.com/rust-lang/rust/labels/A-incr-comp A-incr-comp issues], [https://github.com/rust-lang/rust/labels/WG-compiler-incr WG-compiler-incr issues])
* [https://docs.google.com/document/d/1puZvhWaURtViz8OC0HkB0h2dqJThVyAgLGFbIQpd4fo/ Oxidation 2020 Plan]
** [https://users.rust-lang.org/t/contract-opportunity-mozilla-distributed-compilation-cache-written-in-rust/13898 Distributed compilation cache]
** [https://github.com/rust-lang/cargo/issues/1997 Artifact caching]?
* Inlining between C++ and Rust would reduce cost of crossing the language barrier
* Debugging: improve gdb and lldb support for Rust. The first step is to establish Rust language support in DWARF distinct from the existing C++ support.
* Bindings/interop
** Immature rust-bindgen and cbindgen tools for general cross-language support. Working aroudn clang bugs in different versions and on different platforms can be tricky.
** No IPDL binding generator ({{bug|1379739}})
** No WebIDL binding generator for DOM components (Servo must have something here?)
* Android is currently [https://forge.rust-lang.org/platform-support.html not a Tier 1 platform].
** rillian: "As I understand it, the problem is no one has set up tests on a cloud service, so there's no way for upstream to gate work on Android regressions. Qemu emulation is too slow. There aren't many services with arm support, but there are a couple of new options. If someone does the work of setting something up we should be able to get the Rust team to adopt it."
* Management (landing, merging, etc.) of Shared Gecko/Servo components
** The workflow is annoying and inefficient. Furthermore, permission to land directly on autoland is sometimes needed to fix problems, but few people have that.
** There are continuous integration mismatches.
** Servo CI is potentially under-resourced.
** This is not relevant for Gecko-only Rust components, however.
* Remaining minor crash report issues {{bug|1348896}}
* IDE/symbol lookup support?
* Code coverage?
* Profiling improvements? Especially for parallel code
* Test integration?
* Are Rust's Vec, HashSet/HashMap as performant as Gecko's equivalents? {{bug|1425770}}


= Meetings =
= Meetings =


* Berlin, January 2020 (minutes lost due to technical issues, unfortunately)
* [https://github.com/servo/servo/wiki/Whistler-Oxidation-2019 Whistler, Jun 2019]
* [https://github.com/servo/servo/wiki/Orlando-Oxidation-2018 Orlando, Dec 2018]
* [https://github.com/servo/servo/wiki/San-Francisco-Oxidation San Francisco, Jun 2018]
* [https://github.com/servo/servo/wiki/Austin-Oxidation Austin, Dec 2017]
* [https://github.com/servo/servo/wiki/Austin-Oxidation Austin, Dec 2017]
* [https://github.com/servo/servo/wiki/Mozlando-Oxidation Mozlando, Dec 2015]
* [https://github.com/servo/servo/wiki/Mozlando-Oxidation Mozlando, Dec 2015]
Line 202: Line 233:
* [https://github.com/servo/servo/wiki/Whistler-GFX#servo-in-gecko Whistler, June 2015]
* [https://github.com/servo/servo/wiki/Whistler-GFX#servo-in-gecko Whistler, June 2015]
* [https://github.com/servo/servo/wiki/Mozlandia-Rust-In-Gecko Mozlandia, Dec 2014]
* [https://github.com/servo/servo/wiki/Mozlandia-Rust-In-Gecko Mozlandia, Dec 2014]
<!--
= People =
* Ralph Giles: build system, integration
* Tom Tromey: debugging
* Boris Zbarsky: Stylo, Gecko
* Nathan Froyd: build system, integration, Cargo
* Ted Mielczarek: build system, crash reporting, releng
* Emilio Cobos Álvarez: Stylo
* Manish Goregaokar: Stylo, Rust internals
* Kartikaya Gupta: gfx, WebRender
* Bobby Holley: Stylo, Gecko
* Nicholas Nethercote: coordination
* Anthony Jones: coordination
* Selena Deckelmann: coordination
* Nick Fitzgerald: bindgen
* Nika Layzell: bindings
* Alex Crichton: rustc, Cargo
* Michael Woerister: rustc
* Mike Hommey: build system, allocator, Linux distros
-->

Latest revision as of 20:12, 17 November 2020

Oxidation is a project to integrate Rust code in and around Firefox.

Rust support has been required on all platforms since Firefox 54, and the first major Rust components were shipped in Firefox 56 (encoding_rs) and 57 (Stylo). Moving forward, the goal of Oxidation is to make it easier and more productive to use Rust in Firefox, and correspondingly to increase the amount of Rust code in Firefox.

This page is intended to serve as the starting point for all matters relating to Rust code in Firefox: the what, the why, and the how.

Guidelines

The goal of this section is to provide some high-level guidelines about when Rust should be used.

In summary, Rust should be used in the following situations.

  • For new components and completely rewritten components there should be a strong bias towards using Rust, especially for code around Firefox but not within Firefox.
  • For existing components it's more complicated!

The following sections have more detail. Ultimately, choice of language for a code component is an engineering decision, with corresponding trade-offs, and is best decided by individual teams.

Rust Strengths

Rust has the following strengths.

  • Memory-safety and thread-safety. Crashes and security vulnerabilities are much less likely. (Roughly 70% of critical security vulnerabilities are due to memory safety bugs.)
  • High performance. In particular, the safety enables code that is designed more actively for performance, especially parallel performance.
  • Nimbleness. The safety enables significant changes to existing code to be made quickly and with confidence.
  • Expressiveness. It is powerful and pleasant to use, particularly once a moderate level of experience has been reached.
  • Excellent package management and an extensive ecosystem.
  • Excellent compiler error messages.
  • Excellent documentation.
  • A friendly and helpful community.

Rust Weaknesses

One major issue with Rust relates to personnel.

  • There is a wide variety of experience levels within Mozilla, for both coding and reviewing.
  • Rust's learning curve is steep at the start, which can be intimidating. (The new borrow checker released in Rust 1.31 helped greatly with this.)

There are also technical challenges.

See "Blockers and obstacles" below for more details about work being done to remedy these weaknesses.

Recommendations

Therefore, Rust is most suitable in the following situations.

  • For components that are relatively standalone, with small and simple APIs.
    • This minimizes the C++/Rust boundary layer issues.
    • Infrastructure tools that are standalone programs are ideal.
    • Note that it's good software engineering practice to write loosely-coupled components anyway.
  • For components that process untrusted input, e.g. parsers.
  • For components where parallelism can provide big performance wins.
  • For components where Servo has demonstrated success.

In terms of where to keep Rust crates, there are three options.

  • Put the crate in mozilla-central or in Servo's repository.
    • For binding code, the decision to put it into Gecko or Servo can be difficult. The best choice depends on the details of the binding code in question.
  • Put the crate on crates.io and use Cargo to access it at build-time.
    • This is only suitable for highly general-purpose crates, such as smallvec.
  • Put the crate somewhere else (e.g. a separate GitHub repository), and regularly vendor it into mozilla-central.
    • This makes sense for pre-existing third-party crates that we choose to import.
    • Otherwise, this option is not recommended, because vendoring is something of a hassle.

In general, erring on the side of tighter coupling is advisable. For example, the heapsize crate used in memory reporting was moved to crates.io, and then other crates came to depend on it. Later on it needed major API changes, and we ended up replacing it with a new crate called malloc_size_of (stored in Servo's repository) because that was easier than modifying heapsize.

Documentation and assistance

Rust in general

Rust in Firefox

Policy

Rust in Firefox docs

Getting extra help

  • The Rust channel on Mozilla's Matrix network contains lots of people who know about both Rust and Gecko.
  • Are you new to Rust and not sure if your Rust code could be improved? The following people can review Rust patches for Firefox from an "is this good Rust code?" point of view.
    • Alexis Beingessner (:Gankra)
    • Josh Bowman-Matthews (:jdm)
    • Emilio Cobos Alvarez (:emilio)
    • Manish Goregaokar (:manishearth)
    • Nika Layzell (:nika)
    • Cameron McCormack (:heycam)

Rust Components

Within Firefox

Statistics

Shipped

  • MP4 metadata parser: bug 1161350 (shipped for desktop in Firefox 48)
    • Why Rust? Parses untrusted input, replaces libstagefright, a 3rd-party library with a history of security vulnerabilities.
  • Replace uconv with encoding-rs: bug 1261841 (shipped in Firefox 56)
  • CSS style calculation (from Servo): bug stylo (shipped for desktop in Firefox 57)
    • Why Rust? Code taken from Servo, uses parallel algorithms.
  • U2F HID backend: bug 1388843 (shipped in Firefox 57)
  • libcubeb Audio backend for Linux (PulseAudio): bug 1346665 (shipped in Firefox 59)
  • libcubeb Audio backend for macOS (CoreAudio): bug 1530713 (shipped in Firefox 74)
    • Why Rust? Safer implementation avoids data racing by leveraging lifetime, variable mutability check and better mutex pattern.
  • XPIDL binding generator (bug 1293362) (shipped in Firefox 60)
  • New prefs parser: bug 1423840 (shipped in Firefox 60)
    • Why Rust? Old parser needed replacing. Well-separated component, simple interface, parses untrusted input.
  • Audio remoting for Linux: bug 1434156 (shipped in Firefox 60)
  • WebRender: bug webrender (shipped in Firefox 67, enabled for users with appropriate hardware)
    • Why Rust? Code taken from Servo, has high performance; Rust's memory and thread safety assist productivity, and allow more aggressive optimizations.
  • kvstore (key-value storage backed by LMDB): bug 1490496 (shipped in Firefox 67)
    • Why Rust? The rkv crate provides a safe, ergonomic wrapper around LMDB, our choice for simple key-value storage in Firefox. kvstore wraps rkv in an asynchronous XPCOM API for JS and C++ callers.
  • Profiler symbolication: bug 1509549 (shipped in Firefox 67)
    • Why Rust? Makes use of existing crates that handle object file parsing and symbol iteration. Easy to compile to WebAssembly.
  • XUL store, backed by rkv: bug 1460811 (landed in Firefox 68, used in Nightly only)
  • TLS certificate store, backed by rkv: bug 1429796 (shipped in Firefox 68)
  • Synced bookmark merger: bug 1482608 (shipped in Firefox 68, on by default in Nightly and early Beta)
  • Windows BITS interface: bug 1520321 (shipped in Firefox 68)
  • Japanese encoding detector: bug 1543077 (shipped in Firefox 69)
    • Why Rust? Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer.
  • audio_thread_priority bug 1429847 (shipped in Firefox 69), allow promoting threads to a real-time scheduling class, on Windows/Linux/macOS.
    • Why Rust? This crate is being used by C++ code and by Rust code (audioipc), Rust is nicer to write than C++ (especially for what is essentially just a series of system calls, the error checking style is nice), and cbindgen made it trivial to expose a C ABI.
  • Dogear a bookmark merger for Sync. Shipped pref'd-off in bug 1482608 (Firefox 68), enabled by default in bug 1588005 (shipped in Firefox 72)
    • Why Rust? A single performant and safe implementation shared between desktop and the bookmarks engine in application-services
  • Unicode Language Identifier: bug 1571915 (shipped in Firefox 72)
    • Why Rust? Much faster, parser-heavy, easier to handle low-memory footprint thanks to `tinystr`.
  • Language Negotiation: bug 1581960 (shipped in Firefox 72)
    • Why Rust? Ties into `unic-langid`, easier to handle list filtering and ordering.
  • Replace libhyphen with mapped_hyph, bug 1590167 (shipped in Firefox 72).
  • Encoding detector: bug 1551276 (shipped in Firefox 73)
    • Why Rust? Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer, potentially parallelizable with Rayon.
  • Integrate fluent-rs, a localization system: bug 1560038 (shipped in Firefox 76)
    • Why Rust? Performance and memory wins are substantial over previous JS implementation. It brings zero-copy parsing, and memory savvy resolving of localization strings. It also paves the way for migrating the rest of the Fluent APIs away from JS which is required for Fission.
  • qcms [1] ported from C to Rust
    • Why Rust? Memory safety

In progress

  • Port Localization API to Rust: bug 1613705
  • neqo A QUIC implentation.
  • cranelift, a low-level retargetable code generator: bug 1469027
    • Why Rust? It's a new, well-separated component with a clear interface. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
  • SmooshMonkey A Rust front-end for SpiderMonkey (featuring jsparagus).
    • Why Rust? Parses untrusted input. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
  • Audio remoting for Windows: bug 1432303
  • Audio remoting for Mac OS: bug 1425788
  • SDP parsing in WebRTC: bug 1365792
    • Why Rust? SDP is a complex text protocol and the existing parser in C has a history of security issues. This also allows us to tailor the SDP parser specifically for the subset used in WebRTC, further reducing its surface area. It is currently run in parallel with the C parser in Nightly.
  • Linebreaking with xi-unicode: bug 1290022 (last update late 2016)
  • Background Update Agent for Windows: bug 1343669
  • wgpu, a WebGPU API implementation: bug webgpu-mvp (in Nightly since 72)
    • Why Rust? Complex tracking logic, wide attack area. Also, leverages Rust ecosystem for building libraries on top of our native implementation and the API that will target the Web.
  • Integrate the Glean SDK, a data collection library.

Proposed

  • WebMIDI: bug 1201593, bug 1201596, bug 1201598
  • Gamepad code: bug 1286699
  • Replace the telemetry module(?)
    • Why Rust? The existing C++ code has a history of threading problems.
  • Replace DOM serializers (XML, HTML for Save As.., plain text)
    • Why Rust? Need a rewrite anyway. Minor history of security vulnerabilities.
  • Expose Rust API to JS Debugger: bug 1263317
  • Generate Rust bindings for IPDL actors (bug 1379739)
  • WebM demuxer: bug 1267492
  • Parallel JS parsing: fast preparse to find function boundaries, parse non-overlapping functions in parallel with a unification step to handle free names and such (no bug on file yet)
    • Why Rust? Parses untrusted input. Requires safe threading. And generally, Rust is a better language than C++ for parsers, due to strong typing, algebraic data types, and pattern matching.
  • Crash reporter
    • Why Rust? Code needs rewriting, useful Rust crates exist that could be used.
  • Sync/FxA components
    • Why Rust? Single safe and performant implementation which is shared across all our products.
  • libcubeb Audio backend for Windows (WASAPI)
  • Replace the XML parser, possibly via c2rust: bug 1611289
    • Why Rust? Parses untrusted input, replaces expat, a 3rd-party library with a history of frequent security vulnerabilities.
  • Rewrite the ICE stack used by WebRTC; bug 1616966
    • Why Rust? Works with network data.

Outside Firefox

Completed

  • Testing
    • GeckoDriver, a WebDriver implementation for Firefox integrated via marionette protocol: bug 1340637 (standalone releases)
    • grcov, a tool to collect and aggregate code coverage data for multiple source files, used in Firefox CI.
  • Build system, etc.
    • sccache, compiler cache with s3 storage. Caching C++ and Rust compilation, used in Firefox CI.
    • Parts of mozsearch, the backend for the Searchfox code indexing tool.
    • makecab, a reimplementation of Microsoft's makecab tool. Used to compress PDB files before uploading to symbol server in Firefox CI.
    • dump_syms, a reimplementation of Google Breakpad's dump_syms tool for Windows. Used to parse PDB files and print out Breakpad-compatible symbol files.
      • Why Rust? Doesn't rely on Microsoft closed-source libraries anymore, can be cross-compiled and run on a non-Windows host, is an order of magnitude faster, takes less than a third of the memory, produces better symbols and relies on an active and friendly upstream symbolic
  • Application Services, server-side
    • autopush-rs Rust async based websocket server that implements Mozilla's push/webpush/broadcast protocols.
      • Why Rust? Concise code with the memory efficiency of C.
    • Megaphone, a real-time update broadcast server for Firefox.
    • fxa_email_service, a service for sending email to Firefox Accounts.
    • pairsona, a tool to associate instances of firefox.
  • Application Services, client-side
  • fix-stacks, a stack frame symbolizer: bug 1596292
    • Why Rust? High performance needed, a single implementation can replace multiple platform-specific scripts, and we can use the symbolic crate to do all the hard parts.

In Progress

  • IPDL Parser: bug 1316754
    • Why Rust? Rust is a much better language than Python for writing compilers, due to strong typing, algebraic data types, and pattern matching.

Blockers and obstacles

This section lists areas where Rust integration could be improved.

Meetings

Retrieved from "https://wiki.allizom.org/index.php?title=Oxidation&oldid=1232260"