MozillaWiki

Oxidation: Difference between revisions

Page Discussion

Oxidation (view source)

Revision as of 20:12, 17 November 2020

2,382 bytes added ,  17 November 2020
no edit summary
(Based in Phabricator, the expat rewrite seems to be in progress rather than just planned)
No edit summary
 
(46 intermediate revisions by 13 users not shown)
Line 1: Line 1:
'''Oxidation''' is a project to integrate [https://www.rust-lang.org/ Rust] code in and around Firefox.  
'''Oxidation''' is a project to integrate [https://www.rust-lang.org/ Rust] code in and around Firefox.  


Rust support has been required on all platforms since Firefox 54, and the first major Rust components were shipped in Firefox 56 (encoding_rs) and 57 (Stylo). Moving forward, the goal of Oxidation is to make it easier and more pleasant to use Rust in Firefox, and correspondingly to increase the amount of Rust code in Firefox.
Rust support has been required on all platforms since Firefox 54, and the first major Rust components were shipped in Firefox 56 (encoding_rs) and 57 (Stylo). Moving forward, the goal of Oxidation is to make it easier and more productive to use Rust in Firefox, and correspondingly to increase the amount of Rust code in Firefox.


This page is intended to serve as the starting point for all matters relating to Rust code in Firefox: the what, the why, and the how.
This page is intended to serve as the starting point for all matters relating to Rust code in Firefox: the what, the why, and the how.
Line 18: Line 18:


Rust has the following strengths.
Rust has the following strengths.
* Memory safety, which prevents crashes and security vulnerabilities.
* Memory-safety and thread-safety. Crashes and security vulnerabilities are much less likely. (Roughly 70% of critical security vulnerabilities are due to memory safety bugs.)
* Thread safety, which enables improved performance via parallelism.
* High performance. In particular, the safety enables code that is designed more actively for performance, especially parallel performance.
* Nimbleness: the safety makes it easy to make significant changes quickly and with confidence.
* Nimbleness. The safety enables significant changes to existing code to be made quickly and with confidence.
* Pleasant to use, particularly once a moderate level of experience has been reached.
* Expressiveness. It is powerful and pleasant to use, particularly once a moderate level of experience has been reached.
* A great community.
* Excellent package management and an extensive ecosystem.
* Excellent compiler error messages.
* Excellent documentation.
* A friendly and helpful community.


== Rust Weaknesses ==
== Rust Weaknesses ==
Line 28: Line 31:
One major issue with Rust relates to personnel.
One major issue with Rust relates to personnel.
* There is a wide variety of experience levels within Mozilla, for both coding and reviewing.
* There is a wide variety of experience levels within Mozilla, for both coding and reviewing.
* Rust's learning curve is steep at the start, which can be intimidating.
* Rust's learning curve is steep at the start, which can be intimidating. (The new borrow checker released in Rust 1.31 helped greatly with this.)


There are also technical challenges.
There are also technical challenges.
Line 71: Line 74:
== Rust in Firefox ==
== Rust in Firefox ==


* [https://developer.mozilla.org/en-US/Firefox/Building_Firefox_with_Rust_code Developer Documentation]
Policy
* [https://firefox-source-docs.mozilla.org/build/buildsystem/rust.html Build System Documentation]
* [[Rust_Update_Policy_for_Firefox|Rust Update Policy for Firefox]]
* [[Rust_Update_Policy_for_Firefox|Rust Update Policy for Firefox]]
* [https://docs.google.com/presentation/d/1qkPwISU1BvsTVyqLuVhisSMuIS_DiXb8X09-boszcu0/edit#slide=id.g5bfdbbe5c9_0_71 How to build an XPCOM component in Rust]
 
* [https://groups.google.com/forum/#!topic/mozilla.dev.platform/u8scZop3FkM In-tree helper crates for Rust XPCOM components]
Rust in Firefox docs
* The #servo IRC channel contains lots of people who know about both Rust and Gecko.
* [https://firefox-source-docs.mozilla.org/build/buildsystem/rust.html Adding Rust code]
* [https://firefox-source-docs.mozilla.org/writing-rust-code/index.html Writing Rust code]
* [https://firefox-source-docs.mozilla.org/testing-rust-code/index.html Testing & debugging Rust code]
 
Getting extra help
* The Rust channel on [https://wiki.mozilla.org/Matrix Mozilla's Matrix network] contains lots of people who know about both Rust and Gecko.
* Are you new to Rust and not sure if your Rust code could be improved? The following people can review Rust patches for Firefox from an "is this good Rust code?" point of view.
* Are you new to Rust and not sure if your Rust code could be improved? The following people can review Rust patches for Firefox from an "is this good Rust code?" point of view.
** Alexis Beingessner (:gankro)
** Alexis Beingessner (:Gankra)
** Josh Bowman-Matthews (:jdm)
** Josh Bowman-Matthews (:jdm)
** Emilio Cobos Alvarez (:emilio)
** Emilio Cobos Alvarez (:emilio)
** Manish Goregaokar (:manishearth)
** Manish Goregaokar (:manishearth)
** Nika Layzell (:mystor)
** Nika Layzell (:nika)
** Cameron McCormack (:heycam)
** Cameron McCormack (:heycam)


== FAQ ==
= Rust Components =


'''Q:''' What is the policy for vendoring non-Mozilla crates into mozilla-central?<br />
== Within Firefox ==
 
'''A:''' It is possible. The most important point is that the license must be compatible. Reviewers should also look at the crate code some to check that it looks reasonable (especially for unsafe code) and that it has reasonable tests. Other than that, there is no formal sign-off procedure, but one may be added in the future.
 
 
'''Q:''' Do we support building standalone Rust programs?<br />


'''A:''' Yes! Look for <tt>RUST_PROGRAMS</tt> rules in <tt>moz.build</tt> files.
=== Statistics ===


 
* [https://docs.google.com/spreadsheets/d/1flUGg6Ut4bjtyWdyH_9emD9EAN01ljTAVft2S4Dq620/edit#gid=885787479 Lines of compiled Rust code shipped in Firefox (over time)]
'''Q:''' How are in-tree Rust crates tested?<br />
* [https://4e6.github.io/firefox-lang-stats/ Lines of Rust code in mozilla-central (current)]
 
'''A:''' In general we don't run tests for third-party crates; the assumption is that these crates are sufficiently well-tested elsewhere. (Which means that large test fixtures should be removed when vendoring third-party crates, because they just bloat mozilla-central.) Mozilla crates can be tested with <tt>cargo test</tt> by adding them to <tt>RUST_TESTS</tt> in <tt>toolkit/library/rust/moz.build</tt>. Alternatively, you can write a GTest that uses FFI to call into Rust code.
 
= Rust Components =
 
== Within Firefox ==


=== Shipped ===
=== Shipped ===
Line 113: Line 109:
** '''Why Rust?''' Code taken from Servo, uses parallel algorithms.
** '''Why Rust?''' Code taken from Servo, uses parallel algorithms.
* U2F HID backend: {{bug|1388843}} (shipped in Firefox 57)
* U2F HID backend: {{bug|1388843}} (shipped in Firefox 57)
* libcubeb Audio backend for Linux (PulseAudio): {{bug|1346665}} (shipped in Firefox 59)
* libcubeb Audio backend for macOS (CoreAudio): {{bug|1530713}} (shipped in Firefox 74)
** '''Why Rust?''' Safer implementation avoids data racing by leveraging lifetime, variable mutability check and better mutex pattern.
* XPIDL binding generator ({{bug|1293362}}) (shipped in Firefox 60)
* XPIDL binding generator ({{bug|1293362}}) (shipped in Firefox 60)
* New prefs parser: {{bug|1423840}} (shipped in Firefox 60)
* New prefs parser: {{bug|1423840}} (shipped in Firefox 60)
Line 118: Line 117:
* Audio remoting for Linux: {{bug|1434156}} (shipped in Firefox 60)
* Audio remoting for Linux: {{bug|1434156}} (shipped in Firefox 60)
* WebRender: {{bug|webrender}} (shipped in Firefox 67, enabled for users with appropriate hardware)
* WebRender: {{bug|webrender}} (shipped in Firefox 67, enabled for users with appropriate hardware)
** '''Why Rust?''' Code taken from Servo, has high performance; Rust's memory and thread safety provides protection against complexity.
** '''Why Rust?''' Code taken from Servo, has high performance; Rust's memory and thread safety assist productivity, and allow more aggressive optimizations.
* kvstore (key-value storage backed by LMDB): {{bug|1490496}} (shipped in Firefox 67)
* kvstore (key-value storage backed by LMDB): {{bug|1490496}} (shipped in Firefox 67)
** '''Why Rust?''' The rkv crate provides a safe, ergonomic wrapper around LMDB, our choice for simple key-value storage in Firefox. kvstore wraps rkv in an asynchronous XPCOM API for JS and C++ callers.
** '''Why Rust?''' The rkv crate provides a safe, ergonomic wrapper around LMDB, our choice for simple key-value storage in Firefox. kvstore wraps rkv in an asynchronous XPCOM API for JS and C++ callers.
* Profiler symbolication: {{bug|1509549}} (shipped in Firefox 67)
** '''Why Rust?''' Makes use of existing crates that handle object file parsing and symbol iteration. Easy to compile to WebAssembly.
* XUL store, backed by rkv: {{bug|1460811}} (landed in Firefox 68, used in Nightly only)
* XUL store, backed by rkv: {{bug|1460811}} (landed in Firefox 68, used in Nightly only)
* TLS certificate store, backed by rkv: {{bug|1429796}} (shipped in Firefox 68)
* TLS certificate store, backed by rkv: {{bug|1429796}} (shipped in Firefox 68)
Line 128: Line 129:
* Japanese encoding detector: {{bug|1543077}} (shipped in Firefox 69)
* Japanese encoding detector: {{bug|1543077}} (shipped in Firefox 69)
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer.
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer.
* [https://github.com/padenot/audio_thread_priority audio_thread_priority] {{bug|1429847}} (shipped in Firefox 69), allow promoting threads to a real-time scheduling class, on Windows/Linux/macOS.
** '''Why Rust?''' This crate is being used by C++ code and by Rust code (audioipc), Rust is nicer to write than C++ (especially for what is essentially just a series of system calls, the error checking style is nice), and cbindgen made it trivial to expose a C ABI.
* [https://github.com/mozilla/dogear/ Dogear] a bookmark merger for Sync. Shipped pref'd-off in {{bug|1482608}} (Firefox 68), enabled by default in {{bug|1588005}} (shipped in Firefox 72)
** '''Why Rust?''' A single performant and safe implementation shared between desktop and the bookmarks engine in [https://github.com/mozilla/application-services/tree/master/components/places application-services]
* Unicode Language Identifier: {{bug|1571915}} (shipped in Firefox 72)
* Unicode Language Identifier: {{bug|1571915}} (shipped in Firefox 72)
** '''Why Rust?''' Much faster, parser-heavy, easier to handle low-memory footprint thanks to `tinystr`.
** '''Why Rust?''' Much faster, parser-heavy, easier to handle low-memory footprint thanks to `tinystr`.
* Language Negotiation: {{bug|1581960}} (shipped in Firefox 72)
* Language Negotiation: {{bug|1581960}} (shipped in Firefox 72)
** '''Why Rust?''' Ties into `unic-langid`, easier to handle list filtering and ordering.
** '''Why Rust?''' Ties into `unic-langid`, easier to handle list filtering and ordering.
* Replace libhyphen with mapped_hyph, {{bug|1590167}} (shipped in Firefox 72).
* Encoding detector: {{bug|1551276}} (shipped in Firefox 73)
* Encoding detector: {{bug|1551276}} (shipped in Firefox 73)
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer, potentially parallelizable with Rayon.
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer, potentially parallelizable with Rayon.
* Integrate [https://github.com/projectfluent/fluent-rs fluent-rs], a localization system: {{bug|1560038}} (shipped in Firefox 76)
** '''Why Rust?''' Performance and memory wins are substantial over previous JS implementation. It brings zero-copy parsing, and memory savvy resolving of localization strings. It also paves the way for migrating the rest of the Fluent APIs away from JS which is required for Fission.
* qcms [https://searchfox.org/mozilla-central/source/gfx/qcms] ported from C to Rust
** '''Why Rust?''' Memory safety


=== In progress ===
=== In progress ===


* Integrate [https://github.com/projectfluent/fluent-rs fluent-rs], a localization system: {{bug|1560038}}
* Port [https://github.com/projectfluent/fluent-rs Localization API] to Rust: {{bug|1613705}}
* [https://github.com/mozilla/neqo neqo] A QUIC implentation.
* [https://github.com/mozilla/neqo neqo] A QUIC implentation.
* [https://github.com/CraneStation/cranelift/ cranelift, a low-level retargetable code generator]: {{bug|1469027}}
* [https://github.com/CraneStation/cranelift/ cranelift, a low-level retargetable code generator]: {{bug|1469027}}
** '''Why Rust?''' It's a new, well-separated component with a clear interface. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
** '''Why Rust?''' It's a new, well-separated component with a clear interface. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
* [https://github.com/mozilla-spidermonkey/rust-frontend Project Visage] A Rust front-end for SpiderMonkey (featuring [https://github.com/mozilla-spidermonkey/jsparagus jsparagus]).
* [https://github.com/mozilla-spidermonkey/rust-frontend SmooshMonkey] A Rust front-end for SpiderMonkey (featuring [https://github.com/mozilla-spidermonkey/jsparagus jsparagus]).
** '''Why Rust?''' Parses untrusted input. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
** '''Why Rust?''' Parses untrusted input. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
* Audio remoting for Windows: {{bug|1432303}}
* Audio remoting for Windows: {{bug|1432303}}
* Audio remoting for Mac OS: {{bug|1425788}}
* Audio remoting for Mac OS: {{bug|1425788}}
* SDP parsing in WebRTC: {{bug|1365792}}
* SDP parsing in WebRTC: {{bug|1365792}}
** '''Why Rust?''' SDP is a complex text protocol and the existing parser in C has a history of security issues.
** '''Why Rust?''' SDP is a complex text protocol and the existing parser in C has a history of security issues. This also allows us to tailor the SDP parser specifically for the subset used in WebRTC, further reducing its surface area. It is currently run in parallel with the C parser in Nightly.
* Linebreaking with xi-unicode: {{bug|1290022}} (last update late 2016)
* Linebreaking with xi-unicode: {{bug|1290022}} (last update late 2016)
* Background Update Agent for Windows: {{bug|1343669}}
* Background Update Agent for Windows: {{bug|1343669}}
* Replace the XML parser: {{bug|1611289}}
* [https://github.com/gfx-rs/wgpu wgpu], a [https://gpuweb.github.io/gpuweb/ WebGPU] API implementation: {{bug|webgpu-mvp}} (in Nightly since 72)
** '''Why Rust?''' Parses untrusted input, replaces expat, a 3rd-party library with a history of frequent security vulnerabilities.
** '''Why Rust?''' Complex tracking logic, wide attack area. Also, leverages Rust ecosystem for building libraries on top of our native implementation and the API that will target the Web.
* Integrate the [https://github.com/mozilla/glean/ Glean SDK], a data collection library.


=== Proposed ===
=== Proposed ===


* Parallel layout
** '''Why Rust?''' Existing code from Servo, parallel performance.
* WebMIDI: {{bug|1201593}}, {{bug|1201596}}, {{bug|1201598}}
* WebMIDI: {{bug|1201593}}, {{bug|1201596}}, {{bug|1201598}}
* Gamepad code: {{bug|1286699}}
* Gamepad code: {{bug|1286699}}
Line 162: Line 171:
* Replace DOM serializers (XML, HTML for Save As.., plain text)
* Replace DOM serializers (XML, HTML for Save As.., plain text)
** '''Why Rust?''' Need a rewrite anyway. Minor history of security vulnerabilities.
** '''Why Rust?''' Need a rewrite anyway. Minor history of security vulnerabilities.
* Image decoders?
** '''Why Rust?''' Parsing untrusted input, some history of security vulnerabilities.
* Expose Rust API to JS Debugger: {{bug|1263317}}
* Expose Rust API to JS Debugger: {{bug|1263317}}
* Generate Rust bindings for IPDL actors ({{bug|1379739}})
* Generate Rust bindings for IPDL actors ({{bug|1379739}})
Line 171: Line 178:
* Crash reporter
* Crash reporter
** '''Why Rust?''' Code needs rewriting, useful Rust crates exist that could be used.
** '''Why Rust?''' Code needs rewriting, useful Rust crates exist that could be used.
* [https://github.com/mozilla/application-services Sync/FxA components]
** '''Why Rust?''' Single safe and performant implementation which is shared across all our products.
* libcubeb Audio backend for Windows (WASAPI)
* Replace the XML parser, possibly via c2rust: {{bug|1611289}}
** '''Why Rust?''' Parses untrusted input, replaces expat, a 3rd-party library with a history of frequent security vulnerabilities.
* Rewrite the ICE stack used by WebRTC; {{bug|1616966}}
** '''Why Rust?''' Works with network data.


== Outside Firefox ==
== Outside Firefox ==
Line 183: Line 197:
** Parts of [https://github.com/mozsearch/mozsearch mozsearch], the backend for the [http://searchfox.org Searchfox] code indexing tool.
** Parts of [https://github.com/mozsearch/mozsearch mozsearch], the backend for the [http://searchfox.org Searchfox] code indexing tool.
** [https://github.com/luser/rust-makecab makecab], a reimplementation of Microsoft's makecab tool. Used to compress PDB files before uploading to symbol server in Firefox CI.
** [https://github.com/luser/rust-makecab makecab], a reimplementation of Microsoft's makecab tool. Used to compress PDB files before uploading to symbol server in Firefox CI.
** [https://github.com/mozilla/dump_syms/ dump_syms], a reimplementation of Google Breakpad's dump_syms tool for Windows. Used to parse PDB files and print out Breakpad-compatible symbol files.
*** '''Why Rust?''' Doesn't rely on Microsoft closed-source libraries anymore, can be cross-compiled and run on a non-Windows host, is an order of magnitude faster, takes less than a third of the memory, produces better symbols and relies on an active and friendly upstream [https://github.com/getsentry/symbolic/ symbolic]
* Application Services, server-side
* Application Services, server-side
** [https://github.com/mozilla-services/autopush-rs autopush-rs] Rust async based websocket server that implements Mozilla's push/webpush/broadcast protocols.
** [https://github.com/mozilla-services/autopush-rs autopush-rs] Rust async based websocket server that implements Mozilla's push/webpush/broadcast protocols.
Line 190: Line 206:
** [https://github.com/mozilla-services/pairsona/ pairsona], a tool to associate instances of firefox.
** [https://github.com/mozilla-services/pairsona/ pairsona], a tool to associate instances of firefox.
* Application Services, client-side
* Application Services, client-side
** [https://github.com/mozilla/application-services/tree/master/fxa-rust-client fxa-rust-client], a cross-compiled FxA Rust client that can work with Firefox Sync keys and more.
** [https://github.com/mozilla/application-services/tree/master/components various sync-related components used on iOS and Fenix], includes a cross-compiled FxA Rust client, and storage/syncing of bookmarks, history, logins, tabs and webextensions data.
* [https://github.com/mozilla/fix-stacks/ fix-stacks], a stack frame symbolizer: {{bug|1596292}}
** '''Why Rust?''' High performance needed, a single implementation can replace multiple platform-specific scripts, and we can use the [https://github.com/getsentry/symbolic/ symbolic] crate to do all the hard parts.


=== In Progress ===
=== In Progress ===
Line 201: Line 219:
This section lists areas where Rust integration could be improved.
This section lists areas where Rust integration could be improved.
* Tracking bug: Make the developer experience for Firefox + Rust great: {{Bug|rust-great}}
* Tracking bug: Make the developer experience for Firefox + Rust great: {{Bug|rust-great}}
* Compile speed and memory usage
* [https://docs.google.com/document/d/16FgQPRxNb-Z6sfJy_P7edXzMVWE86jJEZVbmV0oC-m0/ 2020 Questionnaire results]
** Incremental compilation ([https://github.com/rust-lang/rust/labels/A-incr-comp A-incr-comp issues], [https://github.com/rust-lang/rust/labels/WG-compiler-incr WG-compiler-incr issues])
* [https://docs.google.com/document/d/1puZvhWaURtViz8OC0HkB0h2dqJThVyAgLGFbIQpd4fo/ Oxidation 2020 Plan]
** [https://users.rust-lang.org/t/contract-opportunity-mozilla-distributed-compilation-cache-written-in-rust/13898 Distributed compilation cache]
** [https://github.com/rust-lang/cargo/issues/1997 Artifact caching]?
* Debugging: improve gdb and lldb support for Rust. The first step is to establish Rust language support in DWARF distinct from the existing C++ support.
* Bindings/interop
** Immature rust-bindgen and cbindgen tools for general cross-language support. Working aroudn clang bugs in different versions and on different platforms can be tricky.
** No IPDL binding generator ({{bug|1379739}})
** No WebIDL binding generator for DOM components (Servo must have something here?)
* Remaining minor crash report issues {{bug|1348896}}
* IDE/symbol lookup support?
* Code coverage?
* Profiling improvements? Especially for parallel code
* Test integration?


= Meetings =
= Meetings =


* Berlin, January 2020 (minutes lost due to technical issues, unfortunately)
* [https://github.com/servo/servo/wiki/Whistler-Oxidation-2019 Whistler, Jun 2019]
* [https://github.com/servo/servo/wiki/Whistler-Oxidation-2019 Whistler, Jun 2019]
* [https://github.com/servo/servo/wiki/Orlando-Oxidation-2018 Orlando, Dec 2018]
* [https://github.com/servo/servo/wiki/Orlando-Oxidation-2018 Orlando, Dec 2018]
Jrmuizel
Confirmed users
523

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1223880...1232260"