MozillaWiki

Oxidation: Difference between revisions

Page Discussion

Oxidation (view source)

Revision as of 20:12, 17 November 2020

1,396 bytes added ,  17 November 2020
no edit summary
(dogear)
No edit summary
 
(39 intermediate revisions by 12 users not shown)
Line 1: Line 1:
'''Oxidation''' is a project to integrate [https://www.rust-lang.org/ Rust] code in and around Firefox.  
'''Oxidation''' is a project to integrate [https://www.rust-lang.org/ Rust] code in and around Firefox.  


Rust support has been required on all platforms since Firefox 54, and the first major Rust components were shipped in Firefox 56 (encoding_rs) and 57 (Stylo). Moving forward, the goal of Oxidation is to make it easier and more pleasant to use Rust in Firefox, and correspondingly to increase the amount of Rust code in Firefox.
Rust support has been required on all platforms since Firefox 54, and the first major Rust components were shipped in Firefox 56 (encoding_rs) and 57 (Stylo). Moving forward, the goal of Oxidation is to make it easier and more productive to use Rust in Firefox, and correspondingly to increase the amount of Rust code in Firefox.


This page is intended to serve as the starting point for all matters relating to Rust code in Firefox: the what, the why, and the how.
This page is intended to serve as the starting point for all matters relating to Rust code in Firefox: the what, the why, and the how.
Line 18: Line 18:


Rust has the following strengths.
Rust has the following strengths.
* Memory-safety and thread-safety.
* Memory-safety and thread-safety. Crashes and security vulnerabilities are much less likely. (Roughly 70% of critical security vulnerabilities are due to memory safety bugs.)
** Crashes and security vulnerabilities are much less likely. (Roughly 70% of critical security vulnerabilities are due to memory safety bugs.)
* High performance. In particular, the safety enables code that is designed more actively for performance, especially parallel performance.
** The safety guarantees allow code that is designed more actively for performance, especially parallel performance.
* Nimbleness. The safety enables significant changes to existing code to be made quickly and with confidence.
** The safety guarantees allow significant changes to existing code to be made quickly and with confidence.
* Expressiveness. It is powerful and pleasant to use, particularly once a moderate level of experience has been reached.
* Expressive and pleasant to use, particularly once a moderate level of experience has been reached.
* Excellent package management and an extensive ecosystem.
* Excellent compiler error messages.
* Excellent compiler error messages.
* Excellent documentation.
* Excellent documentation.
Line 74: Line 74:
== Rust in Firefox ==
== Rust in Firefox ==


* [https://developer.mozilla.org/en-US/Firefox/Building_Firefox_with_Rust_code Developer Documentation]
Policy
* [https://firefox-source-docs.mozilla.org/build/buildsystem/rust.html Build System Documentation]
* [[Rust_Update_Policy_for_Firefox|Rust Update Policy for Firefox]]
* [[Rust_Update_Policy_for_Firefox|Rust Update Policy for Firefox]]
* [https://docs.google.com/presentation/d/1qkPwISU1BvsTVyqLuVhisSMuIS_DiXb8X09-boszcu0/edit#slide=id.g5bfdbbe5c9_0_71 How to build an XPCOM component in Rust]
 
* [https://groups.google.com/forum/#!topic/mozilla.dev.platform/u8scZop3FkM In-tree helper crates for Rust XPCOM components]
Rust in Firefox docs
* The Rust Matrix channel contains lots of people who know about both Rust and Gecko.
* [https://firefox-source-docs.mozilla.org/build/buildsystem/rust.html Adding Rust code]
* [https://firefox-source-docs.mozilla.org/writing-rust-code/index.html Writing Rust code]
* [https://firefox-source-docs.mozilla.org/testing-rust-code/index.html Testing & debugging Rust code]
 
Getting extra help
* The Rust channel on [https://wiki.mozilla.org/Matrix Mozilla's Matrix network] contains lots of people who know about both Rust and Gecko.
* Are you new to Rust and not sure if your Rust code could be improved? The following people can review Rust patches for Firefox from an "is this good Rust code?" point of view.
* Are you new to Rust and not sure if your Rust code could be improved? The following people can review Rust patches for Firefox from an "is this good Rust code?" point of view.
** Alexis Beingessner (:Gankra)
** Alexis Beingessner (:Gankra)
Line 88: Line 92:
** Cameron McCormack (:heycam)
** Cameron McCormack (:heycam)


== FAQ ==
= Rust Components =


'''Q:''' What is the policy for vendoring non-Mozilla crates into mozilla-central?<br />
== Within Firefox ==
 
'''A:''' It is possible. The most important point is that the license must be compatible. Reviewers should also look at the crate code some to check that it looks reasonable (especially for unsafe code) and that it has reasonable tests. Other than that, there is no formal sign-off procedure, but one may be added in the future.
 
 
'''Q:''' Do we support building standalone Rust programs?<br />


'''A:''' Yes! Look for <tt>RUST_PROGRAMS</tt> rules in <tt>moz.build</tt> files.
=== Statistics ===


 
* [https://docs.google.com/spreadsheets/d/1flUGg6Ut4bjtyWdyH_9emD9EAN01ljTAVft2S4Dq620/edit#gid=885787479 Lines of compiled Rust code shipped in Firefox (over time)]
'''Q:''' How are in-tree Rust crates tested?<br />
* [https://4e6.github.io/firefox-lang-stats/ Lines of Rust code in mozilla-central (current)]
 
'''A:''' In general we don't run tests for third-party crates; the assumption is that these crates are sufficiently well-tested elsewhere. (Which means that large test fixtures should be removed when vendoring third-party crates, because they just bloat mozilla-central.) Mozilla crates can be tested with <tt>cargo test</tt> by adding them to <tt>RUST_TESTS</tt> in <tt>toolkit/library/rust/moz.build</tt>. Alternatively, you can write a GTest that uses FFI to call into Rust code.
 
= Rust Components =
 
== Within Firefox ==


=== Shipped ===
=== Shipped ===
Line 116: Line 109:
** '''Why Rust?''' Code taken from Servo, uses parallel algorithms.
** '''Why Rust?''' Code taken from Servo, uses parallel algorithms.
* U2F HID backend: {{bug|1388843}} (shipped in Firefox 57)
* U2F HID backend: {{bug|1388843}} (shipped in Firefox 57)
* libcubeb Audio backend for Linux (PulseAudio): {{bug|1346665}} (shipped in Firefox 59)
* libcubeb Audio backend for macOS (CoreAudio): {{bug|1530713}} (shipped in Firefox 74)
** '''Why Rust?''' Safer implementation avoids data racing by leveraging lifetime, variable mutability check and better mutex pattern.
* XPIDL binding generator ({{bug|1293362}}) (shipped in Firefox 60)
* XPIDL binding generator ({{bug|1293362}}) (shipped in Firefox 60)
* New prefs parser: {{bug|1423840}} (shipped in Firefox 60)
* New prefs parser: {{bug|1423840}} (shipped in Firefox 60)
Line 121: Line 117:
* Audio remoting for Linux: {{bug|1434156}} (shipped in Firefox 60)
* Audio remoting for Linux: {{bug|1434156}} (shipped in Firefox 60)
* WebRender: {{bug|webrender}} (shipped in Firefox 67, enabled for users with appropriate hardware)
* WebRender: {{bug|webrender}} (shipped in Firefox 67, enabled for users with appropriate hardware)
** '''Why Rust?''' Code taken from Servo, has high performance; Rust's memory and thread safety provides protection against complexity.
** '''Why Rust?''' Code taken from Servo, has high performance; Rust's memory and thread safety assist productivity, and allow more aggressive optimizations.
* kvstore (key-value storage backed by LMDB): {{bug|1490496}} (shipped in Firefox 67)
* kvstore (key-value storage backed by LMDB): {{bug|1490496}} (shipped in Firefox 67)
** '''Why Rust?''' The rkv crate provides a safe, ergonomic wrapper around LMDB, our choice for simple key-value storage in Firefox. kvstore wraps rkv in an asynchronous XPCOM API for JS and C++ callers.
** '''Why Rust?''' The rkv crate provides a safe, ergonomic wrapper around LMDB, our choice for simple key-value storage in Firefox. kvstore wraps rkv in an asynchronous XPCOM API for JS and C++ callers.
* Profiler symbolication: {{bug|1509549}} (shipped in Firefox 67)
** '''Why Rust?''' Makes use of existing crates that handle object file parsing and symbol iteration. Easy to compile to WebAssembly.
* XUL store, backed by rkv: {{bug|1460811}} (landed in Firefox 68, used in Nightly only)
* XUL store, backed by rkv: {{bug|1460811}} (landed in Firefox 68, used in Nightly only)
* TLS certificate store, backed by rkv: {{bug|1429796}} (shipped in Firefox 68)
* TLS certificate store, backed by rkv: {{bug|1429796}} (shipped in Firefox 68)
Line 131: Line 129:
* Japanese encoding detector: {{bug|1543077}} (shipped in Firefox 69)
* Japanese encoding detector: {{bug|1543077}} (shipped in Firefox 69)
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer.
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer.
* [https://github.com/padenot/audio_thread_priority audio_thread_priority] {{bug|1429847}} (shipped in Firefox 69), allow promoting threads to a real-time scheduling class, on Windows/Linux/macOS.
** '''Why Rust?''' This crate is being used by C++ code and by Rust code (audioipc), Rust is nicer to write than C++ (especially for what is essentially just a series of system calls, the error checking style is nice), and cbindgen made it trivial to expose a C ABI.
* [https://github.com/mozilla/dogear/ Dogear] a bookmark merger for Sync. Shipped pref'd-off in {{bug|1482608}} (Firefox 68), enabled by default in {{bug|1588005}} (shipped in Firefox 72)
** '''Why Rust?''' A single performant and safe implementation shared between desktop and the bookmarks engine in [https://github.com/mozilla/application-services/tree/master/components/places application-services]
* Unicode Language Identifier: {{bug|1571915}} (shipped in Firefox 72)
* Unicode Language Identifier: {{bug|1571915}} (shipped in Firefox 72)
** '''Why Rust?''' Much faster, parser-heavy, easier to handle low-memory footprint thanks to `tinystr`.
** '''Why Rust?''' Much faster, parser-heavy, easier to handle low-memory footprint thanks to `tinystr`.
* Language Negotiation: {{bug|1581960}} (shipped in Firefox 72)
* Language Negotiation: {{bug|1581960}} (shipped in Firefox 72)
** '''Why Rust?''' Ties into `unic-langid`, easier to handle list filtering and ordering.
** '''Why Rust?''' Ties into `unic-langid`, easier to handle list filtering and ordering.
* Replace libhyphen with mapped_hyph, {{bug|1590167}} (shipped in Firefox 72).
* Encoding detector: {{bug|1551276}} (shipped in Firefox 73)
* Encoding detector: {{bug|1551276}} (shipped in Firefox 73)
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer, potentially parallelizable with Rayon.
** '''Why Rust?''' Builds upon encoding_rs, has tiny FFI surface, subject matter prone to accesses past the bounds of a buffer, potentially parallelizable with Rayon.
* [https://github.com/mozilla/dogear/ Dogear] a bookmark merger for Sync. Shipped pref'd-off in {{bug|1482608}} (Firefox 68), enabled by default in {{bug|1588005}} (Firefox 72)
* Integrate [https://github.com/projectfluent/fluent-rs fluent-rs], a localization system: {{bug|1560038}} (shipped in Firefox 76)
** '''Why Rust?''' A single performant and safe implementation which is shared between desktop and the bookmarks engine in [https://github.com/mozilla/application-services/tree/master/components/places application-services]
** '''Why Rust?''' Performance and memory wins are substantial over previous JS implementation. It brings zero-copy parsing, and memory savvy resolving of localization strings. It also paves the way for migrating the rest of the Fluent APIs away from JS which is required for Fission.
* qcms [https://searchfox.org/mozilla-central/source/gfx/qcms] ported from C to Rust
** '''Why Rust?''' Memory safety


=== In progress ===
=== In progress ===


* Integrate [https://github.com/projectfluent/fluent-rs fluent-rs], a localization system: {{bug|1560038}}
* Port [https://github.com/projectfluent/fluent-rs Localization API] to Rust: {{bug|1613705}}
* [https://github.com/mozilla/neqo neqo] A QUIC implentation.
* [https://github.com/mozilla/neqo neqo] A QUIC implentation.
* [https://github.com/CraneStation/cranelift/ cranelift, a low-level retargetable code generator]: {{bug|1469027}}
* [https://github.com/CraneStation/cranelift/ cranelift, a low-level retargetable code generator]: {{bug|1469027}}
** '''Why Rust?''' It's a new, well-separated component with a clear interface. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
** '''Why Rust?''' It's a new, well-separated component with a clear interface. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
* [https://github.com/mozilla-spidermonkey/rust-frontend Project Visage] A Rust front-end for SpiderMonkey (featuring [https://github.com/mozilla-spidermonkey/jsparagus jsparagus]).
* [https://github.com/mozilla-spidermonkey/rust-frontend SmooshMonkey] A Rust front-end for SpiderMonkey (featuring [https://github.com/mozilla-spidermonkey/jsparagus jsparagus]).
** '''Why Rust?''' Parses untrusted input. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
** '''Why Rust?''' Parses untrusted input. Also, Rust is a great language for writing compilers, due to algebraic data types and pattern matching.
* Audio remoting for Windows: {{bug|1432303}}
* Audio remoting for Windows: {{bug|1432303}}
* Audio remoting for Mac OS: {{bug|1425788}}
* Audio remoting for Mac OS: {{bug|1425788}}
* SDP parsing in WebRTC: {{bug|1365792}}
* SDP parsing in WebRTC: {{bug|1365792}}
** '''Why Rust?''' SDP is a complex text protocol and the existing parser in C has a history of security issues.
** '''Why Rust?''' SDP is a complex text protocol and the existing parser in C has a history of security issues. This also allows us to tailor the SDP parser specifically for the subset used in WebRTC, further reducing its surface area. It is currently run in parallel with the C parser in Nightly.
* Linebreaking with xi-unicode: {{bug|1290022}} (last update late 2016)
* Linebreaking with xi-unicode: {{bug|1290022}} (last update late 2016)
* Background Update Agent for Windows: {{bug|1343669}}
* Background Update Agent for Windows: {{bug|1343669}}
* Replace the XML parser: {{bug|1611289}}
* [https://github.com/gfx-rs/wgpu wgpu], a [https://gpuweb.github.io/gpuweb/ WebGPU] API implementation: {{bug|webgpu-mvp}} (in Nightly since 72)
** '''Why Rust?''' Parses untrusted input, replaces expat, a 3rd-party library with a history of frequent security vulnerabilities.
** '''Why Rust?''' Complex tracking logic, wide attack area. Also, leverages Rust ecosystem for building libraries on top of our native implementation and the API that will target the Web.
* Integrate the [https://github.com/mozilla/glean/ Glean SDK], a data collection library.


=== Proposed ===
=== Proposed ===


* Parallel layout
** '''Why Rust?''' Existing code from Servo, parallel performance.
* WebMIDI: {{bug|1201593}}, {{bug|1201596}}, {{bug|1201598}}
* WebMIDI: {{bug|1201593}}, {{bug|1201596}}, {{bug|1201598}}
* Gamepad code: {{bug|1286699}}
* Gamepad code: {{bug|1286699}}
Line 167: Line 171:
* Replace DOM serializers (XML, HTML for Save As.., plain text)
* Replace DOM serializers (XML, HTML for Save As.., plain text)
** '''Why Rust?''' Need a rewrite anyway. Minor history of security vulnerabilities.
** '''Why Rust?''' Need a rewrite anyway. Minor history of security vulnerabilities.
* Image decoders?
** '''Why Rust?''' Parsing untrusted input, some history of security vulnerabilities.
* Expose Rust API to JS Debugger: {{bug|1263317}}
* Expose Rust API to JS Debugger: {{bug|1263317}}
* Generate Rust bindings for IPDL actors ({{bug|1379739}})
* Generate Rust bindings for IPDL actors ({{bug|1379739}})
Line 178: Line 180:
* [https://github.com/mozilla/application-services Sync/FxA components]
* [https://github.com/mozilla/application-services Sync/FxA components]
** '''Why Rust?''' Single safe and performant implementation which is shared across all our products.
** '''Why Rust?''' Single safe and performant implementation which is shared across all our products.
* libcubeb Audio backend for Windows (WASAPI)
* Replace the XML parser, possibly via c2rust: {{bug|1611289}}
** '''Why Rust?''' Parses untrusted input, replaces expat, a 3rd-party library with a history of frequent security vulnerabilities.
* Rewrite the ICE stack used by WebRTC; {{bug|1616966}}
** '''Why Rust?''' Works with network data.


== Outside Firefox ==
== Outside Firefox ==
Line 190: Line 197:
** Parts of [https://github.com/mozsearch/mozsearch mozsearch], the backend for the [http://searchfox.org Searchfox] code indexing tool.
** Parts of [https://github.com/mozsearch/mozsearch mozsearch], the backend for the [http://searchfox.org Searchfox] code indexing tool.
** [https://github.com/luser/rust-makecab makecab], a reimplementation of Microsoft's makecab tool. Used to compress PDB files before uploading to symbol server in Firefox CI.
** [https://github.com/luser/rust-makecab makecab], a reimplementation of Microsoft's makecab tool. Used to compress PDB files before uploading to symbol server in Firefox CI.
** [https://github.com/mozilla/dump_syms/ dump_syms], a reimplementation of Google Breakpad's dump_syms tool for Windows. Used to parse PDB files and print out Breakpad-compatible symbol files.
*** '''Why Rust?''' Doesn't rely on Microsoft closed-source libraries anymore, can be cross-compiled and run on a non-Windows host, is an order of magnitude faster, takes less than a third of the memory, produces better symbols and relies on an active and friendly upstream [https://github.com/getsentry/symbolic/ symbolic]
* Application Services, server-side
* Application Services, server-side
** [https://github.com/mozilla-services/autopush-rs autopush-rs] Rust async based websocket server that implements Mozilla's push/webpush/broadcast protocols.
** [https://github.com/mozilla-services/autopush-rs autopush-rs] Rust async based websocket server that implements Mozilla's push/webpush/broadcast protocols.
Line 198: Line 207:
* Application Services, client-side
* Application Services, client-side
** [https://github.com/mozilla/application-services/tree/master/components various sync-related components used on iOS and Fenix], includes a cross-compiled FxA Rust client, and storage/syncing of bookmarks, history, logins, tabs and webextensions data.
** [https://github.com/mozilla/application-services/tree/master/components various sync-related components used on iOS and Fenix], includes a cross-compiled FxA Rust client, and storage/syncing of bookmarks, history, logins, tabs and webextensions data.
* [https://github.com/mozilla/fix-stacks/ fix-stacks], a stack frame symbolizer: {{bug|1596292}}
** '''Why Rust?''' High performance needed, a single implementation can replace multiple platform-specific scripts, and we can use the [https://github.com/getsentry/symbolic/ symbolic] crate to do all the hard parts.


=== In Progress ===
=== In Progress ===
Line 208: Line 219:
This section lists areas where Rust integration could be improved.
This section lists areas where Rust integration could be improved.
* Tracking bug: Make the developer experience for Firefox + Rust great: {{Bug|rust-great}}
* Tracking bug: Make the developer experience for Firefox + Rust great: {{Bug|rust-great}}
* Compile speed and memory usage
* [https://docs.google.com/document/d/16FgQPRxNb-Z6sfJy_P7edXzMVWE86jJEZVbmV0oC-m0/ 2020 Questionnaire results]
** Incremental compilation ([https://github.com/rust-lang/rust/labels/A-incr-comp A-incr-comp issues], [https://github.com/rust-lang/rust/labels/WG-compiler-incr WG-compiler-incr issues])
* [https://docs.google.com/document/d/1puZvhWaURtViz8OC0HkB0h2dqJThVyAgLGFbIQpd4fo/ Oxidation 2020 Plan]
** [https://users.rust-lang.org/t/contract-opportunity-mozilla-distributed-compilation-cache-written-in-rust/13898 Distributed compilation cache]
** [https://github.com/rust-lang/cargo/issues/1997 Artifact caching]?
* Debugging: improve gdb and lldb support for Rust. The first step is to establish Rust language support in DWARF distinct from the existing C++ support.
* Bindings/interop
** Immature rust-bindgen and cbindgen tools for general cross-language support. Working aroudn clang bugs in different versions and on different platforms can be tricky.
** No IPDL binding generator ({{bug|1379739}})
** No WebIDL binding generator for DOM components (Servo must have something here?)
* Remaining minor crash report issues {{bug|1348896}}
* IDE/symbol lookup support?
* Code coverage?
* Profiling improvements? Especially for parallel code
* Test integration?


= Meetings =
= Meetings =


* Berlin, January 2020 (minutes lost due to technical issues, unfortunately)
* [https://github.com/servo/servo/wiki/Whistler-Oxidation-2019 Whistler, Jun 2019]
* [https://github.com/servo/servo/wiki/Whistler-Oxidation-2019 Whistler, Jun 2019]
* [https://github.com/servo/servo/wiki/Orlando-Oxidation-2018 Orlando, Dec 2018]
* [https://github.com/servo/servo/wiki/Orlando-Oxidation-2018 Orlando, Dec 2018]
Jrmuizel
Confirmed users
523

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1223938...1232260"