Media/WebRTC/libwebrtc Update Process/automation plan: Difference between revisions

Jump to navigation Jump to search

Media/WebRTC/libwebrtc Update Process/automation plan (view source)

Revision as of 22:24, 19 December 2022

8,532 bytes added ,  19 December 2022
Add note at top to redirect users to the fast-forward checklist steps, and remove the checklist and resetting elm sections
(instructions for patches on top of upstream github moz-libwebrtc commit)
(Add note at top to redirect users to the fast-forward checklist steps, and remove the checklist and resetting elm sections)
 
(41 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<big><span style="color: red">Note: If you are trying to do a fast-forward update please go</span> [[Media/WebRTC/libwebrtc_Update_Process|here]].
</big><br />
<big><span style="color: red">Note: If you are trying to do a fast-forward update please go</span> [[Media/WebRTC/libwebrtc_Update_Process|here]].
</big><br />
<big><span style="color: red">Note: If you are trying to do a fast-forward update please go</span> [[Media/WebRTC/libwebrtc_Update_Process|here]].
</big>
This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome.  This page will be updated as new information is discovered.
This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome.  This page will be updated as new information is discovered.


Line 19: Line 27:
* [[#build instructions for moz-libwebrtc|verify build instructions for moz-libwebrtc]]
* [[#build instructions for moz-libwebrtc|verify build instructions for moz-libwebrtc]]
* [[#remove dead/unused files|remove dead/unused files in third_party/libwebrtc]]
* [[#remove dead/unused files|remove dead/unused files in third_party/libwebrtc]]
* identify patch set for changes after vendored moz-libwebrtc (587c40771588)
* [[#patches after vendoring moz-libwebrtc into moz-central|identify patch set for changes after vendored moz-libwebrtc (587c40771588)]]
* identify patch set for changes to upstream github revision (edd7f9e94d)
* [[#patches on top of upstream github moz-libwebrtc commit|identify patch set for changes to upstream github revision (edd7f9e94d)]]
* Manually identify how to fast-forward one changeset
* Manually identify how to fast-forward one changeset
:- move one changeset forward in webrtc.org history
:- move one changeset forward in webrtc.org history
Line 29: Line 37:
:- full try run
:- full try run
* write automation scripts "Manually identify how to fast-forward one changeset"
* write automation scripts "Manually identify how to fast-forward one changeset"
* investigate [https://github.com/mozilla-services/updatebot/#for-mozilla-developers updatebot]


== Findings ==
== Findings ==
Line 87: Line 96:
=== verify current vendoring scripts cmd-line params ===
=== verify current vendoring scripts cmd-line params ===
* Vendor libwebrtc (a93c2d1df066)
* Vendor libwebrtc (a93c2d1df066)
:- The following commands produce a93c2d1df066 with some changes that were checked in on the same commit.  We'll need to store a local patch to be applied.
:- The following commands produce a93c2d1df066 with some changes that were checked in on the same commit.  After running the command below, apply the following patch to match the moz-central commit: [[File:Vendor-fixup.patch.zip]]
  cd dom/media/webrtc/third_party_build
  cd dom/media/webrtc/third_party_build
  python3 vendor-libwebrtc.py \
  python3 vendor-libwebrtc.py \
Line 100: Line 109:
         third_party
         third_party
* Vendor libwebrtc/build (587c40771588)
* Vendor libwebrtc/build (587c40771588)
:- The following commands produce 587c40771588 with some changes that were checked in on the same commit.  We'll need to store a local patch to be applied.
:- The following commands produce 587c40771588 with some changes that were checked in on the same commit.  After running the command below, apply the following patch to match the moz-central commit: [[File:Vendor-build-fixup.patch.zip]]
  cd dom/media/webrtc/third_party_build
  cd dom/media/webrtc/third_party_build
  python3 vendor-libwebrtc.py \
  python3 vendor-libwebrtc.py \
Line 108: Line 117:


=== build instructions for moz-libwebrtc ===
=== build instructions for moz-libwebrtc ===
* verify build for webrtc.org
Note: These instructions do not currently work for Ubuntu 22.04, but are confirmed to work on 20.04.
* for a freshly built Ubuntu machine, you'll need to do these prerequisite steps:
sudo apt update
sudo apt install python2
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
python2 get-pip.py
 
* verify build for [https://webrtc.googlesource.com/src webrtc.org]
:- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync
:- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
  export DEPOT_TOOLS=`pwd`/depot_tools
  export DEPOT_TOOLS=`pwd`/depot_tools
  (cd depot_tools ; git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  (cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
  mkdir webrtc-checkout
  mkdir webrtc-checkout
  cd webrtc-checkout
  cd webrtc-checkout
  fetch --nohooks webrtc # populates .gclient and .gclient_entries
  fetch --nohooks webrtc # populates .gclient and .gclient_entries
  (cd src ; git checkout edd7f9e94d)
  # on fresh Ubuntu install, do:
export PATH=$DEPOT_TOOLS:$PATH ; \
  ./src/build/install-build-deps.sh
  export DEPOT_TOOLS_UPDATE=0 ; \
  (cd src && git checkout edd7f9e94d)
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
  gclient sync
  gclient sync
  cd src
  cd src
Line 130: Line 148:
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
  export DEPOT_TOOLS=`pwd`/depot_tools
  export DEPOT_TOOLS=`pwd`/depot_tools
  (cd depot_tools ; git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  (cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  export PATH=$DEPOT_TOOLS:$PATH ; \
  export PATH=$DEPOT_TOOLS:$PATH ; \
   export DEPOT_TOOLS_UPDATE=0 ; \
   export DEPOT_TOOLS_UPDATE=0 ; \
Line 167: Line 185:
     --exclude 'third_party/libwebrtc/third_party' \
     --exclude 'third_party/libwebrtc/third_party' \
     third_party/libwebrtc
     third_party/libwebrtc
:- While the above command works, when used within the fast foward script it requires reapplication of fixes for merge conflicts with each new advance in the webrtc.org changesets.  It makes more sense to reuse the [https://searchfox.org/mozilla-central/rev/d4ebb53e719b913afdbcf7c00e162f0e96574701/dom/media/webrtc/third_party_build/extract-for-git.py extract-for-git.py] script to build a patch set of changes we can apply on top of our current patch stack residing in [https://github.com/mozilla/libwebrtc moz-libwebrtc].  That command looks like:
cd dom/media/webrtc/third_party_build && \
python3 extract-for-git.py 587c40771588::2b187d932ca1
:- where <code>587c40771588</code> is the moz-central vendoring commit and <code>2b187d932ca1</code> is the current tip of moz-central
:- extract-for-git produces a file named <code>mailbox.patch</code> which can be applied in moz-libwebrtc with:
git am mailbox.patch
:- Note, fixes were required during the <code>git am</code> process.  These were dealing with a couple places were we imported directories into moz-central from webrtc.org after the vendoring commit, and a couple cases of Windows vs Linux line endings causing conflicts.


=== patches on top of upstream github moz-libwebrtc commit ===
=== patches on top of upstream github moz-libwebrtc commit ===
Line 182: Line 207:
  curl -o vendor-fixup.patch.zip https://wiki.mozilla.org/images/2/29/Vendor-fixup.patch.zip
  curl -o vendor-fixup.patch.zip https://wiki.mozilla.org/images/2/29/Vendor-fixup.patch.zip
  unzip vendor-fixup.patch.zip
  unzip vendor-fixup.patch.zip
  patch -p1 < vendor-fixup.patch
  (cd third_party/libwebrtc && patch -p1 < vendor-fixup.patch)
 
== Keeping transitive dependencies updated ==
* we will use updatebot to keep our dependencies updated. This may happen after an initial manual update.
* [[Media/WebRTC/dependency-libyuv|libyuv]]
* [[Media/WebRTC/dependency-libvpx|libvpx]]
* [[Media/WebRTC/dependency-pipewire|pipewire]]
 
== Open Questions ==
* How to see release branches in webrtc.org?
  Add "fetch = +refs/branch-heads/*:refs/remotes/branch-heads/*" to .git/config and run 'git pull'.
* How to find versions of webrtc.org that ship with Chrome?
  If you have acquired webrtc.org/src using gclient sync ([https://www.chromium.org/developers/how-tos/get-the-code/working-with-release-branches/ instructions]), there are branches listed that look like 'branch-heads/nnnn'.  You can cross-reference the Chrome release and find the webrtc.org release here: https://chromiumdash.appspot.com/branches
  example: Chrome's 98 release uses webrtc.org 4758, which is sha a6b138d6b4
* Do we want to advance commit-by-commit?
  I think we may need to do this to avoid large-scale breakage and
  make it easier to fix issues.
* Do we want to manually advance the process as quickly as possible, and then modify our scripts to start taking advantage of update-bot?
  This allows a more finely tuned process to help us get caught up,
  but pushes out the integration with update-bot.
* How/when do we start integrating webrtc.org's unittests into our try runs?
  This may need to happen in our tree (probably post-vendoring) because
  the process of getting the webrtc tree buildable is a long process
  requiring lots of network.
* Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?
 
== Moving third_party/libwebrtc/third_party/* to third_party/* ==
=== Motivation ===
(lib)webrtc and Firefox share a number of dependencies. Both projects expect that their dependencies will live under a `third_party` in their respective project roots. As a dependency of libwebrtc, this presents an organizational problem. Additionally there are libraries that are expected to exist in the deployment environments. This can pose a problem where Firefox supports systems with older versions of those libraries than libwebrtc does. Resolving this cluster of issues would directly impact a number of our WebRTC team Q4 goals.
* Reduce the number of libwebrtc patches, with an emphasis on build patches
* Reduce the merge process complexity
* Reduce our response time to security issues
* Uplifting patches
 
Additionally, resolving these issues would assist with broader product goals.
* Improve performance
* Reduce build time
 
==== Patch Reduction ====
We carry a number of libwebrtc build system patches. These patches create numerous merge conflicts each merge cycle. Problematically, Google is unable to perform CI builds with our [https://searchfox.org/mozilla-central/rev/7b5c93295ea5071d3cc5b3ac21c6e8e654f69e1b/third_party/libwebrtc/BUILD.gn#148 build_with_mozilla flag]. They have expressed interest in removing this flag. In order to remove this flag entirely we need to eliminate our build patches. We could then express our build as a series of feature flags within their build system. Perhaps they would be willing to run that flag set in their CI.
 
==== Merge Complexity ====
Updating the dependencies is tied directly to the merge process. Decoupling can be achieved by moving the dependencies into `third_party` and placing them under control of `updatebot`. This would create a smaller merge with less risk.
 
=== Approach ===
We will need to employ several strategies in order to create a robust build environment.
# We will vendor the libraries that libwebrtc requires, but are not provided on our older supported platforms. These dependencies can be kept up to date with `updatebot`.
# We will move vendored shared dependencies from `libwebrtc/third_party` into `third_party`.
As simple as this plan sounds it is complicated by mixed build systems and the current method of importing dependencies into libwebrtc. Eliminating duplicate dependencies also has the risk of running into incompatible version expectations by dependency consumers.
 
=== Notes ===
# (lib)webrtc dependency pinning
#* https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/build/install-build-deps.sh#110
#* https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/build/linux/sysroot_scripts/generated_package_lists
# Chromium Dependencies
#* CPID https://chromium.googlesource.com/infra/luci/luci-go/+/master/cipd/
#* Adding a New Dependency on Linux https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/linux/dependencies.md
# gclient setup
#* https://pypi.org/project/gclient/
#* https://www.chromium.org/developers/how-tos/depottools/
#* https://chromium.googlesource.com/chromium/tools/depot_tools/+/1b8211ff13f933697b9666f8a77c11ddaf43595a/gclient.py#296
# Nika's bazel hack
#* https://bugzilla.mozilla.org/show_bug.cgi?id=1788569#c4
# (lib)drm
# libgmb
# abseil-cpp
# libyuv
# libvpx
# libdav1d
# libaom
# Linux shared library intricacies
# dlopen
#* Use in FireFox https://searchfox.org/mozilla-central/search?q=dlopen%28&path=&case=true&regexp=false
#* Use in Chromium https://source.chromium.org/search?q=dlopen
#* Loading pipewire https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/modules/desktop_capture/linux/wayland/moz_base_capturer_pipewire.cc#32
#*
#* https://linux.die.net/man/3/dlopen
# Ubuntu 18.04 LTS support
#* https://searchfox.org/mozilla-central/source/third_party/libwebrtc/build/install-build-deps.sh#110
# RedHat modifications
# Vendoring headers
# Finding versions of loaded libraries
#* https://wiki.gentoo.org/wiki/Hardened/PaX_Utilities
#* https://manpages.debian.org/testing/pax-utils/scanelf.1.en.html
#* https://stackoverflow.com/questions/50303305/elf-file-type-et-exec-and-et-dyn
Mfroman
119

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1240694...1244951"

Navigation menu