MozillaWiki

Media/WebRTC/libwebrtc Update Process/automation plan: Difference between revisions

Page Discussion

Media/WebRTC/libwebrtc Update Process/automation plan (view source)

Revision as of 22:24, 19 December 2022

3,701 bytes added ,  19 December 2022
Add note at top to redirect users to the fast-forward checklist steps, and remove the checklist and resetting elm sections
(add info on commit message example)
(Add note at top to redirect users to the fast-forward checklist steps, and remove the checklist and resetting elm sections)
 
(20 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<big><span style="color: red">Note: If you are trying to do a fast-forward update please go</span> [[Media/WebRTC/libwebrtc_Update_Process|here]].
</big><br />
<big><span style="color: red">Note: If you are trying to do a fast-forward update please go</span> [[Media/WebRTC/libwebrtc_Update_Process|here]].
</big><br />
<big><span style="color: red">Note: If you are trying to do a fast-forward update please go</span> [[Media/WebRTC/libwebrtc_Update_Process|here]].
</big>
This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome.  This page will be updated as new information is discovered.
This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome.  This page will be updated as new information is discovered.


Line 109: Line 117:


=== build instructions for moz-libwebrtc ===
=== build instructions for moz-libwebrtc ===
Note: These instructions do not currently work for Ubuntu 22.04, but are confirmed to work on 20.04.
* for a freshly built Ubuntu machine, you'll need to do these prerequisite steps:
sudo apt update
sudo apt install python2
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
python2 get-pip.py
* verify build for [https://webrtc.googlesource.com/src webrtc.org]
* verify build for [https://webrtc.googlesource.com/src webrtc.org]
:- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync
:- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
  export DEPOT_TOOLS=`pwd`/depot_tools
  export DEPOT_TOOLS=`pwd`/depot_tools
  (cd depot_tools ; git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  (cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  export PATH=$DEPOT_TOOLS:$PATH ; \
  export PATH=$DEPOT_TOOLS:$PATH ; \
   export DEPOT_TOOLS_UPDATE=0 ; \
   export DEPOT_TOOLS_UPDATE=0 ; \
Line 120: Line 135:
  cd webrtc-checkout
  cd webrtc-checkout
  fetch --nohooks webrtc # populates .gclient and .gclient_entries
  fetch --nohooks webrtc # populates .gclient and .gclient_entries
  (cd src ; git checkout edd7f9e94d)
  # on fresh Ubuntu install, do:
  ./src/build/install-build-deps.sh
  (cd src && git checkout edd7f9e94d)
  gclient sync
  gclient sync
  cd src
  cd src
Line 131: Line 148:
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
  export DEPOT_TOOLS=`pwd`/depot_tools
  export DEPOT_TOOLS=`pwd`/depot_tools
  (cd depot_tools ; git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  (cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
  export PATH=$DEPOT_TOOLS:$PATH ; \
  export PATH=$DEPOT_TOOLS:$PATH ; \
   export DEPOT_TOOLS_UPDATE=0 ; \
   export DEPOT_TOOLS_UPDATE=0 ; \
Line 216: Line 233:
* Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?
* Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?


== Operation Checklist ==
== Moving third_party/libwebrtc/third_party/* to third_party/* ==
* Prerequisites
=== Motivation ===
hg clone https://hg.mozilla.org/projects/elm
(lib)webrtc and Firefox share a number of dependencies. Both projects expect that their dependencies will live under a `third_party` in their respective project roots. As a dependency of libwebrtc, this presents an organizational problem. Additionally there are libraries that are expected to exist in the deployment environments. This can pose a problem where Firefox supports systems with older versions of those libraries than libwebrtc does. Resolving this cluster of issues would directly impact a number of our WebRTC team Q4 goals.
(cd elm && ./mach bootstrap --application-choice=browser && ./mach build)
* Reduce the number of libwebrtc patches, with an emphasis on build patches
git clone https://github.com/mozilla/libwebrtc moz-libwebrtc
* Reduce the merge process complexity
 
* Reduce our response time to security issues
* Update steps
* Uplifting patches
# Start from the elm checkout
cd elm
# Prepare the github repo
bash dom/media/webrtc/third_party_build/prep_repo.sh
# Run loop-ff.sh
bash bash dom/media/webrtc/third_party_build/loop-ff.sh


* Operational notes
Additionally, resolving these issues would assist with broader product goals.
- Builds are done after vendoring in each commit.  Having a .mozconfig file that enables cache (either sccache or ccache) will improve the performance of the scripts.
* Improve performance
* Reduce build time


- Running <code>prep-repo.sh</code> with a fresh github clone of moz-libwebrtc will likely display instructions on how to add recent moz-central changes made in <code>third_party/libwebrtc</code> to the top of the patch stack in github.  This is expected and necessary for successfully vendoring changes into <code>third_party/libwebrtc</code>.
==== Patch Reduction ====
We carry a number of libwebrtc build system patches. These patches create numerous merge conflicts each merge cycle. Problematically, Google is unable to perform CI builds with our [https://searchfox.org/mozilla-central/rev/7b5c93295ea5071d3cc5b3ac21c6e8e654f69e1b/third_party/libwebrtc/BUILD.gn#148 build_with_mozilla flag]. They have expressed interest in removing this flag. In order to remove this flag entirely we need to eliminate our build patches. We could then express our build as a series of feature flags within their build system. Perhaps they would be willing to run that flag set in their CI.


- Two main types of errors will cause <code>loop-ff.sh</code> to exit; the first is a rebase conflict, the second is a build error due to api changes in upstream code.
==== Merge Complexity ====
Updating the dependencies is tied directly to the merge process. Decoupling can be achieved by moving the dependencies into `third_party` and placing them under control of `updatebot`. This would create a smaller merge with less risk.


- When making changes to fix build issues in moz-libwebrtc, mercurial commit messages follow the pattern:
=== Approach ===
Bug xxx - (fix-{upstream-sha}) {description}
We will need to employ several strategies in order to create a robust build environment.
# We will vendor the libraries that libwebrtc requires, but are not provided on our older supported platforms. These dependencies can be kept up to date with `updatebot`.
# We will move vendored shared dependencies from `libwebrtc/third_party` into `third_party`.
As simple as this plan sounds it is complicated by mixed build systems and the current method of importing dependencies into libwebrtc. Eliminating duplicate dependencies also has the risk of running into incompatible version expectations by dependency consumers.


- When making changes to fix Mozilla code in response to changes upstream, mercurial commit messages follow the pattern:
=== Notes ===
Bug xxx (MOZ) - {description}
# (lib)webrtc dependency pinning
#* https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/build/install-build-deps.sh#110
#* https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/build/linux/sysroot_scripts/generated_package_lists
# Chromium Dependencies
#* CPID https://chromium.googlesource.com/infra/luci/luci-go/+/master/cipd/
#* Adding a New Dependency on Linux https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/linux/dependencies.md
# gclient setup
#* https://pypi.org/project/gclient/
#* https://www.chromium.org/developers/how-tos/depottools/
#* https://chromium.googlesource.com/chromium/tools/depot_tools/+/1b8211ff13f933697b9666f8a77c11ddaf43595a/gclient.py#296
# Nika's bazel hack
#* https://bugzilla.mozilla.org/show_bug.cgi?id=1788569#c4
# (lib)drm
# libgmb
# abseil-cpp
# libyuv
# libvpx
# libdav1d
# libaom
# Linux shared library intricacies
# dlopen
#* Use in FireFox https://searchfox.org/mozilla-central/search?q=dlopen%28&path=&case=true&regexp=false
#* Use in Chromium https://source.chromium.org/search?q=dlopen
#* Loading pipewire https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/modules/desktop_capture/linux/wayland/moz_base_capturer_pipewire.cc#32
#*
#* https://linux.die.net/man/3/dlopen
# Ubuntu 18.04 LTS support
#* https://searchfox.org/mozilla-central/source/third_party/libwebrtc/build/install-build-deps.sh#110
# RedHat modifications
# Vendoring headers
# Finding versions of loaded libraries
#* https://wiki.gentoo.org/wiki/Hardened/PaX_Utilities
#* https://manpages.debian.org/testing/pax-utils/scanelf.1.en.html
#* https://stackoverflow.com/questions/50303305/elf-file-type-et-exec-and-et-dyn
Mfroman
119

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1243990...1244951"