Media/WebRTC/libwebrtc Update Process/automation plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Merge complexity)
(remove redudant 'bash' in loop-ff instructions)
Line 242: Line 242:
   
   
  # Run loop-ff.sh
  # Run loop-ff.sh
  bash bash dom/media/webrtc/third_party_build/loop-ff.sh
  bash dom/media/webrtc/third_party_build/loop-ff.sh


* Operational notes
* Operational notes

Revision as of 19:03, 11 November 2022

This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome. This page will be updated as new information is discovered.

Plan Overview

  • repo tasks
- update moz-libwebrtc fork to latest webrtc.org
git checkout master
git remote add upstream https://webrtc.googlesource.com/src
git fetch upstream
git merge upstream/master # fast forward merge
git push origin master
- update elm to latest moz-central
- check moz-central a93c2d1df066 - Bug 1654112 - Vendor libwebrtc for rel86
- check moz-central d1df4970f4f0 - Bug 1654112 - Vendor libwebrtc/third_party for rel86
- check moz-central 587c40771588 - Bug 1654112 - Vendor libwebrtc/build for rel86
- move one changeset forward in webrtc.org history
- verify local build in moz-libwebrtc
- apply moz-libwebrtc-to-upstream patches, verify local build
- vendor moz-libwebrtc
- apply moz-central patches
- full try run
  • write automation scripts "Manually identify how to fast-forward one changeset"
  • investigate updatebot

Findings

current third_party/libwebrtc directories and status

  • Additional vendoring
build (with some changes during vendoring)
third_party
  • Mozilla added
README.mozilla - generated during vendoring process
moz.build - generated by generate-gn-build-files.sh
webrtc_gn - generated by generate-gn-build-files.sh
X11 - imported later by Byron in c91f12b557a1 D113830
tools/grit - imported later by Nico in 3cce5e6938f0 D114027
  • Mozilla vendored after a93c2d1df066
- Both of the following were inadvertently vendored from moz-libwebrtc commit bd4a718667. The only meaningful difference is in sdk/objc/helpers/RTCDispatcher.{h|m}, but these files are not currently used.
sdk/BUILD.gn - imported later by Byron in 9314046d89eb D105015
sdk/objc - imported later by Byron in 9314046d89eb D105015
- We don't build the sdk directory (see here).
- Only sdk:helpers_objc is used (see here)
- From sdk/objc/helpers, only sdk/objc/helpers/scoped_cftyperef.h is used:
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_iosurface.h
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_provider.h
third_party/libwebrtc/modules/desktop_capture/mac/screen_capturer_mac.mm
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_cgimage.h
  • Untouched from Bug 1665166 - Move media/webrtc/trunk/* to third-party/libwebrtc
README.md   (should have been vendored)
DEPS        (should have been vendored)
google_apis (is this referenced/used by current code?)
tools/clang (is this referenced/used by current code?)            
chromium_deps                   
dummy_file.txt                  
net                             
Makefile.old                    
peerconnection_client.target.mk 
peerconnection.Makefile

verify generation/build with extra, unused directories

  • Add missing directories
cp -r ~/git-checkouts/moz-libwebrtc/test ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/pc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/resources ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/docs ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/tools_webrtc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/examples ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/p2p ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/rtc_tools ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/data ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/style-guide ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/stats ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/media/sctp ~/mozilla/moz-central/third_party/libwebrtc/media
  • local build file generation produces no build modified build files
  • Full build results here using ./mach try fuzzy -q 'build-'
  • Full linux test results here using ./mach try fuzzy -q 'linux-'

verify current vendoring scripts cmd-line params

  • Vendor libwebrtc (a93c2d1df066)
- The following commands produce a93c2d1df066 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-fixup.patch.zip
cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-github https://github.com/mozilla/libwebrtc \
        --commit 149d693483e9055f574d9d65b01fe75a186b654b \
        libwebrtc
  • Vendor libwebrtc/third_party (d1df4970f4f0)
cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-googlesource \
        --commit 5dc5a4a45df9592baa8e8c5f896006d9193d8e45 \
        third_party
  • Vendor libwebrtc/build (587c40771588)
- The following commands produce 587c40771588 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-build-fixup.patch.zip
cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-googlesource \
        --commit ac22062b465485d3e1196ae9c506b3617cc16fb5 \
        build

build instructions for moz-libwebrtc

Note: These instructions do not currently work for Ubuntu 22.04, but are confirmed to work on 20.04.

  • for a freshly built Ubuntu machine, you'll need to do these prerequisite steps:
sudo apt update
sudo apt install python2
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
python2 get-pip.py
- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
mkdir webrtc-checkout
cd webrtc-checkout
fetch --nohooks webrtc # populates .gclient and .gclient_entries
  # on fresh Ubuntu install, do:
  ./src/build/install-build-deps.sh
(cd src && git checkout edd7f9e94d)
gclient sync
cd src
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test
  • verify build for moz-libwebrtc
- Use edd7f9e94d because some our patches stacked on top of this upstream revision are specific to building in the moz-central tree.
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
mkdir moz-libwebrtc-checkout
cd moz-libwebrtc-checkout
gclient config https://github.com/mozilla/libwebrtc --name src --unmanaged
gclient sync -D --force --reset --with_branch_heads # ~30 min
cd src
git checkout edd7f9e94d
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test

remove dead/unused files

  • Successfully removed:
chromium_deps
dummy_file.txt
net
Makefile.old
peerconnection_client.target.mk
peerconnection.Makefile
- successful try builds
tools/clang
- successful try builds

patches after vendoring moz-libwebrtc into moz-central

hg diff -r 587c40771588 -r central \
   --exclude '**/moz.build' \
   --exclude 'third_party/libwebrtc/X11' \
   --exclude 'third_party/libwebrtc/sdk/BUILD.gn' \
   --exclude 'third_party/libwebrtc/sdk/objc' \
   --exclude 'third_party/libwebrtc/tools/grit' \
   --exclude 'third_party/libwebrtc/build' \
   --exclude 'third_party/libwebrtc/third_party' \
   third_party/libwebrtc
- While the above command works, when used within the fast foward script it requires reapplication of fixes for merge conflicts with each new advance in the webrtc.org changesets. It makes more sense to reuse the extract-for-git.py script to build a patch set of changes we can apply on top of our current patch stack residing in moz-libwebrtc. That command looks like:
cd dom/media/webrtc/third_party_build && \
python3 extract-for-git.py 587c40771588::2b187d932ca1
- where 587c40771588 is the moz-central vendoring commit and 2b187d932ca1 is the current tip of moz-central
- extract-for-git produces a file named mailbox.patch which can be applied in moz-libwebrtc with:
git am mailbox.patch
- Note, fixes were required during the git am process. These were dealing with a couple places were we imported directories into moz-central from webrtc.org after the vendoring commit, and a couple cases of Windows vs Linux line endings causing conflicts.

patches on top of upstream github moz-libwebrtc commit

  • easiest way
git diff -r edd7f9e94d -r mozilla-modifications-rel86
  • more useful for rebasing on top of upstream commits
git branch mozpatches edd7f9e94d
git checkout mozpatches
git format-patch -k edd7f9e94d..mozilla-modifications-rel86
git am *.patch # applies to branch mozpatches
  • in both cases, due to an anomaly during the previous vendoring operation an additional patch must be applied to match the vendored commit in moz-central

File:Vendor-fixup.patch.zip 

curl -o vendor-fixup.patch.zip https://wiki.mozilla.org/images/2/29/Vendor-fixup.patch.zip
unzip vendor-fixup.patch.zip
(cd third_party/libwebrtc && patch -p1 < vendor-fixup.patch)

Keeping transitive dependencies updated

  • we will use updatebot to keep our dependencies updated. This may happen after an initial manual update.
  • libyuv
  • libvpx
  • pipewire

Open Questions

  • How to see release branches in webrtc.org?
 Add "fetch = +refs/branch-heads/*:refs/remotes/branch-heads/*" to .git/config and run 'git pull'.
  • How to find versions of webrtc.org that ship with Chrome?
 If you have acquired webrtc.org/src using gclient sync (instructions), there are branches listed that look like 'branch-heads/nnnn'.  You can cross-reference the Chrome release and find the webrtc.org release here: https://chromiumdash.appspot.com/branches
 example: Chrome's 98 release uses webrtc.org 4758, which is sha a6b138d6b4
  • Do we want to advance commit-by-commit?
 I think we may need to do this to avoid large-scale breakage and
 make it easier to fix issues.
  • Do we want to manually advance the process as quickly as possible, and then modify our scripts to start taking advantage of update-bot?
 This allows a more finely tuned process to help us get caught up,
 but pushes out the integration with update-bot.
  • How/when do we start integrating webrtc.org's unittests into our try runs?
 This may need to happen in our tree (probably post-vendoring) because
 the process of getting the webrtc tree buildable is a long process
 requiring lots of network.
  • Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?

Operation Checklist

  • Prerequisites

Builds are done after vendoring in each commit. Having a .mozconfig file that enables cache (either sccache or ccache) will improve the performance of the scripts. 

hg clone https://hg.mozilla.org/projects/elm
(cd elm && ./mach bootstrap --application-choice=browser && ./mach build)
git clone https://github.com/mozilla/libwebrtc moz-libwebrtc

On macOS, the python module requests should be installed. 

pip install requests
  • Update steps
# Start from the elm checkout
cd elm

# Prepare the github repo
bash dom/media/webrtc/third_party_build/prep_repo.sh

# Run loop-ff.sh
bash dom/media/webrtc/third_party_build/loop-ff.sh
  • Operational notes

- Running prep-repo.sh with a fresh github clone of moz-libwebrtc will likely display instructions on how to add recent moz-central changes made in third_party/libwebrtc to the top of the patch stack in github. This is expected and necessary for successfully vendoring changes into third_party/libwebrtc.

- Two main types of errors will cause loop-ff.sh to exit; the first is a rebase conflict, the second is a build error due to api changes in upstream code.

- When making changes to fix build issues in moz-libwebrtc, mercurial commit messages follow the pattern: 

Bug xxx - (fix-{upstream-sha}) {description}

- When making changes to fix Mozilla code in response to changes upstream, mercurial commit messages follow the pattern: 

Bug xxx (MOZ) - {description}

- While loop-ff.sh is running, in a separate terminal window it may be helpful to see the overall progress by running: 

tail -f ~/log-loop-ff.txt | grep loop-ff

Moving third_party/libwebrtc/third_party/* to third_party/*

Motivation

(lib)webrtc and Firefox share a number of dependencies. Both projects expect that their dependencies will live under a `third_party` in their respective project roots. As a dependency of libwebrtc, this presents an organizational problem. Additionally there are libraries that are expected to exist in the deployment environments. This can pose a problem where Firefox supports systems with older versions of those libraries than libwebrtc does. Resolving this cluster of issues would directly impact a number of our WebRTC team Q4 goals.

  • Reduce the number of libwebrtc patches, with an emphasis on build patches
  • Reduce the merge process complexity
  • Reduce our response time to security issues
  • Uplifting patches

Additionally, resolving these issues would assist with broader product goals.

  • Improve performance
  • Reduce build time

Patch Reduction

We carry a number of libwebrtc build system patches. These patches create numerous merge conflicts each merge cycle. Problematically, Google is unable to perform CI builds with our build_with_mozilla flag. They have expressed interest in removing this flag. In order to remove this flag entirely we need to eliminate our build patches. We could then express our build as a series of feature flags within their build system. Perhaps they would be willing to run that flag set in their CI.

Merge Complexity

Updating the dependencies is tied directly to the merge process. Decoupling can be achieved by moving the dependencies into `third_party` and placing them under control of `updatebot`. This would create a smaller merge with less risk.

Approach

We will need to employ several strategies in order to create a robust build environment.

  1. We will vendor the libraries that libwebrtc requires, but are not provided on our older supported platforms. These dependencies can be kept up to date with `updatebot`.
  2. We will move vendored shared dependencies from `libwebrtc/third_party` into `third_party`.

As simple as this plan sounds it is complicated by mixed build systems and the current method of importing dependencies into libwebrtc. Eliminating duplicate dependencies also has the risk of running into incompatible version expectations by dependency consumers.

Notes

  1. (lib)webrtc dependency pinning
  2. Chromium Dependencies
  3. gclient setup
  4. Nika's bazel hack
  5. (lib)drm
  6. libgmb
  7. abseil-cpp
  8. libyuv
  9. libvpx
  10. libdav1d
  11. libaom
  12. Linux shared library intricacies
  13. dlopen
  14. Ubuntu 18.04 LTS support
  15. RedHat modifications
  16. Vendoring headers
  17. Finding versions of loaded libraries
Retrieved from "https://wiki.allizom.org/index.php?title=Media/WebRTC/libwebrtc_Update_Process/automation_plan&oldid=1244524"