Revision as of 01:57, 1 December 2022

This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome. This page will be updated as new information is discovered.

Plan Overview

repo tasks

- update moz-libwebrtc fork to latest webrtc.org

git checkout master
git remote add upstream https://webrtc.googlesource.com/src
git fetch upstream
git merge upstream/master # fast forward merge
git push origin master

- update elm to latest moz-central

verify current third_party/libwebrtc directories
verify generation/build with extra, unused directories from stock moz-libwebrtc in place
verify current vendoring scripts cmd-line params

- check moz-central a93c2d1df066 - Bug 1654112 - Vendor libwebrtc for rel86

- check moz-central d1df4970f4f0 - Bug 1654112 - Vendor libwebrtc/third_party for rel86

- check moz-central 587c40771588 - Bug 1654112 - Vendor libwebrtc/build for rel86

verify build instructions for moz-libwebrtc
remove dead/unused files in third_party/libwebrtc
identify patch set for changes after vendored moz-libwebrtc (587c40771588)
identify patch set for changes to upstream github revision (edd7f9e94d)
Manually identify how to fast-forward one changeset

- move one changeset forward in webrtc.org history

- verify local build in moz-libwebrtc

- apply moz-libwebrtc-to-upstream patches, verify local build

- vendor moz-libwebrtc

- apply moz-central patches

- full try run

write automation scripts "Manually identify how to fast-forward one changeset"
investigate updatebot

Findings

current third_party/libwebrtc directories and status

Additional vendoring

build (with some changes during vendoring)
third_party

Mozilla added

README.mozilla - generated during vendoring process
moz.build - generated by generate-gn-build-files.sh
webrtc_gn - generated by generate-gn-build-files.sh
X11 - imported later by Byron in c91f12b557a1 D113830
tools/grit - imported later by Nico in 3cce5e6938f0 D114027

Mozilla vendored after a93c2d1df066

- Both of the following were inadvertently vendored from moz-libwebrtc commit bd4a718667. The only meaningful difference is in sdk/objc/helpers/RTCDispatcher.{h|m}, but these files are not currently used.

sdk/BUILD.gn - imported later by Byron in 9314046d89eb D105015
sdk/objc - imported later by Byron in 9314046d89eb D105015

- We don't build the sdk directory (see here).

- Only sdk:helpers_objc is used (see here)

- From sdk/objc/helpers, only sdk/objc/helpers/scoped_cftyperef.h is used:

third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_iosurface.h
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_provider.h
third_party/libwebrtc/modules/desktop_capture/mac/screen_capturer_mac.mm
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_cgimage.h

Untouched from Bug 1665166 - Move media/webrtc/trunk/* to third-party/libwebrtc

README.md   (should have been vendored)
DEPS        (should have been vendored)
google_apis (is this referenced/used by current code?)
tools/clang (is this referenced/used by current code?)            
chromium_deps                   
dummy_file.txt                  
net                             
Makefile.old                    
peerconnection_client.target.mk 
peerconnection.Makefile

verify generation/build with extra, unused directories

Add missing directories

cp -r ~/git-checkouts/moz-libwebrtc/test ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/pc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/resources ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/docs ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/tools_webrtc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/examples ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/p2p ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/rtc_tools ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/data ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/style-guide ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/stats ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/media/sctp ~/mozilla/moz-central/third_party/libwebrtc/media

local build file generation produces no build modified build files
Full build results here using ./mach try fuzzy -q 'build-'
Full linux test results here using ./mach try fuzzy -q 'linux-'

verify current vendoring scripts cmd-line params

Vendor libwebrtc (a93c2d1df066)

- The following commands produce a93c2d1df066 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-fixup.patch.zip

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-github https://github.com/mozilla/libwebrtc \
        --commit 149d693483e9055f574d9d65b01fe75a186b654b \
        libwebrtc

Vendor libwebrtc/third_party (d1df4970f4f0)

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-googlesource \
        --commit 5dc5a4a45df9592baa8e8c5f896006d9193d8e45 \
        third_party

Vendor libwebrtc/build (587c40771588)

- The following commands produce 587c40771588 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-build-fixup.patch.zip

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-googlesource \
        --commit ac22062b465485d3e1196ae9c506b3617cc16fb5 \
        build

build instructions for moz-libwebrtc

Note: These instructions do not currently work for Ubuntu 22.04, but are confirmed to work on 20.04.

for a freshly built Ubuntu machine, you'll need to do these prerequisite steps:

sudo apt update
sudo apt install python2
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
python2 get-pip.py

verify build for webrtc.org

- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
mkdir webrtc-checkout
cd webrtc-checkout
fetch --nohooks webrtc # populates .gclient and .gclient_entries
  # on fresh Ubuntu install, do:
  ./src/build/install-build-deps.sh
(cd src && git checkout edd7f9e94d)
gclient sync
cd src
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test

verify build for moz-libwebrtc

- Use edd7f9e94d because some our patches stacked on top of this upstream revision are specific to building in the moz-central tree.

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools && git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
mkdir moz-libwebrtc-checkout
cd moz-libwebrtc-checkout
gclient config https://github.com/mozilla/libwebrtc --name src --unmanaged
gclient sync -D --force --reset --with_branch_heads # ~30 min
cd src
git checkout edd7f9e94d
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test

remove dead/unused files

Successfully removed:

chromium_deps
dummy_file.txt
net
Makefile.old
peerconnection_client.target.mk
peerconnection.Makefile

- successful try builds

tools/clang

- successful try builds

patches after vendoring moz-libwebrtc into moz-central

hg diff -r 587c40771588 -r central \
   --exclude '**/moz.build' \
   --exclude 'third_party/libwebrtc/X11' \
   --exclude 'third_party/libwebrtc/sdk/BUILD.gn' \
   --exclude 'third_party/libwebrtc/sdk/objc' \
   --exclude 'third_party/libwebrtc/tools/grit' \
   --exclude 'third_party/libwebrtc/build' \
   --exclude 'third_party/libwebrtc/third_party' \
   third_party/libwebrtc

- While the above command works, when used within the fast foward script it requires reapplication of fixes for merge conflicts with each new advance in the webrtc.org changesets. It makes more sense to reuse the extract-for-git.py script to build a patch set of changes we can apply on top of our current patch stack residing in moz-libwebrtc. That command looks like:

cd dom/media/webrtc/third_party_build && \
python3 extract-for-git.py 587c40771588::2b187d932ca1

- where 587c40771588 is the moz-central vendoring commit and 2b187d932ca1 is the current tip of moz-central

- extract-for-git produces a file named mailbox.patch which can be applied in moz-libwebrtc with:

git am mailbox.patch

- Note, fixes were required during the git am process. These were dealing with a couple places were we imported directories into moz-central from webrtc.org after the vendoring commit, and a couple cases of Windows vs Linux line endings causing conflicts.

patches on top of upstream github moz-libwebrtc commit

easiest way

git diff -r edd7f9e94d -r mozilla-modifications-rel86

more useful for rebasing on top of upstream commits

git branch mozpatches edd7f9e94d
git checkout mozpatches
git format-patch -k edd7f9e94d..mozilla-modifications-rel86
git am *.patch # applies to branch mozpatches

in both cases, due to an anomaly during the previous vendoring operation an additional patch must be applied to match the vendored commit in moz-central

File:Vendor-fixup.patch.zip

curl -o vendor-fixup.patch.zip https://wiki.mozilla.org/images/2/29/Vendor-fixup.patch.zip
unzip vendor-fixup.patch.zip
(cd third_party/libwebrtc && patch -p1 < vendor-fixup.patch)

Keeping transitive dependencies updated

we will use updatebot to keep our dependencies updated. This may happen after an initial manual update.
libyuv
libvpx
pipewire

Open Questions

How to see release branches in webrtc.org?

 Add "fetch = +refs/branch-heads/*:refs/remotes/branch-heads/*" to .git/config and run 'git pull'.

How to find versions of webrtc.org that ship with Chrome?

 If you have acquired webrtc.org/src using gclient sync (instructions), there are branches listed that look like 'branch-heads/nnnn'.  You can cross-reference the Chrome release and find the webrtc.org release here: https://chromiumdash.appspot.com/branches
 example: Chrome's 98 release uses webrtc.org 4758, which is sha a6b138d6b4

Do we want to advance commit-by-commit?

 I think we may need to do this to avoid large-scale breakage and
 make it easier to fix issues.

Do we want to manually advance the process as quickly as possible, and then modify our scripts to start taking advantage of update-bot?

 This allows a more finely tuned process to help us get caught up,
 but pushes out the integration with update-bot.

How/when do we start integrating webrtc.org's unittests into our try runs?

 This may need to happen in our tree (probably post-vendoring) because
 the process of getting the webrtc tree buildable is a long process
 requiring lots of network.

Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?

Operation Checklist

Prerequisites

Builds are done after vendoring in each commit. Having a .mozconfig file that enables cache (either sccache or ccache) will improve the performance of the scripts.

hg clone https://hg.mozilla.org/projects/elm
(cd elm && ./mach bootstrap --application-choice=browser && ./mach build)
git clone https://github.com/mozilla/libwebrtc moz-libwebrtc

On macOS, the python module requests should be installed.

pip install requests

Update steps

# Start from the elm checkout
cd elm

# Prepare the github repo
bash dom/media/webrtc/third_party_build/prep_repo.sh

# Run loop-ff.sh
bash dom/media/webrtc/third_party_build/loop-ff.sh

Operational notes

- Running prep-repo.sh with a fresh github clone of moz-libwebrtc will likely display instructions on how to add recent moz-central changes made in third_party/libwebrtc to the top of the patch stack in github. This is expected and necessary for successfully vendoring changes into third_party/libwebrtc.

- Two main types of errors will cause loop-ff.sh to exit; the first is a rebase conflict, the second is a build error due to api changes in upstream code.

- When making changes to fix build issues in moz-libwebrtc, mercurial commit messages follow the pattern:

Bug xxx - (fix-{upstream-sha}) {description}

- When making changes to fix Mozilla code in response to changes upstream, mercurial commit messages follow the pattern:

Bug xxx (MOZ) - {description}

- While loop-ff.sh is running, in a separate terminal window it may be helpful to see the overall progress by running:

tail -f ~/log-loop-ff.txt | grep loop-ff

Moving third_party/libwebrtc/third_party/* to third_party/*

Motivation

(lib)webrtc and Firefox share a number of dependencies. Both projects expect that their dependencies will live under a `third_party` in their respective project roots. As a dependency of libwebrtc, this presents an organizational problem. Additionally there are libraries that are expected to exist in the deployment environments. This can pose a problem where Firefox supports systems with older versions of those libraries than libwebrtc does. Resolving this cluster of issues would directly impact a number of our WebRTC team Q4 goals.

Reduce the number of libwebrtc patches, with an emphasis on build patches
Reduce the merge process complexity
Reduce our response time to security issues
Uplifting patches

Additionally, resolving these issues would assist with broader product goals.

Improve performance
Reduce build time

Patch Reduction

We carry a number of libwebrtc build system patches. These patches create numerous merge conflicts each merge cycle. Problematically, Google is unable to perform CI builds with our build_with_mozilla flag. They have expressed interest in removing this flag. In order to remove this flag entirely we need to eliminate our build patches. We could then express our build as a series of feature flags within their build system. Perhaps they would be willing to run that flag set in their CI.

Merge Complexity

Updating the dependencies is tied directly to the merge process. Decoupling can be achieved by moving the dependencies into `third_party` and placing them under control of `updatebot`. This would create a smaller merge with less risk.

Approach

We will need to employ several strategies in order to create a robust build environment.

We will vendor the libraries that libwebrtc requires, but are not provided on our older supported platforms. These dependencies can be kept up to date with `updatebot`.
We will move vendored shared dependencies from `libwebrtc/third_party` into `third_party`.

As simple as this plan sounds it is complicated by mixed build systems and the current method of importing dependencies into libwebrtc. Eliminating duplicate dependencies also has the risk of running into incompatible version expectations by dependency consumers.

Notes

(lib)webrtc dependency pinning
- https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/build/install-build-deps.sh#110
- https://searchfox.org/mozilla-central/rev/0a2eba79c24300ce0539f91c1bebac2e75264e58/third_party/libwebrtc/build/linux/sysroot_scripts/generated_package_lists
Chromium Dependencies
- CPID https://chromium.googlesource.com/infra/luci/luci-go/+/master/cipd/
- Adding a New Dependency on Linux https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/linux/dependencies.md
gclient setup
Nika's bazel hack
- https://bugzilla.mozilla.org/show_bug.cgi?id=1788569#c4
(lib)drm
libgmb
abseil-cpp
libyuv
libvpx
libdav1d
libaom
Linux shared library intricacies
dlopen
Ubuntu 18.04 LTS support
- https://searchfox.org/mozilla-central/source/third_party/libwebrtc/build/install-build-deps.sh#110
RedHat modifications
Vendoring headers
Finding versions of loaded libraries

Resetting Elm

Before each new merge cycle. Elm needs to be reset to Central, and the floating patches need to be reapplied.

Ensure you have filed a bug for the next merge cycle, and further ensure that the next merge cycle bug has the previous merge cycle bug as a dependency.
Make sure you have a local checkout of Elm which includes the floating patches.
File a bug as a clone of Bug 1789675 and add your new bug as dependency to the next merge cycle bug.
Wait for the repo admins to reset the repository.
Pull the new head of Elm.
Rebase (--keep), the floating patches onto the new checkout of Elm.
Push the floating patches to Elm.
Celebrate

@@ Line 317: / Line 317: @@
 #* https://manpages.debian.org/testing/pax-utils/scanelf.1.en.html
 #* https://stackoverflow.com/questions/50303305/elf-file-type-et-exec-and-et-dyn
+= Resetting Elm =
+Before each new merge cycle. Elm needs to be reset to Central, and the floating patches need to be reapplied.
+# Ensure you have filed a bug for the next merge cycle, and further ensure that the next merge cycle bug has the previous merge cycle bug as a dependency.
+# Make sure you have a local checkout of Elm which includes the floating patches.
+# File a bug as a clone of [https://bugzilla.mozilla.org/show_bug.cgi?id=1789675 Bug 1789675] and add your new bug as dependency to the next merge cycle bug.
+# Wait for the repo admins to reset the repository.
+# Pull the new head of Elm.
+# Rebase (--keep), the floating patches onto the new checkout of Elm.
+# Push the floating patches to Elm.
+# Celebrate

Media/WebRTC/libwebrtc Update Process/automation plan: Difference between revisions

Revision as of 01:57, 1 December 2022

Contents

Plan Overview

Findings

current third_party/libwebrtc directories and status

verify generation/build with extra, unused directories

verify current vendoring scripts cmd-line params

build instructions for moz-libwebrtc

remove dead/unused files

patches after vendoring moz-libwebrtc into moz-central

patches on top of upstream github moz-libwebrtc commit

Keeping transitive dependencies updated

Open Questions

Operation Checklist

Moving third_party/libwebrtc/third_party/* to third_party/*

Motivation

Patch Reduction

Merge Complexity

Approach

Notes

Resetting Elm

Navigation menu

Media/WebRTC/libwebrtc Update Process/automation plan: Difference between revisions

Revision as of 01:57, 1 December 2022

Plan Overview

Findings

current third_party/libwebrtc directories and status

verify generation/build with extra, unused directories

verify current vendoring scripts cmd-line params

build instructions for moz-libwebrtc

remove dead/unused files

patches after vendoring moz-libwebrtc into moz-central

patches on top of upstream github moz-libwebrtc commit

Keeping transitive dependencies updated

Open Questions

Operation Checklist

Moving third_party/libwebrtc/third_party/* to third_party/*

Motivation

Patch Reduction

Merge Complexity

Approach

Notes

Resetting Elm

Navigation menu

Search