Media/WebRTC/libwebrtc Update Process/automation plan

This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome. This page will be updated as new information is discovered.

Plan Overview

• repo tasks

- update moz-libwebrtc fork to latest webrtc.org

git checkout master
git remote add upstream https://webrtc.googlesource.com/src
git fetch upstream
git merge upstream/master # fast forward merge
git push origin master

- update elm to latest moz-central

• verify current third_party/libwebrtc directories
• verify generation/build with extra, unused directories from stock moz-libwebrtc in place
• verify current vendoring scripts cmd-line params

- check moz-central a93c2d1df066 - Bug 1654112 - Vendor libwebrtc for rel86

- check moz-central d1df4970f4f0 - Bug 1654112 - Vendor libwebrtc/third_party for rel86

- check moz-central 587c40771588 - Bug 1654112 - Vendor libwebrtc/build for rel86

• verify build instructions for moz-libwebrtc
• remove dead/unused files in third_party/libwebrtc
• identify patch set for changes after vendored moz-libwebrtc (587c40771588)
• identify patch set for changes to upstream github revision (edd7f9e94d)
• Manually identify how to fast-forward one changeset

- move one changeset forward in webrtc.org history

- verify local build in moz-libwebrtc

- apply moz-libwebrtc-to-upstream patches, verify local build

- vendor moz-libwebrtc

- apply moz-central patches

- full try run

• write automation scripts "Manually identify how to fast-forward one changeset"
• investigate updatebot

Findings

current third_party/libwebrtc directories and status

• Additional vendoring

build (with some changes during vendoring)
third_party

• Mozilla added

README.mozilla - generated during vendoring process
moz.build - generated by generate-gn-build-files.sh
webrtc_gn - generated by generate-gn-build-files.sh
X11 - imported later by Byron in c91f12b557a1 D113830
tools/grit - imported later by Nico in 3cce5e6938f0 D114027

• Mozilla vendored after a93c2d1df066

- Both of the following were inadvertently vendored from moz-libwebrtc commit bd4a718667. The only meaningful difference is in sdk/objc/helpers/RTCDispatcher.{h|m}, but these files are not currently used.

sdk/BUILD.gn - imported later by Byron in 9314046d89eb D105015
sdk/objc - imported later by Byron in 9314046d89eb D105015

- We don't build the sdk directory (see here).

- Only sdk:helpers_objc is used (see here)

- From sdk/objc/helpers, only sdk/objc/helpers/scoped_cftyperef.h is used:

third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_iosurface.h
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_provider.h
third_party/libwebrtc/modules/desktop_capture/mac/screen_capturer_mac.mm
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_cgimage.h

• Untouched from Bug 1665166 - Move media/webrtc/trunk/* to third-party/libwebrtc

README.md   (should have been vendored)
DEPS        (should have been vendored)
google_apis (is this referenced/used by current code?)
tools/clang (is this referenced/used by current code?)            
chromium_deps                   
dummy_file.txt                  
net                             
Makefile.old                    
peerconnection_client.target.mk 
peerconnection.Makefile

verify generation/build with extra, unused directories

• Add missing directories

cp -r ~/git-checkouts/moz-libwebrtc/test ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/pc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/resources ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/docs ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/tools_webrtc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/examples ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/p2p ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/rtc_tools ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/data ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/style-guide ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/stats ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/media/sctp ~/mozilla/moz-central/third_party/libwebrtc/media

• local build file generation produces no build modified build files
• Full build results here using ./mach try fuzzy -q 'build-'
• Full linux test results here using ./mach try fuzzy -q 'linux-'

verify current vendoring scripts cmd-line params

• Vendor libwebrtc (a93c2d1df066)

- The following commands produce a93c2d1df066 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-fixup.patch.zip

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-github https://github.com/mozilla/libwebrtc \
        --commit 149d693483e9055f574d9d65b01fe75a186b654b \
        libwebrtc

• Vendor libwebrtc/third_party (d1df4970f4f0)

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-googlesource \
        --commit 5dc5a4a45df9592baa8e8c5f896006d9193d8e45 \
        third_party

• Vendor libwebrtc/build (587c40771588)

- The following commands produce 587c40771588 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-build-fixup.patch.zip

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-googlesource \
        --commit ac22062b465485d3e1196ae9c506b3617cc16fb5 \
        build

build instructions for moz-libwebrtc

• verify build for webrtc.org

- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools ; git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
mkdir webrtc-checkout
cd webrtc-checkout
fetch --nohooks webrtc # populates .gclient and .gclient_entries
(cd src ; git checkout edd7f9e94d)
gclient sync
cd src
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test

• verify build for moz-libwebrtc

- Use edd7f9e94d because some our patches stacked on top of this upstream revision are specific to building in the moz-central tree.

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools ; git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
mkdir moz-libwebrtc-checkout
cd moz-libwebrtc-checkout
gclient config https://github.com/mozilla/libwebrtc --name src --unmanaged
gclient sync -D --force --reset --with_branch_heads # ~30 min
cd src
git checkout edd7f9e94d
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test

remove dead/unused files

• Successfully removed:

chromium_deps
dummy_file.txt
net
Makefile.old
peerconnection_client.target.mk
peerconnection.Makefile

- successful try builds

tools/clang

- successful try builds

patches after vendoring moz-libwebrtc into moz-central

hg diff -r 587c40771588 -r central \
   --exclude '**/moz.build' \
   --exclude 'third_party/libwebrtc/X11' \
   --exclude 'third_party/libwebrtc/sdk/BUILD.gn' \
   --exclude 'third_party/libwebrtc/sdk/objc' \
   --exclude 'third_party/libwebrtc/tools/grit' \
   --exclude 'third_party/libwebrtc/build' \
   --exclude 'third_party/libwebrtc/third_party' \
   third_party/libwebrtc

- While the above command works, when used within the fast foward script it requires reapplication of fixes for merge conflicts with each new advance in the webrtc.org changesets. It makes more sense to reuse the extract-for-git.py script to build a patch set of changes we can apply on top of our current patch stack residing in moz-libwebrtc. That command looks like:

cd dom/media/webrtc/third_party_build && \
python3 extract-for-git.py 587c40771588::2b187d932ca1

- where 587c40771588 is the moz-central vendoring commit and 2b187d932ca1 is the current tip of moz-central

- extract-for-git produces a file named mailbox.patch which can be applied in moz-libwebrtc with:

git am mailbox.patch

- Note, fixes were required during the git am process. These were dealing with a couple places were we imported directories into moz-central from webrtc.org after the vendoring commit, and a couple cases of Windows vs Linux line endings causing conflicts.

patches on top of upstream github moz-libwebrtc commit

• easiest way

git diff -r edd7f9e94d -r mozilla-modifications-rel86

• more useful for rebasing on top of upstream commits

git branch mozpatches edd7f9e94d
git checkout mozpatches
git format-patch -k edd7f9e94d..mozilla-modifications-rel86
git am *.patch # applies to branch mozpatches

• in both cases, due to an anomaly during the previous vendoring operation an additional patch must be applied to match the vendored commit in moz-central

File:Vendor-fixup.patch.zip

curl -o vendor-fixup.patch.zip https://wiki.mozilla.org/images/2/29/Vendor-fixup.patch.zip
unzip vendor-fixup.patch.zip
(cd third_party/libwebrtc && patch -p1 < vendor-fixup.patch)

Keeping transitive dependencies updated

• we will use updatebot to keep our dependencies updated. This may happen after an initial manual update.
• libyuv
• libvpx
• pipewire

Open Questions

• How to see release branches in webrtc.org?

 Add "fetch = +refs/branch-heads/*:refs/remotes/branch-heads/*" to .git/config and run 'git pull'.

• How to find versions of webrtc.org that ship with Chrome?

 If you have acquired webrtc.org/src using gclient sync (instructions), there are branches listed that look like 'branch-heads/nnnn'.  You can cross-reference the Chrome release and find the webrtc.org release here: https://chromiumdash.appspot.com/branches
 example: Chrome's 98 release uses webrtc.org 4758, which is sha a6b138d6b4

• Do we want to advance commit-by-commit?

 I think we may need to do this to avoid large-scale breakage and
 make it easier to fix issues.

• Do we want to manually advance the process as quickly as possible, and then modify our scripts to start taking advantage of update-bot?

 This allows a more finely tuned process to help us get caught up,
 but pushes out the integration with update-bot.

• How/when do we start integrating webrtc.org's unittests into our try runs?

 This may need to happen in our tree (probably post-vendoring) because
 the process of getting the webrtc tree buildable is a long process
 requiring lots of network.

• Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?

Operation Checklist

• Prerequisites

Builds are done after vendoring in each commit. Having a .mozconfig file that enables cache (either sccache or ccache) will improve the performance of the scripts.

hg clone https://hg.mozilla.org/projects/elm
(cd elm && ./mach bootstrap --application-choice=browser && ./mach build)
git clone https://github.com/mozilla/libwebrtc moz-libwebrtc

• Update steps

# Start from the elm checkout
cd elm

# Prepare the github repo
bash dom/media/webrtc/third_party_build/prep_repo.sh

# Run loop-ff.sh
bash bash dom/media/webrtc/third_party_build/loop-ff.sh

• Operational notes

- Running prep-repo.sh with a fresh github clone of moz-libwebrtc will likely display instructions on how to add recent moz-central changes made in third_party/libwebrtc to the top of the patch stack in github. This is expected and necessary for successfully vendoring changes into third_party/libwebrtc.

- Two main types of errors will cause loop-ff.sh to exit; the first is a rebase conflict, the second is a build error due to api changes in upstream code.

- When making changes to fix build issues in moz-libwebrtc, mercurial commit messages follow the pattern:

Bug xxx - (fix-{upstream-sha}) {description}

- When making changes to fix Mozilla code in response to changes upstream, mercurial commit messages follow the pattern:

Bug xxx (MOZ) - {description}

- While loop-ff.sh is running, in a separate terminal window it may be helpful to see the overall progress by running:

tail -f ~/log-loop-ff.txt | grep loop-ff