Media/WebRTC/libwebrtc Update Process/automation plan

This describes the rough plan to document and automate the process to "fast-forward" through webrtc.org libwebrtc change-sets to bring Firefox's libwebrtc version as close as possible to Chrome. This page will be updated as new information is discovered.

Plan Overview

• repo tasks

- update moz-libwebrtc fork to latest webrtc.org

git checkout master
git remote add upstream https://webrtc.googlesource.com/src
git fetch upstream
git merge upstream/master # fast forward merge
git push origin master

- update elm to latest moz-central

• verify current third_party/libwebrtc directories
• verify generation/build with extra, unused directories from stock moz-libwebrtc in place
• verify current vendoring scripts cmd-line params

- check moz-central a93c2d1df066 - Bug 1654112 - Vendor libwebrtc for rel86

- check moz-central d1df4970f4f0 - Bug 1654112 - Vendor libwebrtc/third_party for rel86

- check moz-central 587c40771588 - Bug 1654112 - Vendor libwebrtc/build for rel86

• verify build instructions for moz-libwebrtc
• remove dead/unused files in third_party/libwebrtc
• identify patch set for changes after vendored moz-libwebrtc (587c40771588)
• identify patch set for changes to upstream github revision (edd7f9e94d)
• Manually identify how to fast-forward one changeset

- move one changeset forward in webrtc.org history

- verify local build in moz-libwebrtc

- apply moz-libwebrtc-to-upstream patches, verify local build

- vendor moz-libwebrtc

- apply moz-central patches

- full try run

• write automation scripts "Manually identify how to fast-forward one changeset"
• investigate updatebot

Findings

current third_party/libwebrtc directories and status

• Additional vendoring

build (with some changes during vendoring)
third_party

• Mozilla added

README.mozilla - generated during vendoring process
moz.build - generated by generate-gn-build-files.sh
webrtc_gn - generated by generate-gn-build-files.sh
X11 - imported later by Byron in c91f12b557a1 D113830
tools/grit - imported later by Nico in 3cce5e6938f0 D114027

• Mozilla vendored after a93c2d1df066

- Both of the following were inadvertently vendored from moz-libwebrtc commit bd4a718667. The only meaningful difference is in sdk/objc/helpers/RTCDispatcher.{h|m}, but these files are not currently used.

sdk/BUILD.gn - imported later by Byron in 9314046d89eb D105015
sdk/objc - imported later by Byron in 9314046d89eb D105015

- We don't build the sdk directory (see here).

- Only sdk:helpers_objc is used (see here)

- From sdk/objc/helpers, only sdk/objc/helpers/scoped_cftyperef.h is used:

third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_iosurface.h
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_provider.h
third_party/libwebrtc/modules/desktop_capture/mac/screen_capturer_mac.mm
third_party/libwebrtc/modules/desktop_capture/mac/desktop_frame_cgimage.h

• Untouched from Bug 1665166 - Move media/webrtc/trunk/* to third-party/libwebrtc

README.md   (should have been vendored)
DEPS        (should have been vendored)
google_apis (is this referenced/used by current code?)
tools/clang (is this referenced/used by current code?)            
chromium_deps                   
dummy_file.txt                  
net                             
Makefile.old                    
peerconnection_client.target.mk 
peerconnection.Makefile

verify generation/build with extra, unused directories

• Add missing directories

cp -r ~/git-checkouts/moz-libwebrtc/test ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/pc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/resources ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/docs ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/tools_webrtc ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/examples ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/p2p ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/rtc_tools ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/data ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/style-guide ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/stats ~/mozilla/moz-central/third_party/libwebrtc/
cp -r ~/git-checkouts/moz-libwebrtc/media/sctp ~/mozilla/moz-central/third_party/libwebrtc/media

• local build file generation produces no build modified build files
• Full build results here using ./mach try fuzzy -q 'build-'
• Full linux test results here using ./mach try fuzzy -q 'linux-'

verify current vendoring scripts cmd-line params

• Vendor libwebrtc (a93c2d1df066)

- The following commands produce a93c2d1df066 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-fixup.patch.zip

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-github https://github.com/mozilla/libwebrtc \
        --commit 149d693483e9055f574d9d65b01fe75a186b654b \
        libwebrtc

• Vendor libwebrtc/third_party (d1df4970f4f0)

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-googlesource \
        --commit 5dc5a4a45df9592baa8e8c5f896006d9193d8e45 \
        third_party

• Vendor libwebrtc/build (587c40771588)

- The following commands produce 587c40771588 with some changes that were checked in on the same commit. After running the command below, apply the following patch to match the moz-central commit: File:Vendor-build-fixup.patch.zip

cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
        --from-googlesource \
        --commit ac22062b465485d3e1196ae9c506b3617cc16fb5 \
        build

build instructions for moz-libwebrtc

• verify build for webrtc.org

- Note: we can build as far forward as eacbd972ab (HEAD, branch-heads/4287, branch-heads/4286, branch-heads/4285, branch-heads/4284) without needing to gsync

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools ; git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
mkdir webrtc-checkout
cd webrtc-checkout
fetch --nohooks webrtc # populates .gclient and .gclient_entries
(cd src ; git checkout edd7f9e94d)
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
gclient sync
cd src
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test

• verify build for moz-libwebrtc

- Use edd7f9e94d because some our patches stacked on top of this upstream revision are specific to building in the moz-central tree.

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export DEPOT_TOOLS=`pwd`/depot_tools
(cd depot_tools ; git checkout e7d1862b155ac3ccbef72c4d70629b5c88ffcb32 )
export PATH=$DEPOT_TOOLS:$PATH ; \
  export DEPOT_TOOLS_UPDATE=0 ; \
  export DEPOT_TOOLS_WIN_TOOLCHAIN=0
mkdir moz-libwebrtc-checkout
cd moz-libwebrtc-checkout
gclient config https://github.com/mozilla/libwebrtc --name src --unmanaged
gclient sync -D --force --reset --with_branch_heads # ~30 min
cd src
git checkout edd7f9e94d
gn gen out/Default
ninja -C out/Default
./out/Default/rtc_media_unittests # quick smoke test

remove dead/unused files

• Successfully removed:

chromium_deps
dummy_file.txt
net
Makefile.old
peerconnection_client.target.mk
peerconnection.Makefile

- successful try builds

tools/clang

- successful try builds

patches after vendoring moz-libwebrtc into moz-central

hg diff -r 587c40771588 -r central \
   --exclude '**/moz.build' \
   --exclude 'third_party/libwebrtc/X11' \
   --exclude 'third_party/libwebrtc/sdk/BUILD.gn' \
   --exclude 'third_party/libwebrtc/sdk/objc' \
   --exclude 'third_party/libwebrtc/tools/grit' \
   --exclude 'third_party/libwebrtc/build' \
   --exclude 'third_party/libwebrtc/third_party' \
   third_party/libwebrtc

patches on top of upstream github moz-libwebrtc commit

• easiest way

git diff -r edd7f9e94d -r mozilla-modifications-rel86

• more useful for rebasing on top of upstream commits

git branch mozpatches edd7f9e94d
git checkout mozpatches
git format-patch -k edd7f9e94d..mozilla-modifications-rel86
git am *.patch # applies to branch mozpatches

• in both cases, due to an anomaly during the previous vendoring operation an additional patch must be applied to match the vendored commit in moz-central

File:Vendor-fixup.patch.zip

curl -o vendor-fixup.patch.zip https://wiki.mozilla.org/images/2/29/Vendor-fixup.patch.zip
unzip vendor-fixup.patch.zip
(cd third_party/libwebrtc && patch -p1 < vendor-fixup.patch)

Open Questions

• How to find versions of webrtc.org that ship with Chrome?

 There don't appear to be tags or branches in webrtc.org.

• Do we want to advance commit-by-commit?

 I think we may need to do this to avoid large-scale breakage and
 make it easier to fix issues.

• Do we want to manually advance the process as quickly as possible, and then modify our scripts to start taking advantage of update-bot?

 This allows a more finely tuned process to help us get caught up,
 but pushes out the integration with update-bot.

• How/when do we start integrating webrtc.org's unittests into our try runs?

 This may need to happen in our tree (probably post-vendoring) because
 the process of getting the webrtc tree buildable is a long process
 requiring lots of network.

• Do we want to add our fixup-patch to the moz-libwebrtc branch and move the branch forward in the official repo to make it more clear what was previously vendored?