Mobile/AsyncSubframePanning

From MozillaWiki
Jump to navigation Jump to search

Goals

Phase 1

  • land in 23 (not likely)
  • B2G subframe scrolling
  • Performance is
    • as good as main frame
  • subframes can't zoom
  • seamed scrolling

Phase 2

  • Fennec subframe scrolling
  • seamless scrolling

Not doing

  • scroll indicators are a non-goal
  • not fixing page drift
  • concurrently scrolling parents and ancestors
  • subframe zooming

Plan

To dos
  • merge B2G APZC into fennec
  • multiple display ports
  • bind multiple display ports
  • hit detection
  • transform equation cleanup
    • unit tests
  • tuning/profiling
  • concurrently scrolling subframes

Design

Step 1

Performed in Layout module, Main Gecko thread

  • (Bug 866232) Layout (FrameLayerBuilder et al.) builds an initial simple layer tree with the position of the subframes tagged to the best fitting layer
  • (See diagram for sample page & layer tree)

Step 2

Performed in the compositor's main thread

  • Layer tree update/transaction is received by the compositor.
  • Touch event is delivered at (6,51).
  • (Bug 775452) Hit detection is performed on a layer tree in reverse depth-first order and matches 1st subframe.
  • APZC decides of the appropriate buffer/pre-rendering and sends an IPC message to the main thread with an updated FrameMetrics/display port request.
  • (See diagram for sample page & layer tree)

Step 3

Performed in Layout module, Main Gecko thread

  • Display port is requested on the subframe in Layout
  • (Bug 864447) Layer tree is grown to make room for a layer of the 1st subframe.
  • Subframe is painted with a display port allowing Async scrolling. FrameMetrics are updated.
  • (See diagram for sample page & layer tree)

Step 4

Performed in the compositor's main thread

  • (Bug 866265) APZC can now perform Async panning on new sublayer.

Step 5

  • After timeout APZC resets FrameMetrics/display port on the subframe removing the display port, the layer and releasing the memory usage.

Notes and diagram

ApzcDesignPage1.png 9-.png

Bugs

Tracking bug for async subframe scrolling: bug 775452


Full Query
ID Priority Summary Status Assigned to
775459 -- Implement (more) hit-testing logic for async/synchronously scrolled frames on UI thread RESOLVED Benoit Girard (:BenWa)
833795 -- Content jitters when URL bar hidden RESOLVED Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)
864441 -- Add APZC tests RESOLVED Benoit Girard (:BenWa)
864447 -- Support drawing display port on nested APZC frames RESOLVED Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)
865735 -- Disambiguate between Point instances in different coordinate systems RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
866232 -- Support having multiple APZC instances attached to the layer tree RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
866265 -- Transformation equation cleanups RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
872911 -- Find a better solution for handling touch gestures with moving frames RESOLVED
883646 -- Distinguish between CSSPixel and "device pixel" for Fennec RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
890279 -- Create ScrollInfoLayers for frames we want APZC support RESOLVED Benoit Girard (:BenWa)
890280 -- APZC input event transformations should consist of the async transform relative to the correct coordinate system RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
891384 -- APZC Hit Testing doesn't support 3D transform RESOLVED
893872 -- Add TransformShadowTree unit tests RESOLVED Benoit Girard (:BenWa)
894288 -- nsLayoutUtils::FindContentFor aborts when loading any tab RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
894619 -- TransformShadowTree only transforms root APZC RESOLVED
896045 -- Do input events go to the right place with off-main-thread animations? RESOLVED
898443 -- Figure out a good focus model for input events to multi-APZC RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
898444 -- Figure out why overflow:scroll elements aren't scrolling properly RESOLVED Robert O'Callahan (:roc) (email my personal email if necessary)
900638 -- Fix event-fluffing to work with APZC RESOLVED Dale Harvey (:daleharvey)
903460 -- AsyncCompositionManager applies async transform and CSS transform in the wrong order RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
907754 -- Pages with iframes that are less than screen width are not rendering correctly RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
912144 -- Hit testing for subframes is sometimes not right RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
913659 -- Strange behaviour when zooming page with iframe a lot RESOLVED
916125 -- When displayports are set on elements, their layers are not always clipped correctly RESOLVED Timothy Nikkel (:tnikkel)

24 Total; 0 Open (0%); 24 Resolved (100%); 0 Verified (0%);


Tracking bug for allowing Fennec to use APZC: bug 776030


Full Query
ID Priority Summary Status Assigned to
777075 -- Extract an interface for the PanZoomController -> LayerController dependency RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
777351 -- Merge LayerController and GeckoLayerClient RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
777468 -- Extract an interface for dependencies on PanZoomController RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
777896 -- Break dependencies from PanZoomController -> other java code RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
839641 -- Wire up the AsyncPanZoomController to be usable from java RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
859929 -- AsyncPanZoomController doesn't play well with progressive tile painting RESOLVED Botond Ballo [:botond] [away until July 9]
859935 -- AsyncPanZoomController assumes the "UI/controller" has a message loop RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
859939 -- AsyncPanZoomController has some type wrongness RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
859951 -- Refactor the code to convert from MOTION_EVENT to nsTouchEvent instances RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
859962 -- BrowserApp assumes LayerView.getPanZoomController never returns null RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
860162 -- AsyncPanZoomController doesn't apply proper scroll compensation if the root layer has a scale RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
860455 -- AsyncPanZoomController::WidgetSpaceToCompensatedViewportSpace is not fully implemented RESOLVED
860613 -- Add a way to bind fennec's APZC to the primary scrollable layer RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
860940 -- Replace AndroidGeckoEvent constructors with static factory methods RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
867269 -- SyncFrameMetrics override is disconnected and breaks compiles on gcc 4.7+ RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
880024 -- Create a local version of GeckoContentController RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
880578 -- APZC: Scroll events are only sent when repainting is needed RESOLVED Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)
880932 -- APZC: Page stops repainting and zoom/pan stops updating RESOLVED Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)
881451 -- ThebesLayerComposite::GetCompositionBounds() needs to be fixed up in various ways RESOLVED Chris Lord [:cwiiis]
890280 -- APZC input event transformations should consist of the async transform relative to the correct coordinate system RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
895904 -- NativePanZoomController.abortAnimation() needs to be hooked up RESOLVED Botond Ballo [:botond] [away until July 9]
943557 -- Use modifier keys for click events generated by single-tap gestures in Android APZC code RESOLVED
1046344 -- Unbitrot APZ glue code for Fennec RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1049136 -- Hook up touch event flow for APZ usage on Fennec RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1055548 -- With native APZ enabled clicking on stuff sometimes doesn't work quite right RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1055557 -- Properly notify the APZ of root zoom constraints and changes to zoom constraints RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1137872 -- Make the urlbar hiding stuff work with native pan zoom controller RESOLVED
1146020 -- Make APZCCallbackHandler extend from ChromeProcessController RESOLVED Danilo Cesar Lemes de Paula
1146024 -- Fix up input routing for APZ on Fennec RESOLVED Danilo Cesar Lemes de Paula
1156392 -- Remove MOZ_ANDROID_APZ condition from ifdef in ScrollFrameHelper::WantAsyncScroll RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1156401 -- Fix ContextMenu handling when building Fennec with C++ APZ enabled RESOLVED Danilo Cesar Lemes de Paula
1158424 -- Clean up FrameMetrics::mIsRoot and related notions RESOLVED Botond Ballo [:botond] [away until July 9]
1159405 -- Fix "Found a non-root APZ with no handoff parent" warning when root process has a root XUL document with scrollable subframes RESOLVED Danilo Cesar Lemes de Paula
1162701 -- Fix panning on Fennec when built with C++ APZ RESOLVED
1163572 -- Override RequestContentRepaint to detect root-frame updates RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1175282 -- [Fennec-APZ] Unable to scroll outside scrollable subframe after zooming on a non-scrollable document RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1180267 -- Update Fennec code to use MobileViewportManager instead of the big pile of browser.js code RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1188225 -- Implement ChromeProcessController::HandleDoubleTap RESOLVED Randall Barker [:rbarker]
1190979 -- Text is selected when pinch zooming RESOLVED
1190988 -- Hit testing appears to return incorrect positions when --enable-android-apz is specified. RESOLVED Randall Barker [:rbarker]
1195553 -- Double tap gesture fails on Fennec when C++ APZ is enabled. RESOLVED Randall Barker [:rbarker]
1197824 -- Update Fennec code to use ZoomConstraintsClient instead of the big pile of browser.js code RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1200393 -- Java APZ appears more responsive during page load compared to C++ APZ. RESOLVED
1200397 -- Pinch zoom gesture does not always appear to work when C++APZ is enabled. RESOLVED
1200399 -- The dynamic tool bar in Fennec will some times permanently hide RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1201100 -- Perma-checkerboarding when zoomed in with apz-fennec RESOLVED Randall Barker [:rbarker]
1201130 -- Switching tabs in Fennec-APZ sometimes doesn't update the metrics properly RESOLVED
1201217 -- Switching tabs in Fennec-APZ sometimes renders the page at an incorrect zoom RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1201529 -- When APZ is enabled, all browser.js codepaths that set a resolution or displayport should be disabled RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1201581 -- Fixed-position margins/clip are not updated properly in fennec-apz RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1202020 P5 Remove dependencies on async pan/zoom information from Java code RESOLVED
1206620 -- Zoom out does not work at double tap action in Landscape mode on Tablet devices RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1206872 -- [meta] Enable APZ on fennec on nightly builds RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1206874 -- [meta] Let APZ on Fennec ride the trains RESOLVED
1229349 -- Tapping on things in the zoomed view is broken with C++ APZ RESOLVED
1229439 -- Horizontal Scrolling of elements is not registered during page load or janky RESOLVED
1229462 -- Use Android Scroller class for fling animation RESOLVED Randall Barker [:rbarker]
1229840 -- Reader mode toolbar seems to show/hide a lot with C++ APZ RESOLVED
1230081 -- Scrolling on partially rendered pages causes judder and other issues RESOLVED
1230085 -- White checkerboarding visible (instead of page background) RESOLVED
1230176 P3 Weirdness at end of Fennec fling. Don't send duplicate DOM 'scroll' event for sub-pixel scrolling REOPENED
1240065 -- Reconsider viewport update policy when pinch zooming RESOLVED Randall Barker [:rbarker]
1248014 -- Allow horizontal scrolling from a pointer device when positioned over a horizontally scrolling layer. RESOLVED
1260534 P3 Checkerboarding in reader view is white instead of page background color RESOLVED
1260905 -- Fennec APZ flywheel physics result in very strange behavior when you do sequence of quick and short pans RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1262807 P1 Lots of repainting while scrolling around in Fennec RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1265372 -- Scrolling in reader mode enters/exits fullscreen mode too easily RESOLVED
1272478 -- Cannot scroll about:memory horizontally VERIFIED Randall Barker [:rbarker]
1274597 P1 Clicking on links sometimes doesn't work, long-pressing results in text selection instead of context menu RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1278390 P2 Fennec doesn't display Gitter.im content, after scrolling up/down RESOLVED
1280666 P2 Use modified OverScroller to work around bugs RESOLVED James Willcox (:snorp) (jwillcox@mozilla.com) (he/him)
1281042 P3 Displayport expiry can result in perma-checkerboarding RESOLVED
1291678 P3 Some posts on GoT (gathering of tweakers)-forum are scrollable for fractions of pixels NEW
1297148 P3 Crash in mozilla::CrossProcessMutex::CrossProcessMutex RESOLVED
1302150 P2 Dynamic toolbar doesn't reappear after scrolling to bottom of long page VERIFIED Randall Barker [:rbarker]

75 Total; 2 Open (2.67%); 71 Resolved (94.67%); 2 Verified (2.67%);


Meetings

2013-04-26

  • initial meeting, created phases, goals and overall plan

2013-05-21

  • Chris will take the display port work

2013-05-28

  • Kats put up patch for disambiguated point classes
    • will continue propegating the use of the classes through the
    • should be up for review this week
  • Chris should start on display ports per scroll frame this week

2013-06-04

  • Kats
    • propagating type changes to frame metrics and APZC code
  • BenWa
    • unit tests
    • hit testing
  • ajones
    • multiple display ports
    • code already largely supports it
  • does metro need subframe scrolling?
  • next items:
    • hit testing (in progress, owned by BenWa)
    • creating and managing APZCs from the frame loader builder (need owner)
AsyncSubFrameSchedule.jpeg


2013-06-11

  • Getting Jim and Brian up to speed, explaining how things work:
    • Going through some of our "older" bugs, figuring out the history of how this came to be.
    • C++ version that is now being used in Android, almost working completely
    • In depth discussion on what the pieces are, how they work together
  • Jim & Brian will work on getting single APZC without subframes on Metro. Primarily Brian.
    • This will be based off C++ code which B2G uses, and there may be remaining issues in there for a desktop platform. We have to make sure this is tested as soon as it starts working so that we can flush out any remaining bugs.
  • The work that Kats, BenWa, AJones are doing should then just click in place and give us subframe APZC on Metro as well.

2013-06-18

  • Kats:
    • There is another coordinate system that we have to worry about, and B2G seems to be very broken on high dpi devices. This is what the code looks like as well.
    • We don't need high dpi for 1.1, so we probably don't need to uplift this. Need to test these changes, and then we should be OK.
  • Benoit:
    • On hit testing during the last week. More complicated test cases, with transforms, etc.
  • Multiple display ports support in the layout, but it probably already supports it. We just need to use those assumptions and proceed.

2013-06-25

  • BenWa:
    • Pushed the hit testing on try, failed, sorting it out now.
  • Kats:
    • Sorted out regressions. Tests for bad transformation equations.
    • Ran into CSS transforms problem (rotated iFrame, vertical moves end up being diagonal).
Retrieved from "https://wiki.allizom.org/index.php?title=Mobile/AsyncSubframePanning&oldid=670070"