View source for Platform/GFX/Moz2D

The Moz2D graphics API, part of the Azure project, is a cross-platform interface onto the various graphics backends that Gecko uses for rendering such as Direct2D (1.0 and 1.1), Skia, Cairo, Quartz, and NV Path. Adding a new graphics platform to Gecko is accomplished by adding a backend to Moz2D (see [[#Implementing a new backend]]).

Some of the notable features of the API are:
* Mostly stateless&mdash;better suited to CSS rendering and eliminates overhead
* Floating-point&mdash;better suits platform APIs
* API methods line up with HTML canvas

== Current status ==

The Gecko graphics API preceding Moz2D was '''[http://blog.vlad1.com/2007/12/11/graphics-in-mozilla/ Thebes]''' which mostly consists of a C++ wrapper around Cairo, along with some Gecko-specific utility code, and a text API that uses platform text handling. Currently ''both the Thebes API and Moz2D API are in use''. A Thebes wrapper around Moz2D allows code that still uses the Thebes API to run on top of Moz2D. Over time this code will be migrated to using the Moz2D API directly (see [[Platform/GFX/AzureConversion|Azure Conversion]]).

Once the Azure project is completed&mdash;that is, all the backends have been implemented and all the calling code converted to use the new Moz2D API&mdash;the old Thebes API will be removed.

''Side note:'' For historical reasons, there are some classes with &ldquo;Thebes&rdquo; in the name which will be surviving this process as they're not actually part of the graphics API. They may get renamed eventually, to reduce the number of magical words you need to know about, but be warned that just because some code has &ldquo;Thebes&rdquo; in the name, it does not mean that the code is on its way out!

See also:
* [http://www.basschouten.com/blog1.php/progress-on-moz2d Progress on Moz2D] - July 2013 update
* [http://mozillagfx.wordpress.com/2013/08/21/looking-for-a-good-first-place-to-contribute-to-gecko-gfx/ Looking for a good first place to contribute to Gecko gfx?] - August 2013 summary and how you can get involved in moving to Moz2D
* [[Platform/GFX/AzureConversion|Azure Conversion]] - Tracking page for the work to use Moz2D directly

== Stateful vs stateless ==

Many of the performance gains anticipated from Moz2D come about due to its mostly-stateless API.

=== Stateful APIs ===

The Thebes API wrapped the Cairo graphics library which uses a stateful context model much like Postscript or Quartz 2D. To draw content, you make individual API calls to set up various bits of state in a context object, followed by another API call to actually perform drawing. For example, to stroke a shape with a dashed line, you would typically set the color, set the line width, set the dashing style, start a new path, emit path segments, and finally draw the stroke&mdash;''all as separate API calls''. In Cairo, even drawing an image requires the caller to set the source surface, emit a rectangle path, and fill&mdash;at least 6 API calls.

The overhead of setting up state can be justified when that state is re-used, but for CSS rendering this is typically not the case. Rather, for each item that is drawn fresh CSS values are fetched and re-set. Because the browser does not know what it is drawing, it is unable to group operations and re-use state effectively.

On the other hand, the HTML5 canvas element has a stateful API but because its state-tracking differs from that of Cairo drawing state must be tracked twice: once for canvas and once in Cairo.

A further complication specific to Cairo arises because internally Cairo uses a ''stateless'' surface API. On OS X, Cairo uses its Quartz backend for all drawing. However, when using Cairo, despite the fact that both Firefox and Safari ultimately use Quartz as their backend for drawing, Safari is faster than Firefox on some demos. We believe one reason this is the case is because Quartz is stateful. As a result, Cairo needs to convert from its stateful API to its internal, stateless surface API, then back to Quartz’s stateful API.

The Moz2D API, despite being mostly-stateless, was designed to map onto stateful APIs like Quartz in a more efficient manner.

=== Moz2D: Mostly-stateless ===

Almost all the operations on an Moz2D DrawTarget (see [[#Draw targets]] below) take relevant state as parameters and do actual drawing. The only state carried by a DrawTarget is the destination surface itself plus a current transform and a current clip stack. We let the transform and clip state remain in the DrawTarget because those are the only pieces of state not constantly reset by CSS rendering. Our CSS rendering needs to always render under some given transform and clip and we don't want all our rendering code to have to pass those around all the time. Because of this, the Moz2D API is called ''mostly-stateless''.

== Learning Moz2D: Introducing the API ==

Following is an overview of the main pieces of the API. For the details including the most recent changes, please refer to [http://mxr.mozilla.org/mozilla-central/source/gfx/2d/2D.h http://mxr.mozilla.org/mozilla-central/source/gfx/2d/2D.h].

The two main components of the Moz2D API are:
* Source surfaces, and
* Draw targets.

=== Source surfaces ===

Source surfaces represent a ''read-only graphics buffer''. There are actually two categories of source surfaces as follows:

<p style="text-align:center">
[[Image:Source-surface-hierarchy.png|DataSourceSurface inherits from SourceSurface]]
</p>

Both SourceSurface and DataSourceSurface are abstract interfaces that are implemented by the various backends.

A '''SourceSurface''' is an ''opaque representation of a buffer'', a ''handle''. That means you ''can’t'' access its pixel data directly.

So what ''can'' you do?
* Use it in various drawing operations on a DrawTarget. For example,
** Draw it, or just part of it (DrawSurface)
** Use it as a mask when drawing a pattern (MaskSurface)
** Use it as a pattern when filling a rectangle (FillRect after wrapping the SourceSurface in a SourcePattern)
* You can also get a DataSourceSurface from it which ''does'' let you access its data.

A '''DataSourceSurface''' is a subclass of a SourceSurface that provides direct access to pixel data.

It is possible to get the DataSourceSurface corresponding to a SourceSurface using the <code>SourceSurface::GetDataSurface</code> method.

Where do SourceSurfaces come from?

* DrawTarget::Snapshot() &ndash; SourceSurface corresponding to the current contents of a draw target.
* DrawTarget::CreateSourceSurfaceFromData() &ndash; SourceSurface for an existing memory buffer (suitable for using with the draw target it is called on).
* DrawTarget::CreateSourceSurfaceFromNativeSurface() &ndash; SourceSurface for an existing buffer of some… ?
* DrawTarget::OptimizeSourceSurface() &ndash; Converts an existing SourceSurface into one that can be readily used with the given DrawTarget.
* SourceSurface::GetDataSurface() &ndash; Gets a DataSourceSurface corresponding to an existing SourceSurface.
* Factory::CreateDataSourceSurface() &ndash; Creates an new memory buffer to be used as a DataSourceSurface.
* Factory::CreateWrappingDataSourceSurface() &ndash; Creates a DataSourceSurface to wrap an existing buffer.

Each backend provides its own implementation of these interfaces however client code always using the backend-independent interfaces.

The implementation of these interfaces typically takes the following form:

<p style="text-align:center">
[[Image:Source-surface-backend.png|SourceSurfaceD2D implements SourceSurface and DataSourceSurfaceD2D implements DataSourceSurface]]
</p>

=== Draw targets ===

A draw target performs drawing operations on some backing store (e.g. a D3D texture or other buffer).

Includes methods such as the following:

* <code>FillRect</code>
* <code>Fill</code>
* <code>StrokeRect</code>
* <code>StrokeLine</code>
* <code>Stroke</code>
* &hellip;

The following methods are stateful:

* <code>PushClip</code>
* <code>PushClipRect</code>
* <code>PopClip</code>
* <code>SetTransform</code>

Typically the arguments to draw commands take the following order:

# &lt;shape&gt; (Rect / Path) &mdash; the exceptions are FillGlyph and DrawSurface
# A <code>Pattern</code> object that describes the source for filling/stroking
# A <code>DrawOptions</code> object (alpha, comp-op, anti-aliasing, snapping etc.)
# Call-specific draw options

==== Paths ====

To draw paths, we create a <code>PathBuilder</code> for a DrawTarget using <code>CreatePathBuilder</code>. The <code>PathBuilder</code> has methods such as:

* <code>MoveTo</code>
* <code>LineTo</code>
* &hellip;

Finally, calling <code>Finish</code> returns a <code>Path</code> that can be used for drawing (e.g. with <code>Fill</code> or <code>Stroke</code>). After calling <code>Finish</code> the <code>PathBuilder</code> cannot be used again!

==== Patterns ====

Patterns describe the pixels used as the source for a masked composition operation that is done by the different drawing commands. The different types of patterns are:

* <code>ColorPattern</code> &mdash; a solid color fill
* <code>SurfacePattern</code> &mdash; a <code>SourceSurface</code> plus a repeat mode, transformation matrix, and resampling filter
* <code>LinearGradientPattern</code>
* <code>RadialGradientPattern</code>

For linear and radial gradient patterns, the list of gradient stops is backend specific.

These objects are ''not'' ref-counted and may be allocated on the stack.

== Developing Moz2D ==

=== Building Moz2D ===

The Moz2D project can be developed independently of the rest of Gecko/Firefox by checking out the [https://hg.mozilla.org/users/bschouten_mozilla.com/moz2d/ dedicated repository]. The steps for each platform follow.

==== Linux/Mac ====

Depending on which backend you are interested in building, the steps are as follows:

===== Cairo =====

Dependencies:

* libcairo2-dev (http://cairographics.org/download/)
* If you plan on using Player2D, qt4-sdk (http://qt-project.org/downloads)

Steps:

# Clone the repository: <code>hg clone http://hg.mozilla.org/users/bschouten_mozilla.com/moz2d</code>
# On Mac, add the directory containing the cairo.pc file to your PKG_CONFIG_PATH (if it's not there already).
#* One location seen was /usr/X11/lib/pkgconfig 
#* Another location seen was /opt/local/lib/pkgconfig
# On OSX clang++ is required: CXX=clang++
# Build using one of the following targets:
#* release: <code>make release</code>
#* debug: <code>make debug</code>
#* check: <code>make check</code>
#* player2d: <code>make player2d</code>

When building Player2D beware of the following:

* Currently you need to be building with clang in order for Player2D to build due to a Qt dependency.
* If you don't get qmake producing a Makefile, but instead get a xcodeproj file instead, you may need to create the Makefile yourself, using qmake options <span style="background:yellow">(details forthcoming)</span> and run make manually.

===== Skia =====

Dependencies:

* Skia:
** Use a non-debug build. For download and build instructions see https://skia.org/user/download
* If you plan on using Player2D, qt4-sdk (http://qt-project.org/downloads)

Steps:

# Clone the repository: <code>hg clone http://hg.mozilla.org/users/bschouten_mozilla.com/moz2d</code>
# <code>make clean release MOZ2D_SKIA=true</code>
# If you didn't checkout skia under the moz2d src directory then set MOZ2D_SKIA to it's relative path

==== Windows ====

Dependencies:

* [http://www.microsoft.com/visualstudio/eng/downloads VS2012], Express 2012 for Windows Desktop is fine
* A default Cairo and Skia build in the cairo/skia directories in the parent directory of Moz2D if you plan to use skia/cairo

Steps:

# Check out the mercurial repository <code>hg clone http://hg.mozilla.org/users/bschouten_mozilla.com/moz2d</code>
# Open the <code>gfx2d.sln</code> visual studio solution
# Build the solution
#* Build the Debug configuration first since it generates the shader code
#* Don't build Release (Skia) without installing Skia / Cairo first

===== Building Player2D =====

After building Moz2D, you can build the Player2D project (see below) as follows.

Dependencies:

* [http://qt-project.org/downloads QtSDK]
** Use the VS2012 32-bit version
** You will need at least version 5.1.0 since earlier versions don't have a 32-bit binary for VS2012 and have a bug ([https://bugreports.qt-project.org/browse/QTBUG-30822 QTBUG-30822]) that cause Visual Studio to pick up the wrong Windows SDK headers when building in an environment with both VS2010 and VS2012 installed.
** Older versions can be used if needed. Use one of the installers and select "Source Components" when installing. Then, from a VS2012 command prompt run:
*** <code>C:\Qt\Qt5.0.2\5.0.2\Src\qtbase> configure -opensource -confirm-license -static -debug-and-release -no-angle -no-opengl -nomake examples -nomake tests -platform win32-msvc2012</code>
*** <code>nmake</code>
*** (Requires [http://www.activestate.com/activeperl Perl])

From a VS2012 command prompt:

# Go to the <code>moz2d</code> repository clone, <code>cd c:\users\me\moz2d</code>
# <code>mkdir player2d-build</code>
# <code>cd player2d-build</code>
# <code>\qt\Qt5.1.0\5.1.0\msvc2012\bin\qmake.exe ..\player2d\player2d.pro -tp vc -spec win32-msvc2012</code><br>(The final <code>-spec win32-msvc2012</code> might not be necessary, particularly if you only have one version of Visual Studio installed)
# Open the generated <code>player2d.vcxproj</code> file from the <code>player2d-build</code> directory in VS2012.
# Press F7 to build.
# If you get an error about <code>surfaceview.h</code> not being found when referenced from <code>ui_sourcesurfaceview.h</code>, change the line that reads:<br><code>#include &lt;surfaceview.h&gt;</code><br>to<br><code>#include "surfaceview.h"</code>
# Add the directory containing the QT DLLs to your path (e.g. C:\qt\Qt5.1.0\5.1.0\msvc2012\bin)
# Run player2d

=== Recording and playback ===

In order to test different backends and conduct performance analysis it is possible to make a recording of calls to the Moz2D API&mdash;for example, those used when rendering a certain Web page&mdash;and then replay them step by step on different backends.

==== Making a recording ====

On a platform where Moz2D content is enabled in Firefox (currently only Windows):

* Create a profile for recording if you don't already have a suitable one,<br>
  <code>c:\Program Files (x86)\Nightly\firefox.exe -no-remote -ProfileManager</code>
* Assuming we want a recording for http://www.website.com and our profile is called &ldquo;recording&rdquo; we can save a recording to <code>mytest.aer</code> with the following command line:<br>
  <code>c:\Program Files (x86)\Nightly\firefox.exe -no-remote -P recording -recording http://www.website.com/ -recording-output mytest.aer</code>

Sometimes recordings are prematurely ended and come up empty. This is a known bug.

==== Playing a recording ====

Having made a recording it is possible to replay the steps on different backends using the Player2D tool (see [[#Building Moz2D]] above). The steps can be played back one by one and it is also possible to step backwards and forwards to inspect the result of each step.

The result is as follows:

'''<span style="color:maroon">(Note that in order to see anything in the View window you must double-click an appropriate object from the Objects window.)</span>'''

[[Image:Player2d-screenshot.png|Screenshot of Player 2D playing back a recording of slashdot.org.]]

===== Tools available =====

Some of the other tools available in the Moz2D repository include:

* recordbench &mdash; reports timings for a recording across a range of backends. Using this application we should be able to do automated performance comparisons between different drawing backends.
* micro-benchmark suite &mdash; allows us to make detailed performance comparisons on a per-backend basis. Hopefully this will give us the ability to make well-informed decisions as to what drawing backends to prefer on specific platforms/configurations.

=== Implementing a new backend ===

So you want to add your own backend&hellip; great! Here’s how to get started:

# Create a recording using Player 2D
# Implement DrawTarget
# Implement Factory::CreateDrawTarget
# Add BackendType to static array
# Add new backend to TestDrawTarget

== Further reading ==

The following resources describe the motivation for and goals of this API:
* [https://blog.mozilla.org/joe/2011/04/26/introducing-the-azure-project/ Introducing the Azure Project]
* [http://people.mozilla.com/~roc/Samsung/Mozilla2DGraphics.pdf Mozilla 2D graphics] (PDF)
* [http://robert.ocallahan.org/2011/09/graphics-api-design.html Graphics API Design]