Services/Sync/Getting Started: Difference between revisions

From MozillaWiki
< Services‎ | Sync
Jump to navigation Jump to search
(Adding getting started guide)
 
 
(57 intermediate revisions by 6 users not shown)
Line 1: Line 1:
= Getting started with Sync =
= Hacking =


== Prerequisites ==
== Source Code ==


I'm assuming you're running a Mac with Snow Leopard.
Once you have a copy of the Mozilla source tree (read below for how to obtain a copy), you can find the core Sync source code in:


=== Install HomeBrew ===
/services/sync/


  ruby -e "$(curl -fsS http://gist.github.com/raw/323731/install_homebrew.rb)"
You can view this online at https://hg.mozilla.org/mozilla-unified/file/default/services/sync


=== Install dependencies ===
This is just the core JavaScript APIs and logic. Additional Sync code for integration with Firefox lives in:


* Autoconf: put the following in /usr/local/Library/Formula/autoconf.rb:
/browser/base/content


  require 'formula'
Look for files with ''sync'' in their name. Files here define the integration between Firefox and Sync. This includes all the Sync UI.
 
 
  class Autoconf <Formula
Additionally, Firefox Accounts specific code lives in
    url 'http://ftp.gnu.org/pub/gnu/autoconf/autoconf-2.13.tar.gz'
 
    homepage 'http://www.gnu.org/software/autoconf/'
/services/fxaccounts
    md5 '9de56d4a161a723228220b0f425dc711'
 
 
== Debugging Sync ==
    # depends_on 'cmake'
 
 
If you are debugging Sync in your browser, the first thing you want to do is crank up the logging. In about:config, set the following preferences:
    def install
 
      system "./configure", "--disable-debug", "--disable-dependency-tracking",
# Enable logging to Error Console
                            "--prefix=#{prefix}"
services.sync.log.appender.console Trace
      # system "cmake . #{std_cmake_parameters}"
 
      system "make install"
# Enable logging to STDOUT
    end
services.sync.log.appender.dump Trace
  end
 
# Log full sync logs on sync success (typically they are only saved on failure)
services.sync.log.appender.file.logOnSuccess true
 
# Enable trace logging for engines
services.sync.log.logger.engine Trace
# (optional) Enable trace logging for all of sync (this is very verbose and will include all request bodies)
services.sync.log.logger Trace
 
Once you've updated those settings, restart Firefox for them to take effect.
 
If you are logging to STDOUT, you may want to set the environment variable ''MOZ_QUIET'' to silence a lot of spew from other parts of Firefox:
 
export MOZ_QUIET=1
 
You can view full sync logs by opening '''about:sync-log''' in Firefox.
 
== Building Sync ==
 
Sync code is almost exclusively JavaScript, and so you probably should build using an artifact build, as described [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Artifact_builds here].
 
After setting this up, you'll do `mach build` to perform a build, `mach run` to run Firefox, and `mach xpcshell-test /services/sync/tests/unit` to run the sync unit test suite. There's also a separate end-to-end test suite, TPS, which is documented [https://developer.mozilla.org/en-US/docs/Mozilla/Projects/TPS_Tests here]. (For most changes its sufficient you run the unit tests)
 
Depending on your OS, most changes will not require a rebuild to take effect, only restarting and rerunning via `mach run`.
 
= Using Bugzilla =
 
Every code change starts out as a bug in [https://bugzilla.mozilla.org Bugzilla] where the general approach can be discussed. The bug should eventually be assigned to somebody who will upload one or more patches for review. Code can only be reviewed by module owners and their peers; see [[Services/Code_Review|code review policy]] for more information. If the review has been granted and all review comments have been addressed, the patches may be checked into [http://hg.mozilla.org/mozilla-central mozilla-central].
 
For server code written in Python, Tarek has provided a useful tool called [https://bitbucket.org/tarek/flake8 flake8] which does Python style checks against your code.
 
You probably also want to get up to speed on [http://bugzilla.mozilla.org Bugzilla] if you're not already.
* A good start is to go watch [http://vimeo.com/9205730 Jonath's "Bugzilla for Humans" video].
* Our bugs are currently under "Other Products", "Mozilla Services" or just bookmark "[https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services Add a bug]" and "[https://bugzilla.mozilla.org/describecomponents.cgi?product=Mozilla%20Services Browse bugs]".
* Don't forget to add a Bugzilla ID shortcut string to your Bugzilla [https://bugzilla.mozilla.org/userprefs.cgi?tab=account account preferences].  The current standard practice is to append "<whitespace>[:irc_nick]" to your Real Name field.
* You also want to watch the components relevant to your work so you automatically get emails for all bug changes in those components. The easiest way to do this is to add the default QA contact for each component to the list of users your are watching in Bugzilla
** Open the [https://bugzilla.mozilla.org/describecomponents.cgi?product=Mozilla%20Services list of Services components]
** In another tab go to your Bugzilla [https://bugzilla.mozilla.org/userprefs.cgi?tab=email email preferences]
** At the bottom of the page under the <b>User Watching</b> section there is an input form that takes a comma separated list of email addresses
** If you're working on server side components, you will want to add at least:
*** core-server@services.bugs
*** sync-server@services.bugs
* A useful Firefox add-on that makes life in Bugzilla a bit easier is [https://addons.mozilla.org/en-Us/firefox/addon/bugzilla-tweaks/ Bugzilla Tweaks]. Highly recommended!


* Then install a bunch of packages (some required for Sync, some just useful):


  brew install \
= Using Mercurial =
  ack distribute gdbm git gnutls libgcrypt libidl libtasn1 pcre pkg-config \
  python3 spidermonkey wget yasm autoconf gettext glib jsl libgpg-error \
  libpng lynx nspr pip python qrencode readline sqlite wireshark


* Now you can install Mercurial:
Note: This section may be outdated.


  pip install mercurial
Mozilla's development process is very much patch driven. [https://developer.mozilla.org/En/Developer_Guide/Source_Code Mozilla uses Mercurial] (as well as git for some projects) as the version control system. [https://developer.mozilla.org/en/Mercurial_queues Mercurial Patch Queues], although a bit awkward to use at times, fit this process rather well. It is recommended you familiarize yourself with them and use them.


=== Configure Mercurial ===
== Configuration ==


Put the following in ~/.hg/hgrc:
Marco Bonardo has some [http://blog.bonardo.net/2010/06/22/so-youre-about-to-use-checkin-needed general purpose tips] for Mozilla developers on how to format patches before uploading them to Bugzilla.


  [paths]
Essentially it is recommended you put the following in your ~/.hgrc:
  default = http://hg.mozilla.org/services/
  default-push = ssh://hg.mozilla.org/services/
     
and the following in ~/.hgrc:


   [ui]
   [ui]
Line 59: Line 94:
   [diff]
   [diff]
   git = 1
   git = 1
  unified = 8
 
  [defaults]
  diff = -pU8
  qdiff = -pU8
  qnew = -U
  qseries = -sv
If you find yourself mixing up `hg qpush` (push a patch onto your queue) and `hg push` (irreversibly push your changes out into the world) try adding this:
  [hooks]
  preoutgoing.confirm = read -p 'Are you sure you want to push to remote? (y/n): '; echo $REPLY | grep -q 'y'
(Note from philikon: I consider this hook pretty much useless. Mercurial will warn and abort if you're trying to with patches applied. So if you have any applied patches, it's very hard to accidentally push them. If you don't have applied patches, you're most likely not to have anything the remote repo doesn't have either.)
Also, useful path definitions:
  [paths]
  try = ssh://hg.mozilla.org/try/
  places = ssh://hg.mozilla.org/projects/places/
  mc = ssh://hg.mozilla.org/mozilla-central/
== Landing patches ==
You have items in a patch queue that you'd like to push to m-c. Here's the thorough set of steps to follow to get them into the tree (assuming you have commit access, of course). It's recommended that you use autoland via mozreview, or flag bugs as having checkin-needed instead.
=== Make sure mozilla-central is open and green ===
Check [https://treeherder.mozilla.org treeherder].
=== Make sure your tree is up to date ===
  hg qpop --all
  hg pull -u
===  Make sure that all the patches have a proper commit message ===
For simple bugs, the format is
  Bug NNNNNN - Bug Summary here. r=your_reviewers_IRC_nickname
For bugs with several parts, you can do
  Bug NNNNNN - Bug Summary here. r=your_reviewers_IRC_nickname
  `
  Part X: Frobnicate the majiggaberator
Or simply
  Bug NNNNNN - Part X: Frobnicate the majiggaberator. r=your_reviewers_IRC_nickname
Run <tt>hg qser -sv</tt> (omit the <tt>-sv</tt> if you're using the config above). The output will look something like
  0 U favicons-1a: Bug 675996 - Part 1: extend moz_favicons with GUID to support Sync. r=mak
  1 U favicons-1b: Bug 675996 - Part 2: adjust Places migration tests to expect current version, not v11; tests for GUIDs in favicons. r=mak
If they don't, <tt>hg qpush</tt> each one and edit the summary with <tt>hg qref -e</tt>.
=== Ensure that the only patches you have applied are the ones you want to push ===
… and that they're the ones that were reviewed. You can reorder patches by <tt>hg qpop --all</tt> then <tt>hg qpush --move $patchname</tt>, or by manually reordering .hg/patches/series. ''Only edit the series file when all patches have been popped.''
Applied patches are indicated by an 'A' in the 2nd column of qseries. Unapplied are 'U'.
=== Run tests ===
Now is a great time to run tests, because this is the state in which mozilla-central will be after you push. If you apply a <tt>tests</tt> patch to re-enable disabled tests, ''make sure you pop it when you're done''.
=== Finish the applied patches ===
  hg qfinish --applied
=== Verify what you're going to upload ===
  hg out
If the command shows anything other than the patches you're expecting (or they're wrong in some way), ''stop'' and ask for help.
=== Push! ===
If all is good, push:
  hg push
= Using Git =
Various projects exist as Git repositories instead of Mercurial repositories. Even when projects are officially hosted in Mercurial, Git mirrors are maintained. The mozilla-central mirror is available at https://github.com/mozilla/gecko-dev, however we recommend that you use https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development to set up a local git repository that tracks mozilla central. If you already have a gecko-dev clone, you can use [https://github.com/glandium/git-cinnabar/wiki/Mozilla:-Using-a-git-clone-of-gecko%E2%80%90dev-to-push-to-mercurial this guide] to configure it to push to hg using git-cinnabar.
== Configuring Git ==
If this is your first time using Git, you'll need for perform some one-time configuration:
# Configure your global name and email. These will be used for commit info.
$ git config --global user.name "John Doe"
$ git config --global user.email "johndoe@mozilla.com"
# Enable color output for all commands
$ git config --global color.branch auto
$ git config --global color.diff auto
$ git config --global color.grep auto
$ git config --global color.interactive auto
$ git config --global color.status auto
$ git config --global color.ui auto
# Highlight whitespace
$ git config --global core.whitespace "trailing-space,space-before-tab,tab-in-indent"
# Define your editor for commits, other tools
$ git config --global core.editor vim
If you find you have a slow git prompt, you may find the instructions [https://gist.github.com/thomcc/4c23eeed3b248394bffa0bff04f5c00b here] beneficial as well.
= Repositories =
Getting a checkout of a large Mercurial repository can be a pain if you're remote. Nobody wants to wait 6 hours while Python drools on its shoes. The answer is '''bundles'''. [https://developer.mozilla.org/en/Mozilla_Source_Code_%28Mercurial%29 Follow the directions]. More information on bundles is available [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/Mercurial/Bundles here].
== mozilla-central ==
mozilla-central is the repository for Firefox and the base repository for other Mozilla-based applications. Sync code used to live in a repository named services-central, however currently all client side sync code for Desktop Firefox and Firefox for Android lives in mozilla-central.
== Try server ==
To push to try, teave your changes as applied mq patches. Then run:
  hg push -f ssh://hg.mozilla.org/try/
Alternatively, you can push to try using `mach try`, which works for both `git` and `hg` users (assuming git users are using git cinnabar), and is recommended for both.
= Testing =
[https://developer.mozilla.org/en/Writing_xpcshell-based_unit_tests Writing xpcshell-based unit testing]. Sync tests are located at [https://searchfox.org/mozilla-central/source/services/sync/tests/unit/ /services/sync/tests/unit]
[https://developer.mozilla.org/en-US/docs/Mozilla/Projects/TPS_Tests TPS Tests]
= Collaboration =


== IRC ==


=== Check out the Mercurial repos you'll need ===
Being a geographically distributed group, we tend to hang out on IRC. One thing you may wish to consider is running an [http://quadpoint.org/articles/irssi irssi client in a screen] session on a shell host. Here's a truncated config file you can use to connect:


   mkdir -p ~/moz/hg/services
servers = (
   cd ~/moz/hg/services
   {
   for x in \
    address = "irc.mozilla.org";
  admin-scripts fx-home fx-sync-tools loadtest reg-server server-core \
    chatnet = "mozilla";
  server-key-exchange server-storage dashboard fx-sync internal_dash \
    port = "6697";
  minimal-server reg-server-secure server-full server-reg sync-server
    autoconnect = "yes";
  do
    use_ssl = "yes";
  hg clone ssh://hg.mozilla.org/services/$x
  }
  done
   };
chatnets = { type = "IRC"; };
channels = {
   {
    comment = "Public channel for Sync";
    name = "#sync";
    chatnet = "mozilla";
  }
  {
    comment = "Public channel for FxA";
    name = "#fxa";
    chatnet = "mozilla";
  }
}


Check out Mozilla Central, too.
= Miscellany =


  cd ~/moz/hg
* You also want to grab a [http://nightly.mozilla.org/ Nightly] build of Firefox. Everybody in the company should be running a nightly build: it's a great way to find bugs before the code goes out to the general public. If you want to run against different profiles (which is useful for development), you will be interested in [https://developer.mozilla.org/en/Command_Line_Options two extra command flags]: "-P and --no-remote"
  hg clone ssh://hg.mozilla.org/mozilla-central
** -P forces the Profile Picker (You probably want to have more than one.)
** --no-remote allows multiple versions to run on one machine at the same time.

Latest revision as of 16:11, 8 February 2018

Hacking

Source Code

Once you have a copy of the Mozilla source tree (read below for how to obtain a copy), you can find the core Sync source code in:

/services/sync/

You can view this online at https://hg.mozilla.org/mozilla-unified/file/default/services/sync

This is just the core JavaScript APIs and logic. Additional Sync code for integration with Firefox lives in:

/browser/base/content

Look for files with sync in their name. Files here define the integration between Firefox and Sync. This includes all the Sync UI.

Additionally, Firefox Accounts specific code lives in

/services/fxaccounts

Debugging Sync

If you are debugging Sync in your browser, the first thing you want to do is crank up the logging. In about:config, set the following preferences:

# Enable logging to Error Console
services.sync.log.appender.console Trace
# Enable logging to STDOUT
services.sync.log.appender.dump Trace
# Log full sync logs on sync success (typically they are only saved on failure)
services.sync.log.appender.file.logOnSuccess true
# Enable trace logging for engines
services.sync.log.logger.engine Trace

# (optional) Enable trace logging for all of sync (this is very verbose and will include all request bodies)
services.sync.log.logger Trace

Once you've updated those settings, restart Firefox for them to take effect.

If you are logging to STDOUT, you may want to set the environment variable MOZ_QUIET to silence a lot of spew from other parts of Firefox:

export MOZ_QUIET=1

You can view full sync logs by opening about:sync-log in Firefox.

Building Sync

Sync code is almost exclusively JavaScript, and so you probably should build using an artifact build, as described here.

After setting this up, you'll do `mach build` to perform a build, `mach run` to run Firefox, and `mach xpcshell-test /services/sync/tests/unit` to run the sync unit test suite. There's also a separate end-to-end test suite, TPS, which is documented here. (For most changes its sufficient you run the unit tests)

Depending on your OS, most changes will not require a rebuild to take effect, only restarting and rerunning via `mach run`.

Using Bugzilla

Every code change starts out as a bug in Bugzilla where the general approach can be discussed. The bug should eventually be assigned to somebody who will upload one or more patches for review. Code can only be reviewed by module owners and their peers; see code review policy for more information. If the review has been granted and all review comments have been addressed, the patches may be checked into mozilla-central.

For server code written in Python, Tarek has provided a useful tool called flake8 which does Python style checks against your code.

You probably also want to get up to speed on Bugzilla if you're not already.

  • A good start is to go watch Jonath's "Bugzilla for Humans" video.
  • Our bugs are currently under "Other Products", "Mozilla Services" or just bookmark "Add a bug" and "Browse bugs".
  • Don't forget to add a Bugzilla ID shortcut string to your Bugzilla account preferences. The current standard practice is to append "<whitespace>[:irc_nick]" to your Real Name field.
  • You also want to watch the components relevant to your work so you automatically get emails for all bug changes in those components. The easiest way to do this is to add the default QA contact for each component to the list of users your are watching in Bugzilla
    • Open the list of Services components
    • In another tab go to your Bugzilla email preferences
    • At the bottom of the page under the User Watching section there is an input form that takes a comma separated list of email addresses
    • If you're working on server side components, you will want to add at least:
      • core-server@services.bugs
      • sync-server@services.bugs
  • A useful Firefox add-on that makes life in Bugzilla a bit easier is Bugzilla Tweaks. Highly recommended!


Using Mercurial

Note: This section may be outdated.

Mozilla's development process is very much patch driven. Mozilla uses Mercurial (as well as git for some projects) as the version control system. Mercurial Patch Queues, although a bit awkward to use at times, fit this process rather well. It is recommended you familiarize yourself with them and use them.

Configuration

Marco Bonardo has some general purpose tips for Mozilla developers on how to format patches before uploading them to Bugzilla.

Essentially it is recommended you put the following in your ~/.hgrc:

 [ui]
 username = Your Name <you@mozilla.com>
 
 [extensions]
 hgext.mq = 
 
 [diff]
 git = 1
 unified = 8
 
 [defaults]
 diff = -pU8
 qdiff = -pU8
 qnew = -U
 qseries = -sv

If you find yourself mixing up `hg qpush` (push a patch onto your queue) and `hg push` (irreversibly push your changes out into the world) try adding this:

 [hooks]
 preoutgoing.confirm = read -p 'Are you sure you want to push to remote? (y/n): '; echo $REPLY | grep -q 'y'

(Note from philikon: I consider this hook pretty much useless. Mercurial will warn and abort if you're trying to with patches applied. So if you have any applied patches, it's very hard to accidentally push them. If you don't have applied patches, you're most likely not to have anything the remote repo doesn't have either.)

Also, useful path definitions:

 [paths]
 try = ssh://hg.mozilla.org/try/
 places = ssh://hg.mozilla.org/projects/places/
 mc = ssh://hg.mozilla.org/mozilla-central/

Landing patches

You have items in a patch queue that you'd like to push to m-c. Here's the thorough set of steps to follow to get them into the tree (assuming you have commit access, of course). It's recommended that you use autoland via mozreview, or flag bugs as having checkin-needed instead.

Make sure mozilla-central is open and green

Check treeherder.

Make sure your tree is up to date

 hg qpop --all
 hg pull -u

Make sure that all the patches have a proper commit message

For simple bugs, the format is

 Bug NNNNNN - Bug Summary here. r=your_reviewers_IRC_nickname

For bugs with several parts, you can do

 Bug NNNNNN - Bug Summary here. r=your_reviewers_IRC_nickname
 `
 Part X: Frobnicate the majiggaberator

Or simply

 Bug NNNNNN - Part X: Frobnicate the majiggaberator. r=your_reviewers_IRC_nickname

Run hg qser -sv (omit the -sv if you're using the config above). The output will look something like

 0 U favicons-1a: Bug 675996 - Part 1: extend moz_favicons with GUID to support Sync. r=mak
 1 U favicons-1b: Bug 675996 - Part 2: adjust Places migration tests to expect current version, not v11; tests for GUIDs in favicons. r=mak

If they don't, hg qpush each one and edit the summary with hg qref -e.

Ensure that the only patches you have applied are the ones you want to push

… and that they're the ones that were reviewed. You can reorder patches by hg qpop --all then hg qpush --move $patchname, or by manually reordering .hg/patches/series. Only edit the series file when all patches have been popped.

Applied patches are indicated by an 'A' in the 2nd column of qseries. Unapplied are 'U'.

Run tests

Now is a great time to run tests, because this is the state in which mozilla-central will be after you push. If you apply a tests patch to re-enable disabled tests, make sure you pop it when you're done.

Finish the applied patches

 hg qfinish --applied


Verify what you're going to upload

 hg out

If the command shows anything other than the patches you're expecting (or they're wrong in some way), stop and ask for help.

Push!

If all is good, push:

 hg push

Using Git

Various projects exist as Git repositories instead of Mercurial repositories. Even when projects are officially hosted in Mercurial, Git mirrors are maintained. The mozilla-central mirror is available at https://github.com/mozilla/gecko-dev, however we recommend that you use https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development to set up a local git repository that tracks mozilla central. If you already have a gecko-dev clone, you can use this guide to configure it to push to hg using git-cinnabar.

Configuring Git

If this is your first time using Git, you'll need for perform some one-time configuration:

# Configure your global name and email. These will be used for commit info.
$ git config --global user.name "John Doe"
$ git config --global user.email "johndoe@mozilla.com"

# Enable color output for all commands
$ git config --global color.branch auto
$ git config --global color.diff auto
$ git config --global color.grep auto
$ git config --global color.interactive auto
$ git config --global color.status auto
$ git config --global color.ui auto

# Highlight whitespace
$ git config --global core.whitespace "trailing-space,space-before-tab,tab-in-indent"

# Define your editor for commits, other tools
$ git config --global core.editor vim

If you find you have a slow git prompt, you may find the instructions here beneficial as well.

Repositories

Getting a checkout of a large Mercurial repository can be a pain if you're remote. Nobody wants to wait 6 hours while Python drools on its shoes. The answer is bundles. Follow the directions. More information on bundles is available here.

mozilla-central

mozilla-central is the repository for Firefox and the base repository for other Mozilla-based applications. Sync code used to live in a repository named services-central, however currently all client side sync code for Desktop Firefox and Firefox for Android lives in mozilla-central.

Try server

To push to try, teave your changes as applied mq patches. Then run:

 hg push -f ssh://hg.mozilla.org/try/

Alternatively, you can push to try using `mach try`, which works for both `git` and `hg` users (assuming git users are using git cinnabar), and is recommended for both.


Testing

Writing xpcshell-based unit testing. Sync tests are located at /services/sync/tests/unit

TPS Tests

Collaboration

IRC

Being a geographically distributed group, we tend to hang out on IRC. One thing you may wish to consider is running an irssi client in a screen session on a shell host. Here's a truncated config file you can use to connect:

servers = (
 {
   address = "irc.mozilla.org";
   chatnet = "mozilla";
   port = "6697";
   autoconnect = "yes";
   use_ssl = "yes";
  }
 };
chatnets = { type = "IRC"; }; 

channels = { 
 {
   comment = "Public channel for Sync";
   name = "#sync";
   chatnet = "mozilla";
  }
  {
   comment = "Public channel for FxA";
   name = "#fxa";
   chatnet = "mozilla";
  }
}

Miscellany

  • You also want to grab a Nightly build of Firefox. Everybody in the company should be running a nightly build: it's a great way to find bugs before the code goes out to the general public. If you want to run against different profiles (which is useful for development), you will be interested in two extra command flags: "-P and --no-remote"
    • -P forces the Profile Picker (You probably want to have more than one.)
    • --no-remote allows multiple versions to run on one machine at the same time.