MozillaWiki

Services/Sync/Getting Started: Difference between revisions

Page Discussion

Services/Sync/Getting Started (view source)

Revision as of 16:09, 8 February 2018

5,209 bytes removed ,  8 February 2018
Various updates
(Various updates)
Line 56: Line 56:
= Using Bugzilla =
= Using Bugzilla =


Every code change starts out as a bug in [https://bugzilla.mozilla.org Bugzilla] where the general approach can be discussed. The bug should eventually be assigned to somebody who will upload one or more patches for review. Code can only be reviewed by module owners and their peers; see [[Services/Code_Review|code review policy]] for more information. If the review has been granted and all review comments have been addressed, the patches may be checked into [https://hg.mozilla.org/services/services-central services-central], which is the Mercurial repository used for Services client work.
Every code change starts out as a bug in [https://bugzilla.mozilla.org Bugzilla] where the general approach can be discussed. The bug should eventually be assigned to somebody who will upload one or more patches for review. Code can only be reviewed by module owners and their peers; see [[Services/Code_Review|code review policy]] for more information. If the review has been granted and all review comments have been addressed, the patches may be checked into [http://hg.mozilla.org/mozilla-central mozilla-central].
 
On rare occasions it will make more sense to land work on [http://hg.mozilla.org/mozilla-central mozilla-central] instead — for example, if a change needs to propagate upstream before the regular [[Services/Process/MergingBetweenBranches|merge from services-central to mozilla-central]]. This might involve additional procedures (e.g., you need to request approval in addition to review.)


For server code written in Python, Tarek has provided a useful tool called [https://bitbucket.org/tarek/flake8 flake8] which does Python style checks against your code.  
For server code written in Python, Tarek has provided a useful tool called [https://bitbucket.org/tarek/flake8 flake8] which does Python style checks against your code.  
Line 77: Line 75:


= Using Mercurial =
= Using Mercurial =
Note: This section may be outdated.


Mozilla's development process is very much patch driven. [https://developer.mozilla.org/En/Developer_Guide/Source_Code Mozilla uses Mercurial] (as well as git for some projects) as the version control system. [https://developer.mozilla.org/en/Mercurial_queues Mercurial Patch Queues], although a bit awkward to use at times, fit this process rather well. It is recommended you familiarize yourself with them and use them.
Mozilla's development process is very much patch driven. [https://developer.mozilla.org/En/Developer_Guide/Source_Code Mozilla uses Mercurial] (as well as git for some projects) as the version control system. [https://developer.mozilla.org/en/Mercurial_queues Mercurial Patch Queues], although a bit awkward to use at times, fit this process rather well. It is recommended you familiarize yourself with them and use them.
You can find the Services team's code repository at [http://hg.mozilla.org/services].


== Configuration ==
== Configuration ==
Line 120: Line 118:
== Landing patches ==
== Landing patches ==


You have items in a patch queue that you'd like to push to s-c. Here's the thorough set of steps to follow to get them into the tree (assuming you have commit access, of course).
You have items in a patch queue that you'd like to push to m-c. Here's the thorough set of steps to follow to get them into the tree (assuming you have commit access, of course). It's recommended that you use autoland via mozreview, or flag bugs as having checkin-needed instead.  


=== Make sure services-central is open and green ===
=== Make sure -central is open and green ===


Check [https://tbpl.mozilla.org/?tree=Services-Central TBPL].
Check [https://treeherder.mozilla.org treeherder].


=== Make sure your tree is up to date ===
=== Make sure your tree is up to date ===
Line 162: Line 160:
=== Run tests ===
=== Run tests ===


Now is a great time to run tests, because this is the state in which services-central will be after you push. If you apply a <tt>tests</tt> patch to re-enable disabled tests, ''make sure you pop it when you're done''.
Now is a great time to run tests, because this is the state in which mozilla-central will be after you push. If you apply a <tt>tests</tt> patch to re-enable disabled tests, ''make sure you pop it when you're done''.


=== Finish the applied patches ===
=== Finish the applied patches ===
Line 182: Line 180:
= Using Git =
= Using Git =


Various projects exist as Git repositories instead of Mercurial repositories. Even when projects are officially hosted in Mercurial, Git mirrors are maintained.
Various projects exist as Git repositories instead of Mercurial repositories. Even when projects are officially hosted in Mercurial, Git mirrors are maintained. The mozilla-central mirror is available at https://github.com/mozilla/gecko-dev, however we recommend that you use https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development to set up a local git repository that tracks mozilla central. If you already have a gecko-dev clone, you can use [https://github.com/glandium/git-cinnabar/wiki/Mozilla:-Using-a-git-clone-of-gecko%E2%80%90dev-to-push-to-mercurial this guide] to configure it to push to hg using git-cinnabar.
 
The Services team maintains a GitHub organization at https://github.com/mozilla-services. A mirror of services-central can be found at https://github.com/mozilla-services/services-central.


== Configuring Git ==
== Configuring Git ==
Line 193: Line 189:
  $ git config --global user.name "John Doe"
  $ git config --global user.name "John Doe"
  $ git config --global user.email "johndoe@mozilla.com"
  $ git config --global user.email "johndoe@mozilla.com"
 
  # Enable color output for all commands
  # Enable color output for all commands
  $ git config --global color.branch auto
  $ git config --global color.branch auto
Line 201: Line 197:
  $ git config --global color.status auto
  $ git config --global color.status auto
  $ git config --global color.ui auto
  $ git config --global color.ui auto
 
  # Highlight whitespace
  # Highlight whitespace
  $ git config --global core.whitespace "trailing-space,space-before-tab,tab-in-indent"
  $ git config --global core.whitespace "trailing-space,space-before-tab,tab-in-indent"
 
  # Define your editor for commits, other tools
  # Define your editor for commits, other tools
  $ git config --global core.editor vim
  $ git config --global core.editor vim


= Android Sync =
If you find you have a slow git prompt, you may find the instructions [https://gist.github.com/thomcc/4c23eeed3b248394bffa0bff04f5c00b here] beneficial as well.
 
You should carefully read the previous sections on Git and landing patches on services-central.
 
For the moment we assume that you have a working checkout of [http://github.com/mozilla-services/android-sync android-sync]. The rudiments of landing code are this:
 
<pre>
AS=~/moz/git/android-sync
SC=~/moz/hg/services-central
 
# In your services-central checkout, create a new patch.
# Ensure that you're correctly carrying over the author from the Git commit.
cd $SC
hg qnew -m "Bug 123456 - Part 1: frob the noo. r=foobar, a=blocking-fennec" --user "John Smith <theauthor@foo.com>" frob-noo
 
# For each merged piece of work that you'd like to land...
# Ensure you're on the correct git branch. That's typically develop, or an
# ancestor thereof if you've merged multiple feature branches since your last landing.
cd $AS
git checkout develop
 
# Run the appropriate copy-code script, targeting the correct upstream hg repository.
ANDROID=$SC/mobile/android SYNC=$SC/mobile/android/sync ./fennec-copy-code.sh
 
# Refresh the patch, adding and removing any files.
cd $SC
hg st
hg add any-missing-files.java
hg rm anything-removed.java
hg qref
 
# Check that the patch is what you want to commit.
# You are responsible for anything that you land in the tree,
# so it behooves you to make sure you get this right.
less .hg/patches/frob-noo
 
# Ensure that everything builds.
make -sj4 -C objdir-droid
 
# And runs...
make -C objdir-droid package
adb install -r objdir-droid/dist/fennec-15.0a1.en-US.android-arm.apk
</pre>
 
Now follow the steps for a regular services-central landing.
 
This is all eminently scriptable, of course.


After setting up


= Repositories =
= Repositories =


Getting a checkout of a large Mercurial repository can be a pain if you're remote. Nobody wants to wait 6 hours while Python drools on its shoes. The answer is '''bundles'''. [https://developer.mozilla.org/en/Mozilla_Source_Code_%28Mercurial%29 Follow the directions]. The services-central bundle is [http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/services-central.hg here].
Getting a checkout of a large Mercurial repository can be a pain if you're remote. Nobody wants to wait 6 hours while Python drools on its shoes. The answer is '''bundles'''. [https://developer.mozilla.org/en/Mozilla_Source_Code_%28Mercurial%29 Follow the directions]. More information on bundles is available [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/Mercurial/Bundles here].
 
== services-central ==
 
[https://hg.mozilla.org/services/services-central services-central] is the clone of mozilla-central in which the Sync client (and other stable Services client-side projects) are developed.
 
Things that live there:
 
* The Services client libraries in services/crypto/ and services/sync.
 
* Built-in UI in browser/base/content/.
 
== fx-sync ==
 
Sync was primarily developed in the [https://hg.mozilla.org/services/fx-sync/ fx-sync] repository. The code there includes support for the Sync add-on; it's deprecated since Firefox 4.
 
Things that live there:
 
* The Services client libraries in services/crypto/ and services/sync.
 
* The Firefox Sync add-on in addon/*
 
* The Firefox 3.x and Fennec 1.x UIs for the Sync add-on in ui/*


== mozilla-central ==
== mozilla-central ==


mozilla-central is the repository for Firefox and the base repository for other Mozilla-based applications. How mozilla-central is relevant to Sync:
mozilla-central is the repository for Firefox and the base repository for other Mozilla-based applications. Sync code used to live in a repository named services-central, however currently all client side sync code for Desktop Firefox and Firefox for Android lives in mozilla-central.
 
* services-central is [[Services/Process/MergingBetweenBranches|periodically merged into mozilla-central]].
 
* The integrated UI for Firefox Sync lives in browser/base/*, along with the rest of the Firefox UI.
 
* Firefox releases are spun from mozilla-central. Changes made to services-central don't directly reach users.


== Try server ==
== Try server ==
Because Sync is developed in a clone of mozilla-central, you can use ordinary "try build" procedures to test your changes. (You don't always need to, though; one of the advantages of having an integration branch like services-central is that we're at liberty to break it!)


To push to try, teave your changes as applied mq patches. Then run:
To push to try, teave your changes as applied mq patches. Then run:


   hg push -f ssh://hg.mozilla.org/try/
   hg push -f ssh://hg.mozilla.org/try/
Alternatively, you can push to try using `mach try`, which works for both `git` and `hg` users (assuming git users are using git cinnabar), and is recommended for both.




= Testing =
= Testing =


[https://developer.mozilla.org/en/Writing_xpcshell-based_unit_tests Writing xpcshell-based unit testing].
[https://developer.mozilla.org/en/Writing_xpcshell-based_unit_tests Writing xpcshell-based unit testing]. Sync tests are located at [https://searchfox.org/mozilla-central/source/services/sync/tests/unit/ /services/sync/tests/unit]
 
==Python Functional Tests==
 
In the services/ tree are a handful of HTTP servers implemented in JS. These are effectively lightweight clones of the services described by the specs at http://docs.services.mozilla.com/.
 
The main implementations of these services are in Python. And, there exist a Python test suite that you can point at any HTTP endpoint and they will validate the functional behavior of the server. We use these Python functional tests to validate that the in-tree JS server implementations are functionally proper.
 
To run these functional tests, you'll need a built Firefox source tree and will need to obtain the Python server code, which contains the tests.


===Running JS Servers===
[https://developer.mozilla.org/en-US/docs/Mozilla/Projects/TPS_Tests TPS Tests]
 
There is Makefile magic in services/common/Makefile that can launch a standalone JS server instance. From your object directory:
 
make -C services/common storage-server
make -C services/common aitc-server
 
===Running Python Functional Tests===
 
You'll need to obtain the server source code, then launch the tests. For example:
 
git clone git://github.com/mozilla-services/server-syncstorage.git
cd server-syncstorage
make
source bin/activate
python syncstorage/tests/functional/test_storage.py http://localhost:8080
 
If you see lots of errors like
 
Traceback (most recent call last):
  File "syncstorage/tests/functional/test_storage.py", line 52, in setUp
    super(TestStorage, self).setUp()
  File "/Users/ncalexan/Mozilla/server-syncstorage/syncstorage/tests/functional/support.py", line 36, in setUp
    self._authenticate()
  File "/Users/ncalexan/Mozilla/server-syncstorage/syncstorage/tests/functional/support.py", line 49, in _authenticate
    auth_policy = self.config.registry.getUtility(IAuthenticationPolicy)
  File "/Users/ncalexan/Mozilla/server-syncstorage/lib/python2.7/site-packages/zope/interface/registry.py", line 169, in getUtility
    raise ComponentLookupError(provided, name)
ComponentLookupError: (<InterfaceClass pyramid.interfaces.IAuthenticationPolicy>, u'')
 
you may need to follow the instructions at https://mail.mozilla.org/pipermail/services-dev/2012-September/001564.html.


= Collaboration =
= Collaboration =
Line 370: Line 255:
   }
   }
   {
   {
     comment = "Public channel for Identity";
     comment = "Public channel for FxA";
     name = "#identity";
     name = "#fxa";
     chatnet = "mozilla";
     chatnet = "mozilla";
   }
   }
  }
  }
== Team Meetings ==
* Services Weekly Status: Every Tuesday at 9:15 PT in North Bridge (4th floor), [https://intranet.mozilla.org/Teleconferencing conference dial] in number is #8616. Meeting notes live under [https://wiki.mozilla.org/Services/Meetings/ Services/Meetings].
* [[WeeklyUpdates|Mozilla Weekly Updates]]: Every Monday at 11:00 PT in 10-Forward (you'll need to attend the first one in person since we want to show you off.)
* Out-of-office, PTO, and other kinds of availability is managed in a shared "Services Team Availability" calendar. See [https://intranet.mozilla.org/Calendar#Shared_Company_Calendars Shared Company Calenders] for details on how to set it up.


= Miscellany =
= Miscellany =
Tcsc
8

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1188723"