Webmaker/Code: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 241: Line 241:


At this point you can ask your reviewer to land your changes, either via irc, or in a comment in the bug.
At this point you can ask your reviewer to land your changes, either via irc, or in a comment in the bug.
TODO: document whiteboard flags
* l10n changes
* sumo or other doc changes
* other?


===4. Verify your code on Staging===
===4. Verify your code on Staging===

Revision as of 20:04, 11 October 2013

NOTE: this page is under active development, and will be changing rapidly over the coming days. Apologies for any chaos you discover at the moment.

9689638299_10d2cd241d.jpg

Introduction

Webmaker is about empowering everyone to become makers of the web instead of just users. It's a mix of web tools, teaching material, and a social platform for teaching, learning, and sharing what you make. And, it's a lot of fun to use. It's even more fun to develop!

Webmaker is built using modern web technologies including Node.js, HTML5, CSS, and JavaScript (lots of JavaScript!)--many of the same technologies Webmaker teaches. If you know these technologies, or are interested in learning them more deeply, you might be interested in helping us develop Webmaker.

6332218821_9a8c984055.jpg

Getting Involved

There are many ways you can get involved in building the Webmaker project. Here are a few ideas:

  • If you know JavaScript, you could help fix bugs or add features to our servers (e.g., the MakeAPI), JS libraries and modules (e.g., popcorn.js or our localization tools), or web tools.
  • If you know HTML/CSS, you could work on our web sites and web tools, for example webmaker.org, Popcorn Maker, Thimble, etc.
  • If you know MySQL/MongoDB/ElasticSearch and are interested in databases and data, you could help us work on the MakeAPI or Login server.
  • If you're interested in Bug Triage, you could help us keep the Webmaker Product on bugzilla neat and tidy.
  • If you know how to speak more than one language, you could work on localization and help us maintain our localization code.
  • If you know how to work on Firefox/FirefoxOS or another Mozilla product/project, you could help us find ways to integrate Webmaker, for example building browser addons.
  • If you know about automation/scripting/packaging/deployment, you could work with developer operations group to help deploy and manage Webmaker.
  • If you're good at finding issues, confirming bugs, or otherwise breaking things, you could help us test Webmaker code on our staging and production servers.
  • If you know how to do UI/UX work, you could join our design and front-end group to design and implement new user-facing aspects of our sites and tools.

These are just a few ideas to get you thinking--there's many ways that someone who wants to work on Webmaker can get involved. If you're passionate and interested in getting started, let us know and we'll try to find something that fits your interests and skills.

10121615535_b7717819b3.jpg

Communication

In order to get started on Webmaker code, you need to get involved with our community. Here are some ways to introduce yourself and get help:

  • IRC: The #webmaker channel on moznet is our primary dev channel. If you're new to irc, please see the docs on learning how to use it. NOTE: our development team is spread across the world, with primary activity happening in Toronto, Vancouver, London, Berlin, New York, and San Francisco. If you're in a timezone that doesn't overlap with these, the mailing lists might be a better way to reach out.
  • Mailing Lists: There are two primary lists: webmaker@lists.mozilla.org for general Webmaker communication; and webmaker-dev@mozilla.org for developer communication.
  • Bug Tracker: We use Mozilla's Bugzilla to track our work, see the Webmaker Product in Bugzilla.
  • Blog: Updates about the development of the Webmaker Product go on our Webmaker blog.
10121810133_1c67fa1ca2.jpg

Developing Webmaker

Webmaker is a big project, and it can take some time to get set up and contributing. The following is a guide to our workflow, technologies, repositories, etc. Please read this documentation to help you get started, and ask follow-up questions on irc or the webmaker-dev list.

1. Set up a Webmaker Development Environment

Before you can test or make improvements to Webmaker, you need to get it running locally. This involves setting up your OS to host the Webmaker servers and apps, and installing various development tools you'll need. We strongly recommend a Unix-like OS, such as Linux or OS X, but it is also possible to use Windows.

Dependencies

Webmaker relies on a number of technologies and tools, which must first be installed. The following list needs to be installed and working before you can run or test Webmaker locally:

  • node.js and npm
  • bower: after you install npm, you can type npm install -g bower (may require admin/sudo rights)
  • grunt: after you install npm, type npm install -g grunt-cli (may require admin/sudo rights)
  • python 2.7
  • pip

A number of node modules we use (e.g., SQLite) also require a working C/C++ toolchain. You should make sure that you have a working build environment for your OS (see the section of the Webmaker Suite docs on this issue).

Optionally, if you plan to work on, or run the MakeAPI locally, you will also require the following:

  • MongoDB (NOTE: there is some discussion about dropping MongoDB altogether, but at the time of writing, this hasn't happened yet)
  • Java
  • Elastic Search

Setup Option 1: Webmaker Suite

By far the easiest way to get all of Webmaker set up and installed locally, on any platform, is to use Webmaker Suite. This is a set of automation scripts written in node.js by one of our lead developers, and is meant to provide turn-key installation by automatically downloading all the code, and setting default environment variables.

Complete instructions are available in the README for Webmaker Suite on github.

Setup Option 2: Local Native Installation

Installing and running Webmaker on your local OS requires you to firstsetup

Setup Option 3: Using Vagrant (i.e., Ubuntu-based VM)

TODO

Setup Option 4: Heroku

TODO

2. Find or File a Webmaker Bug

All the work we do is tracked in Bugzilla, and knowing how to find exiting bugs and how to file new bugs is important.

Why Bugzilla?

This question comes up sometimes, and it's worth understanding our reasons for choosing Bugzilla over Github Issues or some other tool--we have gone through a long process of discussing and exploring other options, and Bugzilla is what works best for our use case.

Webmaker isn't developed as a single code-base. This is partly due to historic reasons (i.e., Webmaker was assembled out of different, existing projects) and partly due to our deployment needs (i.e., needing to be able to scale parts of Webmaker differently in production). Webmaker is also larger than the sum of its code, and encompasses a large teaching, mentoring, and event-based community, all of which is tied into the development of the tools and apps.

As a result of the highly distributed yet interconnected nature of the Webmaker code, taking a per-repository view of of the issues/bugs has proven difficult. First, a bug might touch multiple parts of the product, and need to be spread across many repositories--it's not uncommon for a single bug to touch 5 or 6 different repositories. Second, often a bug that appears to belong to a tool like Thimble actually needs to get filed and fixed in a repository that isn't named Thimble at all, and users (and many developers not familiar with the code) can't find their way through all the Webmaker repositories.

We have taken a philosophy that favours users and bug-fillers over developers, and one which tries to provide a wide net for collecting feedback. People filing Webmaker bugs, unless they are on the development team, typically don't know our code or where bugs belong. If a bug gets filed in Github Issues, it's not possible to move it to another repository. In Bugzilla this is trivial, and it means that we can triage bug reports, CC the right people, and get our work done faster across all of Webmaker. We also favour an approach that allows us to lump bugs about code, content, and community activities into a single bucket, since Webmaker is more than just HTML, CSS, and JavaScript.

Another advantage to using the Mozilla's Bugzilla has been the opportunity to interface with other areas of Mozilla. A good example is the Security Team, who routinely helps us with security-critical bugs. In Bugzilla one can flag these such that they aren't open to the public, and therefore don't document exploits. This isn't (currently) possible in Github.

When it comes to issue trackers, there is no silver bullet, and it's impossible to please everyone. By using both Github and Bugzilla, the former for pull requests and code review, and the latter for global projecct issue tracking, we think we've struck the right balance.

Using Bugzilla

If you haven't used Bugzilla before, you need to create an account. A good first guide to Bugzilla for Webmaker users is available here: http://blog.webmaker.org/bugzilla. You should also watch Bugzilla for Humans, and read this post. There are many good resources on learning Bugzilla.

Bugzilla is organized into Products, which in turn have Components. All Webmaker bugs live under the Webmaker product, and in one of its components (see the complete list and description of each). Developers often watch particular components, which means an email will be sent for all bugs filed or updated in that component.

New bugs can be filed here: https://bugzilla.mozilla.org/enter_bug.cgi?product=Webmaker. There are many bugs across all Webmaker components. You can see all open bugs for webmaker.org, Popcorn Maker, Thimble, X-Ray Goggles, Make API, etc. or you can see all open Webmaker bugs. Here are NEW (i.e., bugs not yet assigned to anyone) that need someone to look into:

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


TODO:

  • good-first-bug flags?

3. Understand Webmaker Code

The Webmaker code is spread across a large number of repositories, modules, libraries, apps, and web sites. Trying to locate the code for a particular part of Webmaker can be a challenge. The following is a high level introduction to the code.

Technical Overview of Webmaker

Users typically begin at https://webmaker.org, and so our discussion will start there too. The webmaker.org site has four main features, including:

  1. allowing users to access Webmaker tools, including Popcorn Maker, Thimble, and X-Ray Goggles.
  2. allowing users to search for and browse various "makes," which are objects in the Make API made with one of the Webmaker tools, or by a third-party web tool.
  3. allowing users to create or find Webmaker Events, using a Google Maps driven sub-application often referred to as Webmaker Events or just Events.
  4. allowing users to create a Webmaker (and by extension, Persona) account.

Beginning with account creation, webmaker.org (and other tools via the common navigation header) connects to the Login server (i.e., login.webmaker.org). Webmaker uses Mozilla Persona for user authentication and sign-in. Additional code has been written on top of Persona to allow single-sign-on across all Webmaker tools and sites (e.g., webmaker-sso). When a new Webmaker account is created, the Persona email is stored, along with extra information about the user (e.g., name, email notification preferences, etc.). The username is also used to create a new (virtual) domain on makes.org (or mywebmaker.org in our staging environment). For example, user donald would get donald.makes.org. All published makes for this user (i.e., things created with Webmaker tools) will be available at username.makes.org/something. The choice to use makes.org for hosting user content was done in order to provide origin-isolation from our Webmaker apps and code, which lives on webmaker.org (or mofostaging.net in our staging environment).

The username.makes.org page is actually the Webmaker Profile app. Here all the user's makes are displayed, along with various widgets for creating new UI components in the profile. The Profile app, like webmaker.org, uses the Make API and Login servers in order to find all the makes for a particular user--in fact most Webmaker apps work this way. Communication between the Make API, Login server, and other node apps happens over HTTP APIs within in the production (or staging) deployment network.

Users create "makes" using Webmaker tools. All tools allow authenticated Webmaker users (i.e., must have account in the Login server) to publish what they make. Publishing means a number of things. First, user generated content from the tools (e.g., HTML) is saved to an Amazon S3 bucket. A unique URL is also created, such that users can access their content again via their makes.org domain. The Make Valet app provides the routing and logic to map a URL to the content in S3, and common UI for details about a make (i.e., metadata from the Make API). Second, publishing means inserting a record into the Make API, which includes metadata about the make, like URL, date info, which tool was used, tags, etc. The Make API indexes makes using Elastic Search so that they can be found again easily.

Each of the Webmaker tools is a complex thing in its own right. Popcorn Maker publishes embeddable web pages, which are meant to be included via iframes. Thimble allows arbitrary HTML, CSS, and JavaScript to be combined into a single page. X-Ray Goggles allows the live DOM of a web page to be altered and then re-serialized to HTML so it can be published. All of the tools use the Login server for user authentication, S3 for publishing (each app does this step on its own, and in slightly different ways), and the Make API for indexing and metadata storage. Where possible common libraries and modules have been written and are shared across the tools, apps, and servers.

A Tour of Webmaker Code

For the most part, Webmaker code lives in one of many repositories hosted under the Mozilla Github Organization. We also contribute to a number of upstream projects, not all of which are listed below (e.g., node.js modules or libraries we use, but don't maintain directly). There are a few exceptions, but we try to graduate repositories we rely on to the Mozilla Organization instead of hosting them under individual github accounts. For more information about any of these repositories, you are encouraged to consult each their READMEs.

Webmaker Style Guide

Anyone working on Webmaker's front-end, or otherwise altering aspects of user-facing UI/UX should be aware of the Webmaker Style Guide. The Style Guide provides important information about Webmaker's design philosophy and branding, typography, logos, colours, thumbnails, etc.

Localization (l10n)

Webmaker is a global product, focused as much on non-English speakers as English, and used around the world. We have localized our apps and tools using a mix of server-side and client-side localization. The details about how we do this is available in the Webmaker I18N README.

If you are interested in localizing Webmaker, you can join the Webmaker Transifex project, more details are available in this SUMO article.

Even if you don't plan to translate strings, you'll probably have to make yourself aware of how localization works, since any change to code that touches the front-end or UI will likely trigger an l10n change. See the workflow section for more details.

4. Learn the Webmaker Development Workflow

Every project has its own way of working, and we're no exception. This guide will help you navigate our development workflow so you can be successful as a new contributor.

1. Get a bug Filed and Assigned to you

Fixing bugs or adding new features begins in Bugzilla. Whatever you are going to work on, you need to have a bug. Bugs in Bugzilla don't just mean something is broken. A bug is a unit of work, and that might mean fixing a problem in code, but could also mean adding something new, configuring a system differently, setting up an account for someone, etc.

If there isn't a bug filed for the work you'll be doing, you should file a new one. If you're not sure which component it belongs in, you can use General, and someone will help you move it to the right place.

Once a bug exists, you need to get it assigned to you. This is important because it lets other people in the community know that someone is already working on this. When you create a new Bugzilla account, you won't have privileges yet to assign bugs to yourself. However, if you ask on irc or the the webmaker-dev list, someone will gladly help you out.

2. Fix the bug and create a Pull Request

All Webmaker code is in github. You'll need intermediate level understanding of git and github to work on Webmaker, and if you ever get stuck, make sure you ask for help. There are also many good resources online for learning more about git, for example the Pro Git, online book.

A typical workflow looks like this:

  1. Fork the appropriate repo into your own github account
  2. Clone your fork locally: git clone --recursive {url to your github repo}. NOTE: many Webmaker repos have submodules, so it's wise to clone recursively.
  3. Add Mozilla's repo as an upstream remote, so you can stay in sync: git remote add upstream {url to Mozilla's github repo}. NOTE: you can call it anything you like, but upstream is a common convention.
  4. Create a new branch off master, using the bug number as the name: git checkout -b bug12345
  5. Work on the code, committing as you go. Don't worry about keeping your branch's commit log tidy, since we rebase and squash before merging into master.
  6. When you're at a point where you'd like feedback or review (see below for more details), push your branch up to your forked repo: git push origin bug12345
  7. Create a Pull Request in github from your bug12345 branch to Mozilla's master branch.
  8. Copy the URL for your pull request, for example: https://github.com/mozilla/webmaker.org/pull/420
  9. Go back to Bugzilla, and your bug, and Add an attachment. You will be given the option to "Enter the path to the file on your computer (or paste text as attachment)", and since we don't have a file, click paste text as attachment. Now paste your pull request URL into the textbox.
  10. For the attachment's Description, once again paste the pull request URL.
  11. Under Flags you have a number of options. If your code is ready for review, you can select ? in the drop-down beside review and then enter the name of the person (i.e., bugzilla email address) that you'd like to look at this code. If it's not ready for review, but you would like feedback, you can do the same thing with feedback. See the section on code review for more details
  12. Click Submit and wait for your reviewer to carry out the review.

3. Get your code Reviewed

Code review is a critical component of the Mozilla project. Code review increases code quality, distributes knowledge of changes across a team, allows for mentorship of new developers, etc. All changes in Webmaker need review, from a single line change to a 5,000 line feature addition. There are no exceptions, and no one is "good enough" to skip having their code reviewed.

In order to get your code reviewed, you first need to find an appropriate reviewer. There are a few strategies. The first is to ask in the bug, on irc, or the dev list. The second is to look at git blame {filename} information, which will show you who touched the code you're working in last, and often who did the review (e.g., the commit message or bug that changed it will have that info).

Your reviewer will use a mixture of inline Github code comments and comments in the Bugzilla bug. Generally speaking, issues relating to the code itself will go in Github, and general comments in Bugzilla. As your code is being reviewed, you will likely be asked to make fixes. This will include fixing errors, adding comments, adhering to prevailing styles, etc. Here is how you do it:

  1. Go back to your bug fix branch: git checkout bug12345
  2. Make your changes, commit, and push to your remote repo. The existing pull request will get updated.
  3. If your reviewer has given you an r-, you may need to ask for review again. You can do this by clicking Details beside your attachment in Bugzilla, then resetting the review flag to ? from -. Don't update your branch to the latest code in Mozilla's master branch yet--we'll do that when it's ready to land and we rebase (see below).

Don't be discouraged when this process takes multiple tries. All developers make mistakes, or lack knowledge of the entire project sufficient to catch issues outside the particular code in question. It's common for a bug to take many updates before the code is ready to get checked in (aka., landed). When it's time to land your code, your reviewer will likely ask you to rebase. It is highly recommended that you ask for help your first time, unless you have experience doing this, as it will alter your branch's commit history. We prefer rebase over merge so that our repositories have clean, single-change master branches, which makes reverting and bisecting much easier. A rebase generally goes like this:

  1. Switch to your master branch and pull the latest changes from Mozilla: git checkout master && git pull upstream master
  2. Switch back to your bug branch: git checkout bug12345
  3. Do a rebase, and ideally an interactive rebase so you can squash and fix your commit message: git rebase master -i. Now your editor will open and you can squash or fixup your separate commits, and redo your original commit message if you don't like it. A common commit message form is: Fix Bug 12345: Details about this bug r=person_who_did_review.
  4. Once the rebase is finished, and any conflicts dealt with, push your branch back up to github. You'll need to use the -f flag in order to force github to take it, since your history has now changed: git push origin bug12345 -f

At this point you can ask your reviewer to land your changes, either via irc, or in a comment in the bug.

TODO: document whiteboard flags

  • l10n changes
  • sumo or other doc changes
  • other?

4. Verify your code on Staging

TODO

  • tagging
  • pushing to staging/production
9337213776_d77d88eb89.jpg
Retrieved from "https://wiki.allizom.org/index.php?title=Webmaker/Code&oldid=727311"