Webmaker/Code

NOTE: this page is under active development, and will be changing rapidly over the coming days. Apologies for any chaos you discover at the moment.

Introduction

Webmaker is about empowering everyone to become makers of the web instead of just users. It's a mix of web tools, teaching material, and a social platform for teaching, learning, and sharing what you make. And, it's a lot of fun to use. It's even more fun to develop!

Webmaker is built using modern web technologies including Node.js, HTML5, CSS, and JavaScript (lots of JavaScript!)--many of the same technologies Webmaker teaches. If you know these technologies, or are interested in learning them more deeply, you might be interested in helping us develop Webmaker.

Getting Involved

There are many ways you can get involved in building the Webmaker project. Here are a few ideas:

• If you know JavaScript, you could help fix bugs or add features to our servers (e.g., the MakeAPI), JS libraries and modules (e.g., popcorn.js or our localization tools), or web tools.
• If you know HTML/CSS, you could work on our web sites and web tools, for example webmaker.org, Popcorn Maker, Thimble, etc.
• If you know MySQL/MongoDB/ElasticSearch and are interested in databases and data, you could help us work on the MakeAPI or Login server.
• If you're interested in Bug Triage, you could help us keep the Webmaker Product on bugzilla neat and tidy.
• If you know how to speak more than one language, you could work on localization and help us maintain our localization code.
• If you know how to work on Firefox/FirefoxOS or another Mozilla product/project, you could help us find ways to integrate Webmaker, for example building browser addons.
• If you know about automation/scripting/packaging/deployment, you could work with developer operations group to help deploy and manage Webmaker.
• If you're good at finding issues, confirming bugs, or otherwise breaking things, you could help us test Webmaker code on our staging and production servers.
• If you know how to do UI/UX work, you could join our design and front-end group to design and implement new user-facing aspects of our sites and tools.

These are just a few ideas to get you thinking--there's many ways that someone who wants to work on Webmaker can get involved. If you're passionate and interested in getting started, let us know and we'll try to find something that fits your interests and skills.

Communication

In order to get started on Webmaker code, you need to get involved with our community. Here are some ways to introduce yourself and get help:

• IRC: The #webmaker channel on moznet is our primary dev channel. If you're new to irc, please see the docs on learning how to use it. NOTE: our development team is spread across the world, with primary activity happening in Toronto, Vancouver, London, Berlin, New York, and San Francisco. If you're in a timezone that doesn't overlap with these, the mailing lists might be a better way to reach out.
• Mailing Lists: There are two primary lists: webmaker@lists.mozilla.org for general Webmaker communication; and webmaker-dev@mozilla.org for developer communication.
• Bug Tracker: We use Mozilla's Bugzilla to track our work, see the Webmaker Product in Bugzilla.
• Blog: Updates about the development of the Webmaker Product go on our Webmaker blog.

Developing Webmaker

Webmaker is a big project, and it can take some time to get set up and contributing. The following is a guide to our workflow, technologies, repositories, etc. Please read this documentation to help you get started, and ask follow-up questions on irc or the webmaker-dev list.

1. Set up a Webmaker Development Environment

Before you can test or make improvements to Webmaker, you need to get it running locally. This involves setting up your OS to host the Webmaker servers and apps, and installing various development tools you'll need. We strongly recommend a Unix-like OS, such as Linux or OS X, but it is also possible to use Windows.

Dependencies

Webmaker relies on a number of technologies and tools, which must first be installed. The following list needs to be installed and working before you can run or test Webmaker locally:

• node.js and npm
• bower: after you install npm, you can type npm install -g bower (may require admin/sudo rights)
• grunt: after you install npm, type npm install -g grunt-cli (may require admin/sudo rights)
• python 2.7
• pip

A number of node modules we use (e.g., SQLite) also require a working C/C++ toolchain. You should make sure that you have a working build environment for your OS (see the section of the Webmaker Suite docs on this issue).

Optionally, if you plan to work on, or run the MakeAPI locally, you will also require the following:

• MongoDB (NOTE: there is some discussion about dropping MongoDB altogether, but at the time of writing, this hasn't happened yet)
• Java
• Elastic Search

Setup Option 1: Webmaker Suite

By far the easiest way to get all of Webmaker set up and installed locally, on any platform, is to use Webmaker Suite. This is a set of automation scripts written in node.js by one of our lead developers, and is meant to provide turn-key installation by automatically downloading all the code, and setting default environment variables.

Complete instructions are available in the README for Webmaker Suite on github.

Setup Option 2: Local Native Installation

Installing and running Webmaker on your local OS requires you to firstsetup

Setup Option 3: Using Vagrant (i.e., Ubuntu-based VM)

TODO

Setup Option 4: Heroku

TODO

2. Find or File a Webmaker Bug

All the work we do is tracked in Bugzilla, and knowing how to find exiting bugs and how to file new bugs is important.

Why Bugzilla?

This question comes up sometimes, and it's worth understanding our reasons for choosing Bugzilla over Github Issues or some other tool--we have gone through a long process of discussing and exploring other options, and Bugzilla is what works best for our use case.

Webmaker isn't developed as a single code-base. This is partly due to historic reasons (i.e., Webmaker was assembled out of different, existing projects) and partly due to our deployment needs (i.e., needing to be able to scale parts of Webmaker differently in production). Webmaker is also larger than the sum of its code, and encompasses a large teaching, mentoring, and event-based community, all of which is tied into the development of the tools and apps.

As a result of the highly distributed yet interconnected nature of the Webmaker code, taking a per-repository view of of the issues/bugs has proven difficult. First, a bug might touch multiple parts of the product, and need to be spread across many repositories--it's not uncommon for a single bug to touch 5 or 6 different repositories. Second, often a bug that appears to belong to a tool like Thimble actually needs to get filed and fixed in a repository that isn't named Thimble at all, and users (and many developers not familiar with the code) can't find their way through all the Webmaker repositories.

We have taken a philosophy that favours users and bug-fillers over developers, and one which tries to provide a wide net for collecting feedback. People filing Webmaker bugs, unless they are on the development team, typically don't know our code or where bugs belong. If a bug gets filed in Github Issues, it's not possible to move it to another repository. In Bugzilla this is trivial, and it means that we can triage bug reports, CC the right people, and get our work done faster across all of Webmaker. We also favour an approach that allows us to lump bugs about code, content, and community activities into a single bucket, since Webmaker is more than just HTML, CSS, and JavaScript.

Another advantage to using the Mozilla's Bugzilla has been the opportunity to interface with other areas of Mozilla. A good example is the Security Team, who routinely helps us with security-critical bugs. In Bugzilla one can flag these such that they aren't open to the public, and therefore don't document exploits. This isn't (currently) possible in Github.

When it comes to issue trackers, there is no silver bullet, and it's impossible to please everyone. By using both Github and Bugzilla, the former for pull requests and code review, and the latter for global projecct issue tracking, we think we've struck the right balance.

Using Bugzilla

If you haven't used Bugzilla before, you need to create an account. A good first guide to Bugzilla for Webmaker users is available here: http://blog.webmaker.org/bugzilla. You should also watch [blog.johnath.com/2010/02/04/bugzilla-for-humans/ Bugzilla for Humans], and read this post. There are many good resources on learning Bugzilla.

Bugzilla is organized into Products, which in turn have Components. All Webmaker bugs live under the Webmaker product, and in one of its components (see the complete list and description of each). Developers often watch particular components, which means an email will be sent for all bugs filed or updated in that component.

New bugs can be filed here: https://bugzilla.mozilla.org/enter_bug.cgi?product=Webmaker. There are many bugs across all Webmaker components. You can see all open bugs for webmaker.org, Popcorn Maker, Thimble, X-Ray Goggles, Make API, etc. or you can see all open Webmaker bugs. Here are NEW (i.e., bugs not yet assigned to anyone) that need someone to look into:

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

TODO:

• good-first-bug flags?

3. Understand Webmaker Code

The Webmaker code is spread across a large number of repositories, modules, libraries, apps, and web sites. Trying to locate the code for a particular part of Webmaker can be a challenge. The following is a high level introduction to the code.

Technical Overview of Webmaker

Users typically begin at https://webmaker.org, and so our discussion will start there too. The webmaker.org site has four main features, including:

1. allowing users to access Webmaker tools, including Popcorn Maker, Thimble, and X-Ray Goggles.
2. allowing users to search for and browse various "makes," which are objects in the Make API made with one of the Webmaker tools, or by a third-party web tool.
3. allowing users to create or find Webmaker Events, using a Google Maps driven sub-application often referred to as Webmaker Events or just Events.
4. allowing users to create a Webmaker (and by extension, Persona) account.

Beginning with account creation, webmaker.org (and other tools via the common navigation header) connects to the Login server (i.e., login.webmaker.org). Webmaker uses Mozilla Persona for user authentication and sign-in. Additional code has been written on top of Persona to allow single-sign-on across all Webmaker tools and sites (e.g., webmaker-sso). When a new Webmaker account is created, the Persona email is stored, along with extra information about the user (e.g., name, email notification preferences, etc.). The username is also used to create a new (virtual) domain on makes.org (or mywebmaker.org in our staging environment). For example, user donald would get donald.makes.org. All published makes for this user (i.e., things created with Webmaker tools) will be available at username.makes.org/something. The choice to use makes.org for hosting user content was done in order to provide origin-isolation from our Webmaker apps and code, which lives on webmaker.org (or mofostaging.net in our staging environment).

The username.makes.org page is actually the Webmaker Profile app. Here all the user's makes are displayed, along with various widgets for creating new UI components in the profile. The Profile app, like webmaker.org, uses the Make API and Login servers in order to find all the makes for a particular user--in fact most Webmaker apps work this way. Communication between the Make API, Login server, and other node apps happens over HTTP APIs within in the production (or staging) deployment network.

Users create "makes" using Webmaker tools. All tools allow authenticated Webmaker users (i.e., must have account in the Login server) to publish what they make. Publishing means a number of things. First, user generated content from the tools (e.g., HTML) is saved to an Amazon S3 bucket. A unique URL is also created, such that users can access their content again via their makes.org domain. The Make Valet app provides the routing and logic to map a URL to the content in S3, and common UI for details about a make (i.e., metadata from the Make API). Second, publishing means inserting a record into the Make API, which includes metadata about the make, like URL, date info, which tool was used, tags, etc. The Make API indexes makes using Elastic Search so that they can be found again easily.

Each of the Webmaker tools is a complex thing in its own right. Popcorn Maker publishes embeddable web pages, which are meant to be included via iframes. Thimble allows arbitrary HTML, CSS, and JavaScript to be combined into a single page. X-Ray Goggles allows the live DOM of a web page to be altered and then re-serialized to HTML so it can be published. All of the tools use the Login server for user authentication, S3 for publishing (each app does this step on its own, and in slightly different ways), and the Make API for indexing and metadata storage. Where possible common libraries and modules have been written and are shared across the tools, apps, and servers.

A Tour of Webmaker Code

For the most part, Webmaker code lives in one of many repositories hosted under the Mozilla Github Organization. We also contribute to a number of upstream projects, not all of which are listed below (e.g., node.js modules or libraries we use, but don't maintain directly). There are a few exceptions, but we try to graduate repositories we rely on to the Mozilla Organization instead of hosting them under individual github accounts. For more information about any of these repositories, you are encouraged to consult each their READMEs.

• Webmaker.org: https://github.com/mozilla/webmaker.org - the node.js app and front-end code for https://webmaker.org, including things like Events and the Web Literacy Standard.

• Thimble: a friendly real-time updating code editor, includes:
  • thimble.webmaker.org: https://github.com/mozilla/thimble.webmaker.org - the node.js app and main/shell front-end code.
  • friendlycode: https://github.com/mozilla/friendlycode/ - the code editor used by Thimble
  • slowparse: https://github.com/mozilla/slowparse - an HTML5 parser with extra metadata and error handling
  • togetherjs: https://github.com/mozilla/togetherjs/ - TogetherJS real-time collaboration library

• Popcorn Maker: a web-based media editing tool, includes:
  • popcorn.webmaker.org: https://github.com/mozilla/popcorn.webmaker.org - the node.js app and front-end for Popcorn Maker
  • popcorn.js: https://github.com/mozilla/popcorn-js - the Popcorn.js HTML5 media library
  • mediasync: https://github.com/mozilla/webmaker-mediasync - sync/query services for media APIs

• X-Ray Goggles: https://github.com/mozilla/goggles.webmaker.org - the node.js app and front-end bookmarklet code for X-Ray Goggles, a tool for visually exploring and changing the underlying structure of web pages

• Login (aka login.webmaker.org): https://github.com/mozilla/login.webmaker.org - node.js single-sign-on (SSO) server and identity provider for Webmaker apps, as well as front-end UI components for authentication, based on Mozilla's Persona

• MakeAPI: https://github.com/mozilla/makeapi - node.js metadata server for storing and indexing information about "makes" (e.g., things made on the web with Webmaker tools or 3rd party tools).

• Make Valet: https://github.com/mozilla/make-valet - node.js app for hosting information about makes (e.g., Make Details).

• Webmaker Profile: https://github.com/mozilla/webmaker-profile - node.js app and front-end for username.makes.org profile pages, includes:
  • webmaker-profile-service: https://github.com/mozilla/webmaker-profile-service - RESTful API for profile

• Common Node/JS Modules used by many Webmaker apps and code:
  • webmaker-sso: https://github.com/jbuck/node-webmaker-sso - Single-Sign-On solution for Persona
  • makeapi-client: https://github.com/mozilla/makeapi-client - node.js and browser-js client lib for communicating with the MakeAPI
  • webmaker-i18n: https://github.com/mozilla/node-webmaker-i18n - Localization and Internationalization code for node.js and HTML.
  • webmaker-loginapi: https://github.com/mozilla/node-webmaker-loginapi - client lib for accessing Login server from node.js
  • webmaker-postalservice: https://github.com/mozilla/node-webmaker-postalservice - Webmaker mailer for apps requiring email services
  • mox-server: https://github.com/Pomax/mox-server - mock implementation of the Amazon S3 service in node.

TODO

• list of all our git repos, owners/peers

Webmaker Style Guide

Anyone working on Webmaker's front-end, or otherwise altering aspects of user-facing UI/UX should be aware of the Webmaker Style Guide. The Style Guide provides important information about Webmaker's design philosophy and branding, typography, logos, colours, thumbnails, etc.

4. Learn the Webmaker Development Workflow

Every project has its own way of working, and we're no exception. This guide will help you navigate our development workflow so you can be successful as a new contributor.

TODO:

• learning/using git
• bugzilla and github
• code review
• pull requests, inline comments
• rebasing patches when they are ready to land
• flags/whiteboard things to know about (e.g., l10n string changes)
• getting r+'ed patches landed
• tagging
• pushing to staging/production