Taskcluster/Round Tuit Box

That's a great idea, let's do that when we get a round tuit!

All of these projects are long-term, in-depth work that is important for TaskCluster's future, but for which we don't have paid staff right now. These are *not* for beginners! We will be happy to help out, but if you want to get started on something here, please plan to be in it for the long haul.

Ideas

Auth Service Refactoring

This includes a few updates to the authentication service:

• optimization of DFA generation (simpler? smarter?)
• send scope sets to the auth service and it returns a boolean, rather than auth.authenticateHawk's return of the full set of user's scopes (which are getting large)
• security policies (a way to assert that user or client __ does *not* have scope __)
• parameterized roles:
  • 'project-admin:%': ['auth:create-client:project/%/*', ..]
  • With the added hack that: 'project-admin:*' -> ['auth:create-client:project/*, ...'] i.e., drop trailing slash/chars

Login Refactoring

Currently, taskcluster-login and taskcluster-tools are two different projects. Let's combine them into one site, with the login service invoked only via API calls. In the process, integrate more tightly with the Mozilla-IAM team, including allowing users to authenticate with their Github credentials.

Also, provide a means for users to increase their privileges through some sort of "sudo" mechanism

Similarly, allow users who have access to certain github or hg repositories to gain the privileges associated with those repositories.

TaskCluster To-Go

TaskCluster is pretty much a single-deployment service right now. While each of the individual services can be run independently, there's really no practical method of deploying another TaskCluster instance, say at another company or organization. This prevents certain users from adopting TaskCluster, as they are then beholden to Mozilla to continue to support the service.

Build a method to go from zero to a fully operational TaskCluster installation.

Queue Datamining

TaskCluster provides very little information on tasks that are in the queue right now, or on tasks that have been completed. It's difficult to even answer simple questions like "what portion of our tasks are gecko-related?"

Let's dump task information into a good database that would allow the right kind of flexible queries to answer questions like this.

Artifact Downloader

Downloading artifacts reliably is a constant problem. Every time someone decides to use a TaskCluster artifact for some purpose, they do the obvious: curl https://queue.taskcluster.net/v1/.... Then that download fails and their job explodes. So they add --fail, which doesn't actually do what it promises. So they rewrite in Python with urrlib2. Which raises unexpected exceptions, or fails to follow redirects, or times out too quickly. And so on.

Let's build a simple, reliable, automatically-retrying artifact-downloading client that is easy to deploy everywhere we need it (so, in a compiled language like rust or go) that we can then encourage all users to take advantage of.

Docker and Native Engines for TaskCluster-Worker

We have a bunch of worker implementations now. In theory, we want to converge on TaskCluster-Worker, but we haven't found the time, so we are still running Generic-Worker and Docker-Worker. So let's implement the functionality of those workers within TaskCluster-Worker, and then migrate all uses of the old workers and deprecate them.

New Provisioner Architecture

The AWS provisioner is specific to AWS, and mixes the concerns of predicting required load with bidding for and monitoring running EC2 instances. We would like to be able to support other clouds, as well as on-premises hardware, all of which have different behaviors. And we would like to be able to use more sophisticated load-prediction algorithms.

Guidelines

The ideas listed here should be multi-month projects, but should not last forever (so, "build XYZ", but not "maintain ABC indefinitely").

They should be reasonably well-defined, and not require blue-sky design or unproven technologies (so, no quantum computing, sorry).

Avoid linking to discussions. Link to (or just include here) succinct descriptions of the current thinking about the idea.