Security/Projects/Minion: Difference between revisions

Jump to navigation Jump to search

Security/Projects/Minion (view source)

Revision as of 02:27, 5 August 2013

5,168 bytes added ,  5 August 2013
→‎Virtual Appliance
Line 74: Line 74:


==Virtual Appliance==
==Virtual Appliance==
You can download a Virtual Box VM for Minion '''[[https://boily.me/assets/minion-730-x86_64.ova here]]'''


Once you have imported and started the VM, you can SSH into the VM with the following username and password:
You can download a 64-bit Virtual Box VM for Minion '''[https://boily.me/assets/minion-730-x86_64.ova here]'''.
username: vagrant
We provisioned this machine using [https://github.com/yeukhon/minion-bootstrap minion-bootstrap] and a vanilla Ubuntu 12.04.2 vagrant image.
password: vagrant


Yes. This is a Vagrant VM. You can actually create one yourself from scratch using [https://github.com/yeukhon/minion-bootstrap]. Note this script is still in its early adoption stage so there might be bugs.
VM Specification


Proceed from login, you can find .bash_aliases contains several useful aliases.
'''username:''' vagrant


Do '''startsuper''' to start all the servers and workers. You can check supervisor managed processes by typing '''super status'''. This part is explained in [https://github.com/yeukhon/minion-bootstrap#aliases]
'''password:''' vagrant


Finally, if this is your first time, you can setup your database:
'''default ip:''' 192.168.33.50


$ benv && minion-db-init
As we continue, we will upload a vagrant box and a vanilla ova. We will consider building one weekly.


'''benv''' just enters the backend's virtualenv.
=== Getting Started with the VM ===


and then navigate to http://<vm_ip>:8080


(This is a pseduo documentation. A sphinx-based documentation should be up and running soon.)
'''STEP 1: Log in to the VM'''
 
If this is your first time using the VM, or you have just restarted / powered-up the VM, you need to issue <code>startsuper</code> to get Minion up and running again.
<pre>
  $ ssh vagrant@192.168.33.50
  $ startsuper
</pre>
 
By default, we shipped the VM with [http://supervisord.org/ supervisor] to manage Minion servers and queue workers. The <code>startsuper</code> command is an alias that we have defined in <code>~/.bash_aliases</code>. We will explain other aliases later; let's move on to the next step.
 
 
'''STEP 2: Populate your Minion database'''
 
<pre>
  $ benv
  $ minion-db-init
 
</pre>
 
Minion is a Python application and for development purpose we have built a virtual environment for backend and frontend individually. <code>benv</code> is the name of the virtual environment for the backend. The second command <code>minion-db-init</code> is called to engage an interactive session to populate your database.
 
You need to provide answers to the following three questions. You need to have a Persona account because Minion uses Persona for authentication.
 
<pre>
 
Enter the administrator's Persona email:
Enter the administrator's name:
You have the options to import sites and groups:
[1] import security testing sites (default)
[2] import Mozilla sites
[3] import all
Enter the option number:
</pre>
 
You can choose to import any sites, but for development purpose you can import the first one. In the future we might remove the Mozilla option.
 
 
'''STEP 3: Visit your Minion on browser'''
 
You should now be able to login with your administrator account (the one you just filled out) by visiting:
 
<pre>
http://192.168.33.50:8080/
</pre>
 
 
== Aliases ==
 
Minion VM comes with a few handy aliases for common things such as sourcing into a virtual environment.
 
{| class="wikitable" cellpadding="10"
|'''alias'''
|'''purpose'''
|-
|minion
|cd into /opt/minion which holds various repos and scripts.
|-
|backend
|cd into minion-backend repo
|-
|frontend
|cd into minion-frontend repo
|-
|plugins
|cd into a directory currently holding a bunch of minion plugins.
|-
|benv
|alias for sourcing into backend's virtualenv.
|-
|fenv
|alias for sourcing into frontend's virtualenv.
|-
|super
|alias for calling supervisorctl. You can restart minion-backend process by calling <code>[[super restart minion-backend]]</code>, or <code>[[super stop all]]</code> to stop all running prcesses, just to name two.
|-
|startsuper
|alias to start supervisord (in order to start the console you must have the daemon running first).
|}
 
Most of the time you will be dealing with the virtual environments, super and startsuper. If you are not familiar with supervisor, you can get help by:
 
<pre>
$  super help
 
default commands (type help <topic>):
=====================================
add    clear  fg        open  quit    remove  restart  start  stop  update
avail  exit  maintail  pid  reload  reread  shutdown  status  tail  version
</pre>
 
To check all processes, you call
 
<pre>
 
$ super status
minion-backend                  RUNNING    pid 8500, uptime 2 days, 4:47:45
minion-frontend-server          RUNNING    pid 8340, uptime 2 days, 4:47:45
minion-plugin-worker            RUNNING    pid 8467, uptime 2 days, 4:47:45
minion-scan-worker              RUNNING    pid 8514, uptime 2 days, 4:47:45
minion-state-worker              RUNNING    pid 8468, uptime 2 days, 4:47:45
 
</pre>
 
Should you have more questions on how to use these aliases, please don't hesitate to contact us on <code>#websectools</code>.
 
 
== Log files ==
 
All log files are under <code>/var/log/supervisor</code>. Most of the logs are logged into minion-*.stderr.log. We are currently fixing [https://github.com/mozilla/minion-frontend/issues/99 stderr log is not generated].
 
To get around with this problem, as a developer I normally do this after starting up all minion services (using <code>startsuper</code>):
 
<pre>
$ fenv && super stop minion-frontend-server && minion-frontend -a 0.0.0.0 -d
</pre>
 
== Development workflow ==
 
There is an additional complexity with you need to deal with our aliases and our supervisord. Here is the plain, vanilla guide how to get Minion running:
 
 
<pre>
$ ssh vagrant@192.168.33.50
$ benv && minion-backend-api runserver -d  (terminal #1)
$ benv && minion-plugin-worker (terminal #2)
$ benv && minion-scan-worker (terminal #3)
$ benv && minion-state-worker (terminal #4)
$ femv && minion-frontend runserver -d -a 0.0.0.0  (terminal #5)
</pre>
 
Some might find this workflow easier!
 
I (yeukhon) has my own development workflow relying on supervisor with three terminals:
 
<pre>
$ ssh vagrant@192.168.33.50
 
  --- assume I have database setup ---
 
$ tail -f /var/log/supervisor/minion-backend.stderr.log      (do this on terminal #1)
$ startsuper && super restart all && fenv && super stop minion-frontend-server && minion-frontend -a 0.0.0.0 -d      (do this on terminal #2)
$ tail -f /var/log/supervisor/minion-plugin-worker.stderr.log    (do this on terminal #3)
 
  --- now do some editing, then ---
 
$ startsuper && super restart all && fenv && super stop minion-frontend-server && minion-frontend -a 0.0.0.0 -d      (do this on terminal #2)
</pre>
 
The first will watch the backend log, the second will start all minion services, stop only the frontend server, and run the frontend server without daemonizing it. The last tail will watch the plugin log. This log is helpful when you are developing a plugin. When I am done editing I usually restart all and run the frontend without daemonizing it again.
 
Again, don't force yourself to adopt to either workflow. Whatever works for you, that's the best workflow!
 
 
== Developers ==


Developers:
* Stefan Arentz
* Stefan Arentz
* Simon Bennetts  
* Simon Bennetts  
Yeukhon
Confirmed users
16

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/689759"

Navigation menu