Security/Projects/Minion: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (added zest and garmr plugins)
(Remove link to dead vm.)
 
Line 73: Line 73:
* We also use the #websectools channel on irc.mozilla.org
* We also use the #websectools channel on irc.mozilla.org


= Virtual Appliance =
= Minion Virtual Machine =


You can download a 64-bit Virtual Box VM for Minion '''[https://boily.me/assets/minion-730-x86_64.ova here]'''.
You can find instructions and tools to build a Minion virtual machine [https://github.com/mozilla/minion-vm/ on Github].
We provisioned this machine using [https://github.com/yeukhon/minion-bootstrap minion-bootstrap] and a vanilla Ubuntu 12.04.2 vagrant image.


'''VM Specification'''
= Developers =
 
'''username:''' vagrant
 
'''password:''' vagrant
 
'''default ip:''' 192.168.33.50
 
As we continue, we will upload a vagrant box and a vanilla ova. We will consider building one weekly.
 
== Getting Started with the VM ==
 
 
'''STEP 1: Log in to the VM'''
 
If this is your first time using the VM, or you have just restarted / powered-up the VM, you need to issue <code>startsuper</code> to get Minion up and running again.
<pre>
  $ ssh vagrant@192.168.33.50
  $ startsuper
</pre>
 
By default, we shipped the VM with [http://supervisord.org/ supervisor] to manage Minion servers and queue workers. The <code>startsuper</code> command is an alias that we have defined in <code>~/.bash_aliases</code>. We will explain other aliases later; let's move on to the next step.
 
 
'''STEP 2: Populate your Minion database'''
 
<pre>
  $ benv
  $ minion-db-init
</pre>
 
Minion is a Python application and for development purpose we have built a virtual environment for backend and frontend individually. <code>benv</code> is the name of the virtual environment for the backend. The second command <code>minion-db-init</code> is called to engage an interactive session to populate your database.
 
You need to provide answers to the following three questions. You need to have a Persona account because Minion uses Persona for authentication.
 
<pre>
Enter the administrator's Persona email:
Enter the administrator's name:
You have the options to import sites and groups:
[1] import security testing sites (default)
[2] import Mozilla sites
[3] import all
Enter the option number:
</pre>
 
You can choose to import any sites, but for development purpose you can import the first one. In the future we might remove the Mozilla option.
 
 
'''STEP 3: Visit your Minion on browser'''
 
You should now be able to login with your administrator account (the one you just filled out) by visiting:
 
<pre>
http://192.168.33.50:8080/
</pre>
 
== Aliases ==
 
Minion VM comes with a few handy aliases for common things such as sourcing into a virtual environment.
 
{| class="wikitable" cellpadding="10"
|'''alias'''
|'''purpose'''
|-
|minion
|cd into /opt/minion which holds various repos and scripts.
|-
|backend
|cd into minion-backend repo
|-
|frontend
|cd into minion-frontend repo
|-
|plugins
|cd into a directory currently holding a bunch of minion plugins.
|-
|benv
|alias for sourcing into backend's virtualenv.
|-
|fenv
|alias for sourcing into frontend's virtualenv.
|-
|super
|alias for calling supervisorctl. You can restart minion-backend process by calling <code>[[super restart minion-backend]]</code>, or <code>[[super stop all]]</code> to stop all running prcesses, just to name two.
|-
|startsuper
|alias to start supervisord (in order to start the console you must have the daemon running first).
|}
 
Most of the time you will be dealing with the virtual environments, super and startsuper. If you are not familiar with supervisor, you can get help by:
 
<pre>
$  super help
 
default commands (type help <topic>):
=====================================
add    clear  fg        open  quit    remove  restart  start  stop  update
avail  exit  maintail  pid  reload  reread  shutdown  status  tail  version
</pre>
 
To check all processes, you call
 
<pre>
$ super status
minion-backend                  RUNNING    pid 8500, uptime 2 days, 4:47:45
minion-frontend-server          RUNNING    pid 8340, uptime 2 days, 4:47:45
minion-plugin-worker            RUNNING    pid 8467, uptime 2 days, 4:47:45
minion-scan-worker              RUNNING    pid 8514, uptime 2 days, 4:47:45
minion-state-worker              RUNNING    pid 8468, uptime 2 days, 4:47:45
</pre>
 
Should you have more questions on how to use these aliases, please don't hesitate to contact us on <code>#websectools</code>.
 
== Log files ==
 
All log files are under <code>/var/log/supervisor</code>. Most of the logs are logged into minion-*.stderr.log. <strike>We are currently fixing [https://github.com/mozilla/minion-frontend/issues/99 stderr log is not generated]</strike>.
 
'''(This issue is fixed by editing /etc/supervisord/conf.d/minion-frontend.supervisor.conf to give stderr_log the same parent path as stdout_log)'''
 
To get around with this problem, as a developer I normally do this after starting up all minion services (using <code>startsuper</code>):
 
<pre>
$ fenv && super stop minion-frontend-server && minion-frontend -a 0.0.0.0 -d
</pre>
 
== Development workflow ==
 
There is an additional complexity when you deal with our aliases and our supervisord. Here is the plain, vanilla guide on how to get Minion running:
 
<pre>
$ ssh vagrant@192.168.33.50
$ benv && minion-backend-api runserver -d  (terminal #1)
$ benv && minion-plugin-worker (terminal #2)
$ benv && minion-scan-worker (terminal #3)
$ benv && minion-state-worker (terminal #4)
$ fenv && minion-frontend runserver -d -a 0.0.0.0  (terminal #5)
</pre>
 
Some might find this workflow easier!
 
I (yeukhon) has my own development workflow relying on supervisor with three terminals:
 
<pre>
$ ssh vagrant@192.168.33.50
 
  --- assume I have database setup ---
 
$ tail -f /var/log/supervisor/minion-backend.stderr.log      (do this on terminal #1)
$ startsuper && super restart all && fenv && super stop minion-frontend-server && minion-frontend -a 0.0.0.0 -d      (do this on terminal #2)
$ tail -f /var/log/supervisor/minion-plugin-worker.stderr.log    (do this on terminal #3)
 
  --- now do some editing, then ---
 
$ startsuper && super restart all && fenv && super stop minion-frontend-server && minion-frontend -a 0.0.0.0 -d      (do this on terminal #2)
</pre>
 
The first will watch the backend log, the second will start all minion services, stop only the frontend server, and run the frontend server without daemonizing it. The last tail will watch the plugin log. This log is helpful when you are developing a plugin. When I am done editing I usually restart all and run the frontend without daemonizing it again.
 
Again, don't force yourself to adopt to either workflow. Whatever works for you, that's the best workflow!
 
== Developers ==


You can find most of us on #websectools or on Github.
You can find most of us on #websectools or on Github.


* April King (april)
* Stefan Arentz (st3fan)
* Stefan Arentz (st3fan)
* Simon Bennetts (psiinon)
* Simon Bennetts (psiinon)
Line 247: Line 87:
* Matthew Fuller
* Matthew Fuller
* Mark Goodwin (mgoodwin)
* Mark Goodwin (mgoodwin)
* Yvan Boily (yvan)




Latest revision as of 18:35, 27 August 2015

Minion Overview

Minion is an open source Security Automation platform. The 0.3 release of Minion allows Development, QA, and Security team members to perform automated web security scans with a set of tools, and re-execute those scans as needed.

The 0.3 release incorporates significant changes, including a migration away from Django and Bootstrap to a Flask and Angular.js based application. It also involves significant improvements in back-end performance and scaling, and an updated plugin architecture.

This is the first Minion release that is ready for large scale adoption with access management features to constrain which users can access scan results, and an invitation system to actively engage new users.

Minion Principles

Minion should be easy to use.

A user should be able to log into Minion and immediately start to launch scans against sites available to them. Results should be easy to discover and contain detailed explanations.

The target audience for Minion should be developers

Developers should be able to use Minion the moment they have code written that can be tested. The initial focus will be on supporting web applications and services, but this should extend to other applications with the cohort feature set. The initial guidance surfaced by the default reporting engine should be useful, accurate, and actionable by developers, and meaningful to other audiences (QA, Security, Management).

Minion is a platform, not a security tool

Minion is an extensible platform that allows automation of security tasks. As such the focus should be on providing strong abstractions and a reliable, extensible platform without binding the platform to a specific suite of tools. All security testing functionality should be external to Minion and implemented via plugins.

Minion is scalable and extensible

It should be simple to add support for unsupported tools. In addition to the ease with which plugins can be authored, the plugin architecture must isolate plugin related issues from the core platform and allow delegation out to separate infrastructure, and must support scaling from small teams to enterprises that may deliver services to third parties.

Minion is a secure tool

Ongoing security testing and security assessment should be performed on Minion. Major releases and milestones should be accompanied with a detailed explanation of data collected, and risks or threats posed by the tools and plugins that are shipped by the core development team.

Minion is open

Minion is an open source platform, and the core team actively encourages developers, security professionals, and interested parties to provide feedback and feature requests.

Architecture

Minion-03-diagram-draft.png

The Minion Front-End is an Angular.js based application that invokes a set of API on the backend. The backend is comprised of a set of services that are collectively referred to as the Task Engine, and a set of plugins. The architecture is designed to be distributed so the plugins can run on any platform that can expose the correct API, ensuring that tools can be incorporated into Minion regardless of operating system dependencies.

Roadmap

Q3 2013

  • Site Ownership Verification
  • Results Reporting Improvements

Q4 2013

  • Reporting Plugins
  • Landing Pages
  • Deferred Execution Plugins
  • Scan Intensity Level

Q1 2014

  • Cohort PoC
  • Historical Issue Tracking
  • Interpolation Engine
  • Common Configuration Schema (Extending DEX-JSON)

Wish List

  • Site and User Data Privacy
  • Minion Event Model Extensions (simple extensibility)
  • Scramble - interactive script for generating plugins

Details

This information is preserved from an early iteration of the project. The information is obsolete and in most cases invalid, but the core concepts remain the same.

Links

All of the following are publicly accessible:

Minion Virtual Machine

You can find instructions and tools to build a Minion virtual machine on Github.

Developers

You can find most of us on #websectools or on Github.

  • April King (april)
  • Stefan Arentz (st3fan)
  • Simon Bennetts (psiinon)
  • Yeuk Hon Wong (yeukhon)
  • Matthew Fuller
  • Mark Goodwin (mgoodwin)
  • Yvan Boily (yvan)


List of plugins

The following projects are optional plugins for minion that add more functionality or wrap existing tools: 

   https://github.com/mozilla/minion-zap-plugin
   https://github.com/mozilla/minion-ssl-plugin
   https://github.com/mozilla/minion-skipfish-plugin
   https://github.com/mozilla/minion-nmap-plugin
   https://github.com/mozilla/minion-breach-plugin
   https://github.com/mozilla/minion-garmr-plugin
   https://github.com/mozilla/minion-zest-plugin
Retrieved from "https://wiki.allizom.org/index.php?title=Security/Projects/Minion&oldid=1092359"