Architecture overview

Buster.JS consists of many small Git repositories/npm modules. We try to keep things small and separated. The repositories and installable modules both promotes reusability (e.g. you can use Buster assertions with any test framework) and helps us avoid tight coupling between modules. However, some people feel that the number of repositories are a bit daunting when trying to contribute. This document sheds some light on existing modules, what they’re for, and what role they play in the bigger part.

Note that some of our modules contain stronger abstractions than others, and the less obvious ones may very well change until we have something that we’re comfortable with. If you have suggestions for how certain modules can improve (renaming/merging/splitting/refactoring/whatever), feel free to let us know.

Types of modules

Buster.JS core modules

Core modules are those that are installed when you type npm install buster. They constitute the core of the test framework, and contains everything you need to run node tests, simple browser tests, and automated crowd-sourced browser tests.

Buster.JS auxilliary modules

Developer tools and docs.

Buster.JS extensions and optional modules

These modules provide additional and optional features/extensions for Buster.JS, such as linting, JsTestDriver support, and more.

Dependency graphs

Matthias Kling generated some visual dependency graphs. They’re a great complement to this article, visualizing how packages depend on each other.

https://github.com/meisl/buster-dev-tools/raw/dependency_graph/dependencyGraphs/buster-dependencies_normal.png

Core modules

buster

Status:
Stable
Source code:
buster
Build status:

Meta-package that’s the install target, thus carries quite a lot of dependencies. Includes some very rudimentary wiring across Buster repositories.

buster-analyzer

Status:
In development, may change significantly
Source code:
buster-analyzer
Build status:

A simple and generic mechanism for flagging warnings. The analyzer is an event emitter, and provides fatal, error and warning methods, which emit corresponding events. Additionally, the analyzer has a concept of ok/not ok. This is decided from a threshold (i.e., a threshold of “error” means “not ok” if any error or fatal events where flagged).

The analyzer also comes with a reporter which can be used to log events of interest. The analyzer and the reporter is used by buster-test-cli and extensions to provide various insight about your code. Examples of practical usage includes linting and syntax checking (for browser tests).

buster-autotest

Status:
TODO
Source code:
buster-autotest
Build status:

TODO Write description

buster-bayeux-emitter

Status:
Stable
Source code:
buster-bayeux-emitter
Build status:

Given subscribe and publish methods, this module produces an object that looks and behaves like an event emitter (i.e. no specific requirements on event names and so on).

Specifically, the bayeux emitter is used to allow the test runner in the browser (via ramp) ship its progress events directly over the wire (which is a Bayeux wire).

ramp

Status:
Unstable, currently undergoing API changes.
Source code:
ramp
Build status:

The capture server captures browsers as slaves, and offers a completely generic API for carrying out work across those slaves. A workload is known as a “session”, and a test run is typically a session. Other uses include for instance synced-across-devices slide shows (for which a POC has been built).

In general, the server knows nothing specifically of testing. It knows how to accept and server resource sets, capture and command browser slaves, and coordinate every piece using messaging (Bayeux on the HTTP level).

buster-cli

Status:
Stable
Source code:
buster-cli
Build status:

Somewhat arbitrary collection of utilities useful to CLIs that aim to behave more or less like existing Buster.JS CLIs. Is used by buster-test-cli and buster-static.

buster-configuration

Status:
Stable. Occasionally learns about new properties.
Source code:
buster-configuration
Build status:

Programmatic access to buster.js configuration files. Allows you to extract resourceSets (i.e. all the file names/contents), environment options, run extension hooks and filter out groups.

buster-core

Status:
Stable
Source code:
buster-core
Build status:

Somewhat arbitrary collection of functions used in several other buster modules. Includes the event emitter implementation used throughout, some limited flow-control utilities, and a few functional enhancements. Hopefully, we can get rid of this one day.

buster-evented-logger

Status:
Stable
Source code:
buster-evented-logger
Build status:

A logger-like utility that simply emits events. This is useful in any number of cases, most importantly when running tests in browsers via ramp. In this case, we pass the events over the wire instead of printing them to the console.

buster-format

Status:
Stable
Source code:
buster-format
Build status:

ASCII formatting of arbitrary JavaScript objects. This module is used to give pretty feedback in certain cases. It is used to format objects in assertion error messages, to format objects passed to buster.log (and console.log, if captured) and may be used in more places later. Also intended for reuse outside of the Buster.JS sphere.

buster-glob

Status:
TODO
Source code:
buster-glob
Build status:

TODO Write description

ramp-resources

Status:
Stable, has a few known issues waiting to be fixed
Source code:
ramp-resources
Build status:

Represents files in a project that may be included in a test run. For Node.js, Buster.JS only use ramp-resources to look up which paths to require. For browsers, Buster.JS uses ramp-resources to build a virtual file system, send it over HTTP and mount it on the server. All of these components are available in this module.

ramp-resources also includes intelligent caching of resources to allow test runs that only reads changed tests from file and so on. Resources typically map to files on disk, but really can be anything, including one-off strings in a configuration file.

buster-sinon

Status:
Stable
Source code:
buster-sinon
Build status:

Integrating Sinon.JS with Buster.JS. Adds Sinon specific assertions, wires up Sinon.JS to use buster-format for error messages, adds automatic sandboxing/restoration of fakes for test cases and so on.

buster-static

Status:
Largely incomplete
Source code:
buster-static
Build status:

A small and (currently, too) simple/limited way of easily running tests directly in a browser (i.e. without buster-server). Builds scaffolding markup and serves tests on a simple server.

stream-logger

Status:
Stable
Source code:
stream-logger
Build status:

Accepts a standard and error output stream, and returns a buster-evented-logger object that will print certains events directly to the passed-in stdout, and certain errors to stderr.

buster-syntax

Status:
Stable, but integrates with buster-analyzer, which is not.
Source code:
buster-syntax
Build status:

A small extension (but installed and activated by default) that provides server side syntax checking of scripts sent for testing with ramp. When Buster.JS loads scripts in browsers, the browser in question will be the one responsible for the level of detail when errors arise, Syntax checking on the server allows us to catch these errors in one place, and produce a pretty nice report, regardless of browser intended to run the tests.

buster-terminal

Status:
Stable, mostly complete.
Source code:
buster-terminal
Build status:

A small library for working with ANSI escape sequences in the terminal. Mostly used for colored output, and positional output. Also includes a “labeled list” object that is used to progressively print multiple lines of output at once, e.g. when running tests on multiple browsers (the dots reporter).

buster-test

Status:
May be split into several modules and/or renamed.
Source code:
buster-test
Build status:

Implements testCase, describe (and friends), the actual test runner, reporters and supporting objects. buster-test is centered around the concept of “test contexts”, which is just a bag of tests, and possibly more bags of tests. This is the shared data format produced by both the xUnit and BDD style tests/specs.

The reason we’re considering breaking up this module is that it includes parts that are complete and unlikely to change (such as test case and spec definitions, the context data format and the test runner) <strong>and</strong> parts that are likely to change and/or have abilities added, such as reporters. The name also indicates it is what powers the buster-test binary, which is not true.

buster-test-cli

Status:
In development
Source code:
buster-test-cli
Build status:

“The kitchen sink” behind buster test. Coordinates many other modules to read configuration file, loop all matching groups, creating runners and running those groups. In charge of options passed to buster test, colored printing and so on.

This module is mostly stable, but is rapidly gaining features and extension points. My gut feeling tells me that this module houses too many things, and it will likely be broken up before we go 1.0.

buster-user-agent-parser

Status:
Stable (new user agents occasionally added)
Source code:
buster-user-agent-parser
Build status:

A generic user-agent parser that does a best-effort attempt at extracting browser, version and platform. Only used for “friendly” browser names in test result reports, list of slaves and so on.

fs-watch-tree

Status:
TODO
Source code:
fs-watch-tree
Build status:

TODO Write description

posix-argv-parser

Status:
Stable, has a few known (API design) issues waiting to be fixed
Source code:
posix-argv-parser
Build status:

General purpose command line argument parser. Only parses command line options, no printing to the console, no --help generation, no flow control. Also tries as best it can to adhere to UNIX conventions. Fails early (typically when using non-existent options ++).

referee

Status:
Stable, awaiting a few additions before 1.0
Source code:
referee
Build status:

Assertions and expectations, for Buster.JS and everyone else.

Auxilliary modules

buster-dev-tools

Status:
In development, awaiting Windows support
Source code:
buster-dev-tools
Build status:

Allows developers to set up their Buster.JS development environment quickly and painlessly.

buster-docs

Status:
Will never be “done”
Source code:
buster-docs

You’re reading them.

buster-util

Status:
Stable, but will hopefully be removed down the line
Source code:
buster-util

Contains a simple and ugly test runner that’s used to test some of the more fundamental parts of Buster.JS.

Extensions and optional modules

buster-amd

Status:
In development
Source code:
https://github.com/johlrogge/buster-amd

Work in progress. Extension that will allow AMD projects to use Buster.JS without any specific configuration. It modifies the load path of the resourceSet used to represent user files and creates an anonymous AMD module that depends on all tests, thus loading files using an AMD loader rather than simple script tags. Currently developed by Joakim Ohlrogge.

buster-coffee

Status:
Stable, but future changes may be required to support Node.js and require()
Source code:
https://github.com/busterjs/buster-coffee
Build status:

Extension that automatically compile CoffeeScript files before running tests. In its current state, this extension does not work for files that are to be included using require(), and is thus not very useful for Node.js projects. Currently developed by Stein Magnus Jodal.

buster-coverage

Status:
In development
Source code:
https://github.com/ebi/buster-coverage

Work in progress. Extension to calculate line coverage. Uses the resourceSet to instrument code, and emits custom messages over the test runner to build up the report. Currently developed by Tobias Ebnöther.

buster-html-doc

Status:
Stable
Source code:
buster-html-doc
Build status:

An extension that implements “markup-in-comments”, using the /*:DOC el = ... */ format originally found in JsTestDriver. The extension was originally developed to be API compatible with JsTestDriver in the buster-jstestdriver extension, but works well with vanilla Buster.JS test cases (and specs) too.

buster-jstestdriver

Status:
Stable, but lacks async test cases.
Source code:
buster-jstestdriver
Build status:

An extension that allows Buster.JS to run JsTestDriver test suites, given a configuration file Buster.JS understands.

buster-lint

Status:
Stable, but relies on buster-analyzer, which is not.
Source code:
buster-lint

Extension that enables the integration of JsLint and JsHint by way of autolint. Using the buster-analyzer module, the lint extension is able to flag lint errors as “error” in buster. This allows the end-user to choose if lint errors should only be printed as warnings, or actually fail the build (which can be achieved with buster test -F error). Currently developed by Magnar Sveen <https://github.com/magnars>.

By example: buster test --browser

This section runs through what happens when you automate browser tests from the command line, e.g. when you type something like buster test --browser. The idea is to highlight roughly the flow through the various parts of Buster.JS, and to illustrate practically how the modules depend on and interact with each other.

In this section, files are referred to as package/path/to/file, meaning that buster/lib/buster/buster-wiring.js refers to the file lib/buster/buster-wiring.js in the buster package.

The binary

buster test executes the “binary” script buster/bin/buster This is a small wrapper script that can print some help, and that can look for other commands on the path called buster-<something>. In this case, it finds the buster/bin/buster-test script in the same package. The buster package is a “meta package”, meaning that it does not contain much implementation, it’s just there to glue all the pieces together and give you a convenient install target.

Command line options

The buster-test “binary” simply delegates to buster-test-cli/lib/buster-test-cli/cli/test.js which defines the CLI interface for running tests. Command line options are handled by posix-argv-parser, and to some extend, buster-cli.

buster-cli/lib/buster-cli.js is not so much a real abstraction, as it is a collection of routines useful in buster CLIs. It centralizes help text formatting, provides helpers for adding CLI options with help text, locates the configuration file and coordinates loading it with running.

The --browser option is a posix-argv-parser shorthand that expands to --environment browser, which is the long form for specifying environment.

Loading configuration

The configuration file is located and “resolved” in buster-cli/lib/buster-cli.js. It tries to find the configuration file in one of ./, ./test/ or ./spec/. If it is not found, the parent directory will be consulted in the same way until we’re at the root.

If the --config option was provided, only that file will be consulted. buster-cli contains some error handling in case configuration could not be located.

For loading the contents of the configuration file into memory, a separate package, buster-configuration, is used. Buster.JS defers actually reading source files from disks as long as possible, so “resolving” the configuration file only loads relevant groups with their extensions and builds lazy resource sets to represent files. buster-cli uses several options to filter out the groups found in the configuration file to figure out which ones will eventually be run.

Detour: Extension hooks

A configuration group has a method called runExtensionHook. You call this method with the name of a hook and some arguments. Any extension in the configuration that has a method of the same name will then be called with the passed in arguments.

Loading the browser runner

Depending on what groups resulted from reading the configuration file and filtering it according to command line options, the following steps may be repeated several times. For simplicity, this example assumes only one configuration was loaded.

Now that buster-test-cli knows that we’re running tests for the browser environment, it loads the browser runner. The runner will have its run method called with the configuration loaded from file, an options object, which contains prepared options for things like color etc, and a callback that will be called when the run is over.

The runner now uses a little abstraction that is shared between the browser and the node runner. It creates an analyzer for general-purpose health-checks, and fires the "beforeRun" extension hook, allowing extensions to register analyzers.

The browser runner then proceeds to instantiate a buster-client/lib/client.js. This object is a JavaScript interface that speaks HTTP to a running buster server. Because the server component is currently being reworked, this document will only briefly touch on the concepts it implements.

The server: A brief overview

The server has the ability to capture browsers as slaves. A slave is a browser that has loaded a frameset, where one frame, “the control frame”, keeps a persitent connection to the server, awaiting instructions. The server also provides an HTTP API for creating a session - a piece of work to be carried out in available slaves. When this happens, the server uses the bidirectional connection to instruct slaves to load the session in a separate frame.

When loading a session in a browser, an index.html file is loaded in a separate frame, and this file will include <script> tags that loads all the files originally specified in the configuration under libs, sources, testHelpers and tests.

The browser part of the server

Alongside the sources, a little wiring script is loaded. This file configures a listener for new test cases and specs Finally it wires up a test runner with a JSON proxy reporter and defines buster.run() as a way to start the whole thing. The test runner is completely evented, and the JSON proxy reporter is just a way of making sure the events are only data, thus HTTP-encodable. The events from the test runner are sent directly over the wire.

Back on the client

Back on the client, buster-test-cli is now ready to use its HTTP client to create a session. The client starts by asking the server for available cache manifests. These will be handed to the resourceSet, and will make sure Buster does not read any files from disk that are already hosted in the same version on the server.

Any file that isn’t already cached will now be “serialized” (i.e. read from disk) and sent to the server as part of the HTTP POST request to create a session. The server uses a resource set cache to cache and look up cached resources, and a resource set middleware to actually serve them over HTTP.

With the session readily created, buster-test-cli‘s browser runner is listening for messages. These messages are piped into its remote runner. This object accepts messages from multiple test runners (i.e. one per browser), and emits messages as if it was one test runner. The originating browser is represented as an outer context for all tests. This allows the “remote runner” to be used directly with any existing reporter written for the regular test runner.

The “testRun” extension hook

After the remote runner has been initialized, but before the tests are actually started, the client issues the "testRun" extension hook, which allows extensions to interact with the test runner (e.g. to listen to specific messages etc).

Finishing up

When the session is created and the remote runner is initialized, the browser runner will listen for the test runner’s "suite:end" event. This event comes with a short summary, which is passed to the browser runner’s done callback (passed to run).

The test.js CLI interface will now use the test report to decide if the run was successful or not and exit with a corresponding exit code. In the case where there are more configuration groups to be run, these will be run before exiting.

In summary

This is of course a brief and incomplete overview, but it should provide some insight into how some of the more important parts work together.