Architecture overview¶
Buster.JS consists of many small Git repositories/npm modules. We try to keep things small and separated. The repositories and installable modules both promotes reusability (e.g. you can use Buster assertions with any test framework) and helps us avoid tight coupling between modules. However, some people feel that the number of repositories are a bit daunting when trying to contribute. This document sheds some light on existing modules, what they’re for, and what role they play in the bigger part.
Note that some of our modules contain stronger abstractions than others, and the less obvious ones may very well change until we have something that we’re comfortable with. If you have suggestions for how certain modules can improve (renaming/merging/splitting/refactoring/whatever), feel free to let us know.
Types of modules¶
Buster.JS core modules
Core modules are those that are installed when you typenpm install buster
. They constitute the core of the test framework, and contains everything you need to run node tests, simple browser tests, and automated crowd-sourced browser tests.
Buster.JS auxilliary modules
Developer tools and docs.
Buster.JS extensions and optional modules
These modules provide additional and optional features/extensions for Buster.JS, such as linting, JsTestDriver support, and more.
Dependency graphs¶
Matthias Kling generated some visual dependency graphs. They’re a great complement to this article, visualizing how packages depend on each other.
Core modules¶
buster¶
- Status:
- Stable
- Source code:
- buster
- Build status:
Meta-package that’s the install target, thus carries quite a lot of dependencies. Includes some very rudimentary wiring across Buster repositories.
buster-analyzer¶
- Status:
- In development, may change significantly
- Source code:
- buster-analyzer
- Build status:
A simple and generic mechanism for flagging warnings. The analyzer is an event
emitter, and provides fatal
, error
and warning
methods, which emit
corresponding events. Additionally, the analyzer has a concept of ok/not ok.
This is decided from a threshold (i.e., a threshold of “error” means “not ok”
if any error or fatal events where flagged).
The analyzer also comes with a reporter which can be used to log events of interest. The analyzer and the reporter is used by buster-test-cli and extensions to provide various insight about your code. Examples of practical usage includes linting and syntax checking (for browser tests).
buster-bayeux-emitter¶
- Status:
- Stable
- Source code:
- buster-bayeux-emitter
- Build status:
Given subscribe
and publish
methods, this module produces an object
that looks and behaves like an event emitter (i.e. no specific requirements on
event names and so on).
Specifically, the bayeux emitter is used to allow the test runner in the
browser (via ramp
) ship its progress events directly over
the wire (which is a Bayeux wire).
ramp¶
- Status:
- Unstable, currently undergoing API changes.
- Source code:
- ramp
- Build status:
The capture server captures browsers as slaves, and offers a completely generic API for carrying out work across those slaves. A workload is known as a “session”, and a test run is typically a session. Other uses include for instance synced-across-devices slide shows (for which a POC has been built).
In general, the server knows nothing specifically of testing. It knows how to accept and server resource sets, capture and command browser slaves, and coordinate every piece using messaging (Bayeux on the HTTP level).
buster-cli¶
- Status:
- Stable
- Source code:
- buster-cli
- Build status:
Somewhat arbitrary collection of utilities useful to CLIs that aim to behave
more or less like existing Buster.JS CLIs. Is used by buster-test-cli
and
buster-static
.
buster-configuration¶
- Status:
- Stable. Occasionally learns about new properties.
- Source code:
- buster-configuration
- Build status:
Programmatic access to buster.js
configuration files. Allows you to extract
resourceSets
(i.e. all the file names/contents), environment options, run
extension hooks and filter out groups.
buster-core¶
- Status:
- Stable
- Source code:
- buster-core
- Build status:
Somewhat arbitrary collection of functions used in several other buster modules. Includes the event emitter implementation used throughout, some limited flow-control utilities, and a few functional enhancements. Hopefully, we can get rid of this one day.
buster-evented-logger¶
- Status:
- Stable
- Source code:
- buster-evented-logger
- Build status:
A logger-like utility that simply emits events. This is useful in any number of cases, most importantly when running tests in browsers via ramp. In this case, we pass the events over the wire instead of printing them to the console.
buster-format¶
- Status:
- Stable
- Source code:
- buster-format
- Build status:
ASCII formatting of arbitrary JavaScript objects. This module is used to give
pretty feedback in certain cases. It is used to format objects in assertion
error messages, to format objects passed to buster.log
(and
console.log
, if captured) and may be used in more places later. Also
intended for reuse outside of the Buster.JS sphere.
ramp-resources¶
- Status:
- Stable, has a few known issues waiting to be fixed
- Source code:
- ramp-resources
- Build status:
Represents files in a project that may be included in a test run. For Node.js,
Buster.JS only use ramp-resources
to look up which paths to require
.
For browsers, Buster.JS uses ramp-resources
to build a virtual file
system, send it over HTTP and mount it on the server. All of these components
are available in this module.
ramp-resources
also includes intelligent caching of resources to allow
test runs that only reads changed tests from file and so on. Resources
typically map to files on disk, but really can be anything, including one-off
strings in a configuration file.
buster-sinon¶
- Status:
- Stable
- Source code:
- buster-sinon
- Build status:
Integrating Sinon.JS with Buster.JS. Adds Sinon specific assertions, wires up
Sinon.JS to use buster-format
for error messages, adds automatic
sandboxing/restoration of fakes for test cases and so on.
buster-static¶
- Status:
- Largely incomplete
- Source code:
- buster-static
- Build status:
A small and (currently, too) simple/limited way of easily running tests directly in a browser (i.e. without buster-server). Builds scaffolding markup and serves tests on a simple server.
stream-logger¶
- Status:
- Stable
- Source code:
- stream-logger
- Build status:
Accepts a standard and error output stream, and returns a
buster-evented-logger
object that will print certains events directly to
the passed-in stdout, and certain errors to stderr.
buster-syntax¶
- Status:
- Stable, but integrates with
buster-analyzer
, which is not. - Source code:
- buster-syntax
- Build status:
A small extension (but installed and activated by default) that provides server
side syntax checking of scripts sent for testing with
ramp
. When Buster.JS loads scripts in browsers, the
browser in question will be the one responsible for the level of detail when
errors arise, Syntax checking on the server allows us to catch these errors in
one place, and produce a pretty nice report, regardless of browser intended to
run the tests.
buster-terminal¶
- Status:
- Stable, mostly complete.
- Source code:
- buster-terminal
- Build status:
A small library for working with ANSI escape sequences in the terminal. Mostly used for colored output, and positional output. Also includes a “labeled list” object that is used to progressively print multiple lines of output at once, e.g. when running tests on multiple browsers (the dots reporter).
buster-test¶
- Status:
- May be split into several modules and/or renamed.
- Source code:
- buster-test
- Build status:
Implements testCase
, describe
(and friends), the actual test runner,
reporters and supporting objects. buster-test
is centered around the
concept of “test contexts”, which is just a bag of tests, and possibly more
bags of tests. This is the shared data format produced by both the xUnit and
BDD style tests/specs.
The reason we’re considering breaking up this module is that it includes parts
that are complete and unlikely to change (such as test case and spec
definitions, the context data format and the test runner) <strong>and</strong>
parts that are likely to change and/or have abilities added, such as reporters.
The name also indicates it is what powers the buster-test
binary, which is
not true.
buster-test-cli¶
- Status:
- In development
- Source code:
- buster-test-cli
- Build status:
“The kitchen sink” behind buster test
. Coordinates many other modules to
read configuration file, loop all matching groups, creating runners and running
those groups. In charge of options passed to buster test
, colored printing
and so on.
This module is mostly stable, but is rapidly gaining features and extension points. My gut feeling tells me that this module houses too many things, and it will likely be broken up before we go 1.0.
buster-user-agent-parser¶
- Status:
- Stable (new user agents occasionally added)
- Source code:
- buster-user-agent-parser
- Build status:
A generic user-agent parser that does a best-effort attempt at extracting browser, version and platform. Only used for “friendly” browser names in test result reports, list of slaves and so on.
posix-argv-parser¶
- Status:
- Stable, has a few known (API design) issues waiting to be fixed
- Source code:
- posix-argv-parser
- Build status:
General purpose command line argument parser. Only parses command line options,
no printing to the console, no --help
generation, no flow control. Also
tries as best it can to adhere to UNIX conventions. Fails early (typically when
using non-existent options ++).
Auxilliary modules¶
buster-dev-tools¶
- Status:
- In development, awaiting Windows support
- Source code:
- buster-dev-tools
- Build status:
Allows developers to set up their Buster.JS development environment quickly and painlessly.
buster-util¶
- Status:
- Stable, but will hopefully be removed down the line
- Source code:
- buster-util
Contains a simple and ugly test runner that’s used to test some of the more fundamental parts of Buster.JS.
Extensions and optional modules¶
buster-amd¶
- Status:
- In development
- Source code:
- https://github.com/johlrogge/buster-amd
Work in progress. Extension that will allow AMD projects to use
Buster.JS without any specific configuration. It modifies the load path
of the resourceSet
used to represent user files and creates
an anonymous AMD module that depends on all tests, thus loading files
using an AMD loader rather than simple script tags. Currently developed by
Joakim Ohlrogge.
buster-coffee¶
- Status:
- Stable, but future changes may be required to support Node.js and
require()
- Source code:
- https://github.com/busterjs/buster-coffee
- Build status:
Extension that automatically compile CoffeeScript files before running tests.
In its current state, this extension does not work for files that are to be
included using require()
, and is thus not very useful for Node.js projects.
Currently developed by Stein Magnus Jodal.
buster-coverage¶
- Status:
- In development
- Source code:
- https://github.com/ebi/buster-coverage
Work in progress. Extension to calculate line coverage. Uses the
resourceSet
to instrument code, and emits custom messages over the test
runner to build up the report. Currently developed by Tobias Ebnöther.
buster-html-doc¶
- Status:
- Stable
- Source code:
- buster-html-doc
- Build status:
An extension that implements “markup-in-comments”, using the
/*:DOC el = ... */
format originally found in JsTestDriver. The extension
was originally developed to be API compatible with JsTestDriver in the
buster-jstestdriver
extension, but works well with vanilla Buster.JS test
cases (and specs) too.
buster-jstestdriver¶
- Status:
- Stable, but lacks async test cases.
- Source code:
- buster-jstestdriver
- Build status:
An extension that allows Buster.JS to run JsTestDriver test suites, given a configuration file Buster.JS understands.
buster-lint¶
- Status:
- Stable, but relies on
buster-analyzer
, which is not. - Source code:
- buster-lint
Extension that enables the integration of JsLint and JsHint by way of
autolint. Using the
buster-analyzer
module, the lint extension is able to flag lint errors as
“error” in buster. This allows the end-user to choose if lint errors should
only be printed as warnings, or actually fail the build (which can be achieved
with buster test -F error
). Currently developed by Magnar Sveen
<https://github.com/magnars>.
By example: buster test --browser
¶
This section runs through what happens when you automate browser tests from the
command line, e.g. when you type something like buster test --browser
. The
idea is to highlight roughly the flow through the various parts of Buster.JS,
and to illustrate practically how the modules depend on and interact with each
other.
In this section, files are referred to as package/path/to/file
, meaning
that buster/lib/buster/buster-wiring.js
refers to the file
lib/buster/buster-wiring.js in the buster package.
The binary¶
buster test
executes the “binary” script buster/bin/buster This is a
small wrapper script that can print some help, and that can look for other
commands on the path called buster-<something>
. In this case, it finds the
buster/bin/buster-test script in the same package. The buster
package is
a “meta package”, meaning that it does not contain much implementation, it’s
just there to glue all the pieces together and give you a convenient install
target.
Command line options¶
The buster-test
“binary” simply delegates to
buster-test-cli/lib/buster-test-cli/cli/test.js which defines the CLI
interface for running tests. Command line options are handled by
posix-argv-parser
, and to some extend, buster-cli
.
buster-cli/lib/buster-cli.js is not so much a real abstraction, as it is a collection of routines useful in buster CLIs. It centralizes help text formatting, provides helpers for adding CLI options with help text, locates the configuration file and coordinates loading it with running.
The --browser
option is a posix-argv-parser
shorthand that expands to
--environment browser
, which is the long form for specifying environment.
Loading configuration¶
The configuration file is located and “resolved” in
buster-cli/lib/buster-cli.js. It tries to find the configuration file in
one of ./
, ./test/
or ./spec/
. If it is not found, the parent
directory will be consulted in the same way until we’re at the root.
If the --config
option was provided, only that file will be
consulted. buster-cli
contains some error handling in case
configuration could not be located.
For loading the contents of the configuration file into memory, a separate
package, buster-configuration
, is used. Buster.JS defers actually reading
source files from disks as long as possible, so “resolving” the configuration
file only loads relevant groups with their extensions and builds lazy resource
sets to represent files. buster-cli
uses several options to filter out
the groups found in the configuration file to figure out which ones will
eventually be run.
Detour: Extension hooks¶
A configuration group has a method called runExtensionHook
. You call this
method with the name of a hook and some arguments. Any extension in the
configuration that has a method of the same name will then be called with the
passed in arguments.
Loading the browser runner¶
Depending on what groups resulted from reading the configuration file and filtering it according to command line options, the following steps may be repeated several times. For simplicity, this example assumes only one configuration was loaded.
Now that buster-test-cli
knows that we’re running tests for the browser
environment, it loads the browser runner. The runner will have its run
method called with the configuration loaded from file, an options object, which
contains prepared options for things like color etc, and a callback that will
be called when the run is over.
The runner now uses a little abstraction that is shared between the browser and
the node runner. It creates an analyzer for general-purpose health-checks,
and fires the "beforeRun"
extension hook, allowing extensions to register
analyzers.
The browser runner then proceeds to instantiate a buster-client/lib/client.js. This object is a JavaScript interface that speaks HTTP to a running buster server. Because the server component is currently being reworked, this document will only briefly touch on the concepts it implements.
The server: A brief overview¶
The server has the ability to capture browsers as slaves. A slave is a browser that has loaded a frameset, where one frame, “the control frame”, keeps a persitent connection to the server, awaiting instructions. The server also provides an HTTP API for creating a session - a piece of work to be carried out in available slaves. When this happens, the server uses the bidirectional connection to instruct slaves to load the session in a separate frame.
When loading a session in a browser, an index.html file is loaded in a separate
frame, and this file will include <script>
tags that loads all the files
originally specified in the configuration under libs
, sources
,
testHelpers
and tests
.
The browser part of the server¶
Alongside the sources, a little wiring script is loaded. This file
configures a listener for new test cases and specs Finally it wires up a
test runner with a JSON proxy reporter and defines buster.run()
as a
way to start the whole thing. The test runner is completely evented, and the
JSON proxy reporter is just a way of making sure the events are only data, thus
HTTP-encodable. The events from the test runner are sent directly over the
wire.
Back on the client¶
Back on the client, buster-test-cli
is now ready to use its HTTP client to
create a session. The client starts by asking the server for available cache
manifests. These will be handed to the resourceSet
, and will make sure
Buster does not read any files from disk that are already hosted in the same
version on the server.
Any file that isn’t already cached will now be “serialized” (i.e. read from disk) and sent to the server as part of the HTTP POST request to create a session. The server uses a resource set cache to cache and look up cached resources, and a resource set middleware to actually serve them over HTTP.
With the session readily created, buster-test-cli
‘s browser runner is
listening for messages. These messages are piped into its remote runner.
This object accepts messages from multiple test runners (i.e. one per browser),
and emits messages as if it was one test runner. The originating browser is
represented as an outer context for all tests. This allows the “remote runner”
to be used directly with any existing reporter written for the regular test
runner.
The “testRun” extension hook¶
After the remote runner has been initialized, but before the tests are actually
started, the client issues the "testRun"
extension hook, which allows
extensions to interact with the test runner (e.g. to listen to specific
messages etc).
Finishing up¶
When the session is created and the remote runner is initialized, the browser
runner will listen for the test runner’s "suite:end"
event. This event
comes with a short summary, which is passed to the browser runner’s done
callback (passed to run
).
The test.js
CLI interface will now use the test report to decide if the run
was successful or not and exit with a corresponding exit code. In the case
where there are more configuration groups to be run, these will be run before
exiting.
In summary¶
This is of course a brief and incomplete overview, but it should provide some insight into how some of the more important parts work together.