Blanket.js

Seamless JavaScript Code Coverage

An open source project from Migrii / @alex_seville, @dervalp, @flrent

Who am I?

  • Developer at Nurun
  • Co-Founder of Migrii
  • Travel Enthusiast

Disclaimer

Blanket.js is NOT production ready.
It is beyond the proof-of-concept stage, but is still NOT stable.
Use at your own risk.

What we'll cover

  • What is code coverage and why is it important?
  • Existing approaches for frontend code coverage
  • Blanket.js approach
  • Demo of Blanket.js!
  • Travis-CI & Blanket.js
  • The future of Blanket.js

Code Coverage

Code coverage is a form of testing designed to identify areas of code that have not been covered by tests.

Code Coverage

Here's an example from NUnit:

Code Coverage

NCover.Reporting.exe ... let you specify a minimum acceptable coverage for your project or for any part of it. If an element's coverage is below that minimum acceptable coverage percentage, NCover.Reporting.exe will ... make your build fail.
-NCover V3 Docs.

Code Coverage

  • A quantitative metric for judging tests
  • Is it really production ready?

Existing Approaches

  • JSCoverage, node-jscoverage, JSCover
  • JSTestDriver
  • Yahoo/Istanbul

JSCoverage, node-jscoverage, JSCover

  • Executable written in Java.
  • Need to run a command to instrument files
  • Creates wrapper for test runner (in browser)
  • node-jscoverage needs to be compiled with make, and need to change source files

node-jscoverage setup

module.exports = process.env.EXPRESS_COV
   ? require('./lib-cov/express')
   : require('./lib/express');
-Mocha Test Coverage.

JSTestDriver

  • Also runs using Java.
  • Is based on JUnit, so you need to use an adapter for QUnit
  • Works as a server, but no asynchronous tests
  • A lot of setup and configuration
  • More of a test farm solution, then a development tool

Yahoo/Istanbul

  • 100% JavaScript!
  • Still creates instrumented files
  • No way of instrumenting files in the browser

Yahoo/Istanbul

Usage in a browser
Load esprima.js, escodegen.js and instrumenter.js (this file) using script tags or other means.
...
Aside from demonstration purposes, it is unclear why you would want to instrument code in a browser.
-Instanbul API Docs.

Blanket Approach

  • Easy to install
  • Easy to use
  • Easy to understand
  • 100% JavaScript

Mechanism

  • Load source files
  • Parse them
  • Instrument them
  • Output coverage details

Load source files

In the browser, we bundle a copy of requirejs and use a modified version of requirejs.load. We load the script file and pass the raw source to our instrumenting function.

Custom requirejs loader


requirejs.load = function (context, moduleName, url) {
 ...
 requirejs.cget(url, function (content) {
   blanket.instrument({
     inputFile: content,
     inputFileName: url
   },function(instrumented){	
	   blanketEval(instrumented);
	    context.completeLoad(moduleName);
     }
   });
 });
};

Load source files

In NodeJS, we use require.extensions['.js'] to instrument all the source files as they are processed.

Custom Node Require loader


require.extensions['.js'] = function(module, filename) {
  if (filename.indexOf(subdir) > -1){
    var content = fs.readFileSync(filename, 'utf8');
    var dirname = path.dirname(filename);
      exports.blanket.instrument({
        inputFile: content,
        inputFileName: filename
      },function(instrumented){
          return eval(instrumented);
        });
  }else{
    oldLoader(module,filename);
  }
};

Parse source files

We've bundled a copy of the Esprima JavaScript parser to generate parse trees for the source files, and a copy node-falafel to modify the parse trees.

Esprima JavaScript Parser

Esprima is a high performance, standard-compliant ECMAScript parser written in ECMAScript
...
sensible syntax tree format, compatible with Mozilla Parser AST
-Esprima.org

Parse Tree Example


// Life, Universe, and Everything
var answer = 6 * 7;
						
From Esprima.org

Parse Tree Example

{ "type": "Program",
    "body": [{
      "type": "VariableDeclaration",
      "declarations": [{
        "type": "VariableDeclarator",
        "id": {
          "type": "Identifier",
          "name": "answer"
        },
        "init": {
          "type": "BinaryExpression",
          "operator": "*",
          "left": {
            "type": "Literal",
            "value": 6,
            "raw": "6"
          },
          "right": {
            "type": "Literal",
            "value": 7,
            "raw": "7"
          }
        }
      }],
      "kind": "var"
    }]
}
From Esprima.org

node-falafel

A lightweight node library to modify the parse tree.

node-falafel


var src = '(' + function () {
    var xs = [ 1, 2, [ 3, 4 ] ];
    var ys = [ 5, 6 ];
    console.dir([ xs, ys ]);
} + ')()';

var output = falafel(src, function (node) {
    if (node.type === 'ArrayExpression') {
        node.update('fn(' + node.source() + ')');
    }
});
console.log(output);
From Substack/node-falafel

node-falafel


(function () {
    var xs = fn([ 1, 2, fn([ 3, 4 ]) ]);
    var ys = fn([ 5, 6 ]);
    console.dir(fn([ xs, ys ]));
})()

blanket instrumentation


blanket = {
  instrument: function(config, next){
    var inFile = config.inputFile;
    instrumented =  falafel(inFile,{loc:true}, checkForOneLiner);
    next(instrumented)
};

blanket instrumentation


var checkForOneLiner = function (node) {
  if (linesToAddTracking.indexOf(node.type) > -1){
    node.update(
      "_$blanket['"+inFileName+"']["+node.loc.start.line+"]++;\n"
         +node.source());
  }
};
/* which creates the following:  */
 _$blanket['test.js'][2]++;

Output Details

We store all the coverage details in a global variable _$blanket.
When the test runner is finished we pass that variable to the appropriate reporter

QUnit

We wrote code to append the data from the coverage variable onto the end of the test runner file.

Mocha

We assign _$blanket to _$jscoverage and then use the built-in mocha reporters to output to json or HTML.

Demo

This example can be seen on the Blanket.js website. We took the unit tests for Addy Osmani's Backbone Koans and added Blanket.js to the testrunner.

Existing Backbone Koans Test Runner


<script src="js/ext/qunit.js"></script>
...
<script src="js/koans/aboutEvents.js"></script>
<script src="js/koans/aboutModels.js"></script>
<script src="js/koans/aboutCollections.js"></script>
<script src="js/koans/aboutViews.js"></script>
<script src="js/koans/aboutApps.js"></script>
...
</head><body>
  <h1>Backbone Koans</h1>
  <h1 id="qunit-header">QUnit Test Suite</h1> 
  ...

Traditional JS Code Coverage

  • We would need to install JSCoverage as an executable.
  • Need to run a command line to instrument our files
  • Change test runner to refer to instrumented files instead of original source files
  • Run test runner inside JSCoverage frame

Backbone Koans Test Runner with Traditional JS Code Coverage


<script src="js/ext/qunit.js"></script>

...
<script src="js-cov/koans/aboutEvents.js" ></script>
<script src="js-cov/koans/aboutModels.js" ></script>
<script src="js-cov/koans/aboutCollections.js" ></script>
<script src="js-cov/koans/aboutViews.js" ></script>
<script src="js-cov/koans/aboutApps.js" >></script>
...
</head><body>
  <h1>Backbone Koans</h1>
  <h1 id="qunit-header">QUnit Test Suite</h1> 
  ...

Blanket.js Code Coverage

  • Download Blanket.js, or reference it from a CDN
  • Add data-cover attribute to scripts
  • Coverage can easily be disabled by removing the reference to blanket.js
  • Run test runner as usual

Backbone Koans Test Runner with Blanket.js


<script src="js/ext/qunit.js"></script>
<script src="blanket.js"></script>
...
<script src="js/koans/aboutEvents.js" data-cover></script>
<script src="js/koans/aboutModels.js" data-cover></script>
<script src="js/koans/aboutCollections.js" data-cover></script>
<script src="js/koans/aboutViews.js" data-cover></script>
<script src="js/koans/aboutApps.js" data-cover></script>
...
</head><body>
  <h1>Backbone Koans</h1>
  <h1 id="qunit-header">QUnit Test Suite</h1> 
  ...

Just show the demo already!

Ok, here it is: Blanket.js Demo.

What about the Node Demo!?

I don't have one prepared yet, but this is the process for coverage with node.

Traditional NodeJS Code Coverage

  • Clone node-jscoverage repo and build with make (requires gcc)
  • Need to run a command line to instrument our files
  • Change testrunner/test files to refer to instrumented files instead of original source files
    require("src-cov/src1")
  • Run test runner with mocha

Blanket NodeJS Code Coverage

  • Install from NPM:
    npm install blanket
  • Add the following code to the test runner:
    require("blanket")("/src/");
  • Run mocha as usual:
    mocha -R html-cov > coverage.html

Travis-CI & Blanket.js

We mentioned continuous integration servers earlier.
Blanket.js official supports Travis-CI at this time.

Travis-CI

A hosted continuous integration service for the open source community.
From travis-ci.org

A free (!) continuous integration server instance for open source projects, and it's hooked right in to GitHub (Admin->Hooks).

Travis-CI

For projects using NPM, Travis CI will execute
npm test
From about.travis-ci.org, Building a Node.js project

travis-cov

  • Mocha reporter designed to check if coverage data is below threshold
  • Easy to configure, just add the threshold value into your package.json
    scripts: {
      "travis-cov": {
        threshold: 75
      }
    }
  • Easy to use, works just like any mocha reporter:
    mocha -R travis-cov

travis-cov & blanket.js

Blanket.js default test is:

mocha -R travis-cov test-node/testrunner.js && phantomjs phantom_runner.js test/testrunner.js

If any Pull request drops the coverage below the threshold the build will fail.

GitHub allows us to see the result of the travis-ci build before merging the Pull Request (!).

Other Features

  • Custom reporters and API to create your own
  • Works with existing requirejs based tests
  • Adapters to work with other test runners (Jasmine) and API to create your own

Future of Blanket.js

  • Stable version (not far!)
  • Running with large test suites
  • <delusion of gradeur>
    Becoming the "official" code coverage tool of jQuery???
    </delusion of gradeur>

How to help

  • Download the project and use it
  • Submit issues, corrections, suggestions
  • Fork the repo on GitHub and submit Pull Requests
  • Spread the word!

Questions?


Migrii/blanket

@alex_seville