Code Coverage

Code coverage is a form of testing designed to identify areas of code that have not been covered by tests.

Code Coverage

Here's an example from NUnit:

Code Coverage

NCover.Reporting.exe ... let you specify a minimum acceptable coverage for your project or for any part of it. If an element's coverage is below that minimum acceptable coverage percentage, NCover.Reporting.exe will ... make your build fail.

Code Coverage

• A quantitative metric for judging tests
• Is it really production ready?

Existing Approaches

• JSCoverage, node-jscoverage, JSCover
• JSTestDriver
• Yahoo/Istanbul

JSCoverage, node-jscoverage, JSCover

• Executable written in Java.
• Need to run a command to instrument files
• Creates wrapper for test runner (in browser)
• node-jscoverage needs to be compiled with make, and need to change source files

node-jscoverage setup

module.exports = process.env.EXPRESS_COV
   ? require('./lib-cov/express')
   : require('./lib/express');

JSTestDriver

• Also runs using Java.
• Is based on JUnit, so you need to use an adapter for QUnit
• Works as a server, but no asynchronous tests
• A lot of setup and configuration
• More of a test farm solution, then a development tool

Yahoo/Istanbul

• 100% JavaScript!
• Still creates instrumented files
• No way of instrumenting files in the browser

Yahoo/Istanbul

Usage in a browser
Load esprima.js, escodegen.js and instrumenter.js (this file) using script tags or other means.
...
Aside from demonstration purposes, it is unclear why you would want to instrument code in a browser.

Load source files

In the browser, we bundle a copy of requirejs and use a modified version of requirejs.load. We load the script file and pass the raw source to our instrumenting function.

Custom requirejs loader

requirejs.load = function (context, moduleName, url) {
 ...
 requirejs.cget(url, function (content) {
   blanket.instrument({
     inputFile: content,
     inputFileName: url
   },function(instrumented){	
	   blanketEval(instrumented);
	    context.completeLoad(moduleName);
     }
   });
 });
};

Load source files

In NodeJS, we use require.extensions['.js'] to instrument all the source files as they are processed.

Custom Node Require loader

require.extensions['.js'] = function(module, filename) {
  if (filename.indexOf(subdir) > -1){
    var content = fs.readFileSync(filename, 'utf8');
    var dirname = path.dirname(filename);
      exports.blanket.instrument({
        inputFile: content,
        inputFileName: filename
      },function(instrumented){
          return eval(instrumented);
        });
  }else{
    oldLoader(module,filename);
  }
};

Parse source files

We've bundled a copy of the Esprima JavaScript parser to generate parse trees for the source files, and a copy node-falafel to modify the parse trees.

Esprima JavaScript Parser

Esprima is a high performance, standard-compliant ECMAScript parser written in ECMAScript
...
sensible syntax tree format, compatible with Mozilla Parser AST

Parse Tree Example

// Life, Universe, and Everything
var answer = 6 * 7;
						

Parse Tree Example

{ "type": "Program",
    "body": [{
      "type": "VariableDeclaration",
      "declarations": [{
        "type": "VariableDeclarator",
        "id": {
          "type": "Identifier",
          "name": "answer"
        },
        "init": {
          "type": "BinaryExpression",
          "operator": "*",
          "left": {
            "type": "Literal",
            "value": 6,
            "raw": "6"
          },
          "right": {
            "type": "Literal",
            "value": 7,
            "raw": "7"
          }
        }
      }],
      "kind": "var"
    }]
}

node-falafel

A lightweight node library to modify the parse tree.

node-falafel

var src = '(' + function () {
    var xs = [ 1, 2, [ 3, 4 ] ];
    var ys = [ 5, 6 ];
    console.dir([ xs, ys ]);
} + ')()';

var output = falafel(src, function (node) {
    if (node.type === 'ArrayExpression') {
        node.update('fn(' + node.source() + ')');
    }
});
console.log(output);

node-falafel

(function () {
    var xs = fn([ 1, 2, fn([ 3, 4 ]) ]);
    var ys = fn([ 5, 6 ]);
    console.dir(fn([ xs, ys ]));
})()

blanket instrumentation

blanket = {
  instrument: function(config, next){
    var inFile = config.inputFile;
    instrumented =  falafel(inFile,{loc:true}, checkForOneLiner);
    next(instrumented)
};

blanket instrumentation

var checkForOneLiner = function (node) {
  if (linesToAddTracking.indexOf(node.type) > -1){
    node.update(
      "_$blanket['"+inFileName+"']["+node.loc.start.line+"]++;\n"
         +node.source());
  }
};
/* which creates the following:  */
 _$blanket['test.js'][2]++;

Output Details

We store all the coverage details in a global variable _$blanket.
When the test runner is finished we pass that variable to the appropriate reporter

QUnit

We wrote code to append the data from the coverage variable onto the end of the test runner file.

Mocha

We assign _$blanket to _$jscoverage and then use the built-in mocha reporters to output to json or HTML.

Demo

This example can be seen on the Blanket.js website. We took the unit tests for Addy Osmani's Backbone Koans and added Blanket.js to the testrunner.

Existing Backbone Koans Test Runner

<script src="js/ext/qunit.js"></script>
...
<script src="js/koans/aboutEvents.js"></script>
<script src="js/koans/aboutModels.js"></script>
<script src="js/koans/aboutCollections.js"></script>
<script src="js/koans/aboutViews.js"></script>
<script src="js/koans/aboutApps.js"></script>
...
</head><body>
  <h1>Backbone Koans</h1>
  <h1 id="qunit-header">QUnit Test Suite</h1> 
  ...

Traditional JS Code Coverage

• We would need to install JSCoverage as an executable.
• Need to run a command line to instrument our files
• Change test runner to refer to instrumented files instead of original source files
• Run test runner inside JSCoverage frame

Backbone Koans Test Runner with Traditional JS Code Coverage

<script src="js/ext/qunit.js"></script>

...
<script src="js-cov/koans/aboutEvents.js" ></script>
<script src="js-cov/koans/aboutModels.js" ></script>
<script src="js-cov/koans/aboutCollections.js" ></script>
<script src="js-cov/koans/aboutViews.js" ></script>
<script src="js-cov/koans/aboutApps.js" >></script>
...
</head><body>
  <h1>Backbone Koans</h1>
  <h1 id="qunit-header">QUnit Test Suite</h1> 
  ...

Blanket.js Code Coverage

• Download Blanket.js, or reference it from a CDN
• Add data-cover attribute to scripts
• Coverage can easily be disabled by removing the reference to blanket.js
• Run test runner as usual

Backbone Koans Test Runner with Blanket.js

<script src="js/ext/qunit.js"></script>
<script src="blanket.js"></script>
...
<script src="js/koans/aboutEvents.js" data-cover></script>
<script src="js/koans/aboutModels.js" data-cover></script>
<script src="js/koans/aboutCollections.js" data-cover></script>
<script src="js/koans/aboutViews.js" data-cover></script>
<script src="js/koans/aboutApps.js" data-cover></script>
...
</head><body>
  <h1>Backbone Koans</h1>
  <h1 id="qunit-header">QUnit Test Suite</h1> 
  ...

Just show the demo already!

Ok, here it is: Blanket.js Demo.

What about the Node Demo!?

I don't have one prepared yet, but this is the process for coverage with node.

Traditional NodeJS Code Coverage

• Clone node-jscoverage repo and build with make (requires gcc)
• Need to run a command line to instrument our files
• Change testrunner/test files to refer to instrumented files instead of original source files

  require("src-cov/src1")

• Run test runner with mocha

Blanket NodeJS Code Coverage

• Install from NPM:

  npm install blanket

• Add the following code to the test runner:

  require("blanket")("/src/");

• Run mocha as usual:

  mocha -R html-cov > coverage.html

Travis-CI & Blanket.js

We mentioned continuous integration servers earlier.
Blanket.js official supports Travis-CI at this time.

Travis-CI

A hosted continuous integration service for the open source community.

Travis-CI

For projects using NPM, Travis CI will execute

npm test

travis-cov

• Mocha reporter designed to check if coverage data is below threshold
• Easy to configure, just add the threshold value into your package.json
  

  scripts: {
    "travis-cov": {
      threshold: 75
    }
  }

• Easy to use, works just like any mocha reporter:
  

  mocha -R travis-cov

travis-cov & blanket.js

Blanket.js default test is:

mocha -R travis-cov test-node/testrunner.js && phantomjs phantom_runner.js test/testrunner.js