The Testing Game

Test Coverage

It’s important to test all your code. Code coverage is frequently used as a measuring stick for a developer’s success in creating quality tests. The basic rule of thumb is comprehensive tests should execute every line of code.

Coverage, a tool that measures code coverage for Python code, will be used to check what percentage of the tutorial code is being tested.

Intalling Coverage

First let’s install coverage:

$ pip install coverage

Before we continue, we need to remember to add this new dependency to our requirements.txt file. Let’s use pip freeze to discover the version of coverage we installed:

$ pip freeze

Now let’s add coverage to our requirements.txt file:


Using Coverage

Now let’s run our tests. As we run our tests from the command line, coverage records and creates a coverage report:

$ coverage run --include='./*' test blog
Creating test database for alias 'default'...
Ran 19 tests in 0.234s

Destroying test database for alias 'default'...

Let’s take a look at our code coverage report:

$ coverage report
Name              Stmts   Miss  Cover
blog/__init__         0      0   100%
blog/admin            4      0   100%
blog/forms           14      0   100%
blog/models          21      0   100%
blog/tests           87      0   100%
blog/urls             2      0   100%
blog/views           22      0   100%
manage                6      0   100%
myblog/__init__       0      0   100%
myblog/settings      29      0   100%
myblog/urls           5      0   100%
myblog/views          6      0   100%
TOTAL               196      0   100%

Let’s take a look at the coverage report. On the left, the report shows the name of the file being tested. Stmts, or code statements, indicate the number of lines of code that could be tested. Miss, or Missed lines, indicates the number of lines that are not executed by the unit tests. Cover, or Coverage, is the percentage of code covered by the current tests (equivalent to (Stmts - Miss)/Stmts). For example, myblog/views has 6 code statements that can be tested. We see that our tests did not miss testing any statements for a Code Coverage of 100%.


Note that code coverage can only indicate that you’ve forgotten tests; it will not tell you whether your tests are good. Don’t use good code coverage as an excuse to write lower quality tests.

HTML Coverage Report

Our current command-line coverage reports are useful, but they aren’t very detailed. Fortunately coverage includes a feature for generating HTML coverage reports that visually demonstrate coverage by coloring our code based on the results.

Let’s prettify the coverage report above into HTML format by running the following command:

$ coverage html

This command will create a htmlcov directory containing our test coverage. The index.html is the overview file which links to the other files. Let’s open up our htmlcov/index.html in our web browser.

Our HTML coverage report should look something like this:


Branch Coverage

So far we’ve been testing statement coverage to ensure we execute every line of code during our tests. We can do better by ensuring every code branch is taken. The coverage documentation contains a good description of branch coverage.

From now on we will add the --branch argument when we record code coverage. Let’s try it on our tests:

$ coverage run --include='./*' --branch test blog
$ coverage report
Name              Stmts   Miss Branch BrMiss  Cover
blog/__init__         0      0      0      0   100%
blog/admin            4      0      0      0   100%
blog/forms           14      0      0      0   100%
blog/models          21      0      0      0   100%
blog/tests           87      0      0      0   100%
blog/urls             2      0      0      0   100%
blog/views           22      0      0      0   100%
manage                6      0      2      1    88%
myblog/__init__       0      0      0      0   100%
myblog/settings      29      0      0      0   100%
myblog/urls           5      0      0      0   100%
myblog/views          6      0      0      0   100%
TOTAL               196      0      2      1    99%

Notice the new Branch and BrMiss columns and note that we are missing a branch in our file. We’ll take a look at that later.

Coverage Configuration

Coverage allows us to specify a configuration file (.coveragerc files) to specify default coverage attributes. The documentation explains how .coveragerc work.

Let’s add a .coveragerc file to our project that looks like this:

include = ./*
branch = 1

Now we can run coverage without any extra arguments:

$ coverage run test blog

Inspecting Missing Coverage

Now let’s figure out why our branch coverage is not 100%. First we need to regenerate the HTML coverage report and have a look at it:

$ coverage html

Let’s click on manage to see why our file has 88% coverage:


We’re missing the False case for that if statement in our file. We always run from the command line so that code is always executed.

We don’t intend to ever test that missing branch, so let’s ignore the issue and accept our imperfect coverage statistics.


For extra credit, figure out how we can exclude that if __name__ == "__main__": line from our coverage count. Check out the .coveragerc documentation for help.