Tarpaulin is a code coverage reporting tool for the Cargo build system, named for a waterproof cloth used to cover cargo on a ship. Currently, tarpaulin provides working line coverage and while fairly reliable may still contain minor inaccuracies in the results. A lot of work has been done to get it working on a wide range of projects, but often unique combinations of packages and build features can cause issues so please report anything you find that's wrong. Also, check out our roadmap for planned features.
Tarpaulin only supports x86_64 processors running Linux. This is because instrumenting breakpoints into executables and tracing their execution requires processor and OS specific code. It is a goal when greater stability is reached to add wider system support, however this is sufficient to run Tarpaulin on popular CI tools like Travis.
It can also be run in Docker, which is useful for when you don't use Linux but want to run it locally, e.g. during development. See below for how to do that.
Below is the help-text for a thorough explanation of the flags and features available:
Note on tests using signals
If your tests or application make use of unix signals they may not work with
tarpaulin. This is because tarpaulin relies on the sigtrap signal to catch when
the instrumentation points are hit. The
--forward option results in
forwarding the signals from process stops not caused by SIGSTOP, SIGSEGV or
SIGILL to the test binary.
Below is a list of features currently implemented. As Tarpaulin loads binary files into memory and parses the debugging information, different setups could lead to coverage not working. In this instance, please raise an issue detailing your setup and an example project and I'll attempt to fix it (please link us to a repo and the commit containing your project and paste the verbose output).
- Line coverage
- Full compatibility with cargo test CLI arguments
- Uploading coverage to https://coveralls.io or https://codecov.io
- HTML report generation and other coverage report types
- Coverage of tests, doctests, benchmarks and examples possible
- Excluding irrelevant files from coverage
- Config file for mutually exclusive coverage settings (see
Config filesection for details)
Tarpaulin is a command-line program, you install it into your linux development environment with cargo install:
Tarpaulin used to rely on Cargo as a dependency and then require an ssl install as well as other libraries but now it uses your system cargo simplifying the installation and massively reducing the install time on CI.
When using the Nix package manager, the
nixpkgs.cargo-tarpaulin package can be used.
This ensures that tarpaulin will be built with the same rust version as the rest of your packages.
You can also use cargo-binstall:
When tarpaulin runs your tests it strives to run them in the same environment as if they were ran via cargo test. In order to achieve this it sets the following environment variables when executing the test binaries:
- RUST_BACKTRACE - When --verbose flag is used
- CARGO_MANIFEST_DIR - Path to Cargo.toml From --root | --manifest-path or guessed from the current or parent directory
- CARGO_PKG_NAME - From Cargo.toml
- CARGO_PKG_AUTHORS - From Cargo.toml
- CARGO_PKG_VERSION - From Cargo.toml
- LLVM_PROFILE_FILE - Used for LLVM coverage
In order for tarpaulin to construct the Cargo environment correctly, tarpaulin needs to find Cargo.toml by either:
- Using --root or --manifest-path or
- By invoking Cargo from the current working directory within the project holding Cargo.toml manifest or
- By invoking Cargo from a sub-directory within the project
If Cargo does not find any Cargo.toml from using either of above methods the run will error "cargo metadata" and exit.
Several RFCs are open in rust-lang to expose more of these directly in order to avoid the issues arising out of this.
To get detailed help on available arguments when running tarpaulin call:
Currently no options are required, if no root directory is defined Tarpaulin will run in the current working directory.
Below is a Tarpaulin run utilising one of our example projects. This is a relatively simple project to test and if you check the test, you can see the output correctly reports the lines the test hits.
Tarpaulin can also report the change in coverage for each file between runs. If the tests were updated in the previous example to cover all the lines we would expect the following output.
Hint: if using coveralls.io with travis-ci run with the options
--ciserver travis-ci --coveralls $TRAVIS_JOB_ID. The coveralls.io repo-token
is mainly designed for private repos and it won't generate a badge for the
coverage results submitted (although you can still see them on the coveralls
web interface). For an example of a project using Tarpaulin, you can check out
my crate keygraph-rs.
Ignoring code in files
Before tarpaulin 0.13.4 you could ignore code in blocks with
#[cfg_attr(tarpaulin, skip)] this has changed with 0.13.4 and onwards
and the new instructions are described below. If you get compiler errors
mentioning unknown attribute skip use the
--avoid-cfg-tarpaulin flag, this
affects a small number of users as it wasn't a largely adopted feature so also
look to updating your code or seeing if any of your dependencies are out of
Tarpaulin allows you to ignore modules or functions using attributes. Below is an example of ignoring the main function in a project:
However, the skip attribute only allows you to exclude code from coverage
it doesn't change the code present in the binaries or what tests are ran.
Because of this,
--cfg=tarpaulin is used when building your project for
Tarpaulin allowing you to also conditionally include/exclude code from
compilation entirely. For example to have a test that isn't included in
the test binaries when built with tarpaulin and cannot be ran just do:
If you still want the test included in the binary just ignored by default you can use:
There is also nightly support for using tool attributes with tarpaulin for skip. For example:
Continuous Integration Services
Tarpaulin aims to be easy to add to your CI workflow. With well tested support for Travis-CI it also supports sending CI specific meta-data to coveralls.io for Circle, Semaphore, Jenkins and Codeship (though only Jenkins has been tested).
You can also use Tarpaulin on Azure, check out crate-ci/azure-pipelines for an example config.
Travis-ci and Coverage Sites
The expected most common usecase is launching coverage via a CI service to upload to a site like codecov or coveralls. Given the built in support and ubiquity of travis-ci it seems prudent to document the required steps here for new users. To follow these steps you'll first need a travis-ci and a project setup for your coverage reporting site of choice.
We recommend taking the minimal rust .travis.yml, installing the libssl-dev
dependency tarpaulin has and then running Tarpaulin with the version of
rustc you require. Tarpaulin is installed in
before_cache to allow it to be cached
and prevent having to reinstall every Travis run. You can also replace
with a verbose run of tarpaulin to see the test results as well as coverage output.
Tarpaulin is ran after success as there are still some unstable features which could
cause coverage runs to fail. If you don't rely on any of these features you can
cargo test with a call to
For codecov.io you'll need to export CODECOV_TOKEN are instructions on this in the settings of your codecov project.
If you rely on certain nightly features you may need to change the
before_cache to force tarpaulin to reinstall each time. However, if it can be avoided it
will speed up your CI runs.
Alternatively, there are the prebuilt docker images or the travis-install shell script.
The travis-install script will install the latest tagged release built on travis to your
travis instance and significantly speeds up the travis builds. You can install via that script
bash <(curl https://raw.githubusercontent.com/xd009642/tarpaulin/master/travis-install.sh).
The prebuilt binary is built using github actions ubuntu:latest image, because of this it doesn't work on xenial or trusty, but it works on bionic. You should still keep the rest of the recommended travis settings.
Example how to run coverage within
seccomp in GitHub Actions and push the result
To run tarpaulin on CircleCI you need to run tarpaulin in docker and set the machine flag to true as shown below:
To get the coverage results showing up in your Gitlab pipelines add the following regex to the
Test coverage parsing section in the CI/CD settings.
Or add the regex to the job definition in
Gitlab can show coverage information in the diff of a merge request. For that, use
and generate a
cobertura.xml as described under Pycobertura.
For installation add
cargo install cargo-tarpaulin -f to the script section.
Tarpaulin has builds deployed to docker-hub, to run Tarpaulin on any system that has Docker, run this in your project directory:
This builds your project inside Docker and runs Tarpaulin without any arguments. There are also tags available for the latest version on the develop branch in stable or nightly. And versions after 0.5.6 will have the latest release built with the rust stable and nightly compilers. To get the latest development version built with rustc-nightly run the following:
Note that the build might fail if the Docker image doesn't contain any necessary dependencies. In that case, you can install dependencies before, like this:
Tarpaulin has a config file setting where multiple coverage setups can be
encoded in a toml file. This can be provided by an argument or if a
tarpaulin.toml is present in the same directory as
the projects manifest or in the root directory that will be used unless
--ignore-config is passed. Below is an example file:
Here we'd create three configurations, one would run your tests with
feature_a enabled, and the other with the tests built in release and
feature_b enabled. The last configuration uses a reserved
report and this doesn't result in a coverage run but
affects the report output. This is a reserved feature name and any non-reporting
based options chosen will have no effect on the output of tarpaulin.
For reference on available keys and their types refer to the CLI help text
at the start of the readme or
src/config/mod.rs for the concrete types
if anything is unclear. For arguments to be passed into the test binary that
-- in tarpaulin use
args in the toml file.
Setting the field
config will have no effect on the run as it won't be parsed
for additional configuration.
For the flags
--bins use the
run-types entry in the config file.
There are some tools available which can extend tarpaulin functionality for other potential user needs.
Normally, Tarpaulin can't report on code coverage within the code for a
procedural macro. You'll need to add a test that expands the macro at run-time
in order to get those stats. The
runtime-macros crate was made for
this purpose, and its documentation describes how to use it with Tarpaulin.
pycobertura is a python library
for working with cobertura reports. It offers a report diffing tool as well as
its own report implementations.
To generate a
cobertura.xml simply run the following tarpaulin command:
pycobertura with pip and execute the desired command.
As tarpaulin doesn't allow you to change the name of the generated cobertura report be mindful of this if diffing reports between multiple commits.
Issues and Contributing
Issues, feature requests and pull requests are always welcome! For a guide on how to approach bugs found in Tarpaulin and adding features please check CONTRIBUTING.
Rust 1.23 introduced a regression in the compiler affecting tarpaulin's accuracy. If you see missing lines or files, check your compiler version.
- Branch coverage for tests
- Condition coverage for tests
- MCDC coverage reports
- LLVM coverage support
- Support for embedded targets
- OSX support
- Windows support
Tarpaulin is currently licensed under the terms of both the MIT license and the Apache License (Version 2.0). See LICENSE-MIT and LICENSE-APACHE for more details.