Wrapper for source based code coverage (-Z instrument-coverage, rust-lang/rust#79121)

cargo-llvm-cov

Wrapper for source based code coverage (-Z instrument-coverage, rust-lang/rust#79121).

  • Installation
  • Usage
    • Continuous Integration
  • Known limitations
  • License

Installation

Prerequisites

cargo-llvm-cov requires nightly toolchain and llvm-tools-preview:

rustup component add llvm-tools-preview --toolchain nightly

From source

cargo install cargo-llvm-cov --version 0.1.0-alpha.5

cargo-llvm-cov relies on unstable compiler flags so it requires a nightly toolchain to be installed, though does not require nightly to be the default toolchain or the one with which cargo-llvm-cov itself is executed. If the default toolchain is one other than nightly, running cargo llvm-cov will find and use nightly anyway.

From prebuilt binaries

You can download prebuilt binaries from the Release page. Prebuilt binaries are available for macOS, Linux (gnu and musl), and Windows.

Via Homebrew

You can install cargo-llvm-cov using Homebrew tap on macOS and Linux:

brew install taiki-e/tap/cargo-llvm-cov

Usage

Click to show a complete list of options
$ cargo llvm-cov --help
cargo-llvm-cov
Wrapper for source based code coverage (-Z instrument-coverage).

Use -h for short descriptions and --help for more details.

USAGE:
    cargo llvm-cov [OPTIONS] [-- <args>...]

ARGS:
    <args>...
            Arguments for the test binary

OPTIONS:
        --json
            Export coverage data in "json" format

            If --output-path is not specified, the report will be printed to
            stdout.

            This internally calls `llvm-cov export -format=text`. See
            <https://llvm.org/docs/CommandGuide/llvm-cov.html#llvm-cov-export>
            for more.

        --lcov
            Export coverage data in "lcov" format.

            If --output-path is not specified, the report will be printed to
            stdout.

            This internally calls `llvm-cov export -format=lcov`. See
            <https://llvm.org/docs/CommandGuide/llvm-cov.html#llvm-cov-export>
            for more.

        --text
            Generate coverage reports in &ldquo;text&rdquo; format.

            If --output-path or --output-dir is not specified, the report will
            be printed to stdout.

            This internally calls `llvm-cov show -format=text`. See
            <https://llvm.org/docs/CommandGuide/llvm-cov.html#llvm-cov-show> for
            more.

        --html
            Generate coverage reports in "html" format. If --output-dir is not
            specified, the report will be generated in `target/llvm-cov`
            directory.

            This internally calls `llvm-cov show -format=html`. See
            <https://llvm.org/docs/CommandGuide/llvm-cov.html#llvm-cov-show> for
            more.

        --open
            Generate coverage reports in "html" format and open them in a
            browser after the operation.

            See --html for more.

        --summary-only
            Export only summary information for each file in the coverage data.

            This flag can only be used together with either --json or --lcov.

        --output-path <PATH>
            Specify a file to write coverage data into.

            This flag can only be used together with --json, --lcov, or --text.
            See --output-dir for --html and --open.

        --output-dir <DIRECTORY>
            Specify a directory to write coverage reports into (default to
            `target/llvm-cov`).

            This flag can only be used together with --text, --html, or --open.
            See also --output-path.

        --ignore-filename-regex <PATTERN>
            Skip source code files with file paths that match the given regular
            expression

        --doctests
            Including doc tests (unstable)

        --no-run
            Compile, but don't run tests (unstable)

        --no-fail-fast
            Run all tests regardless of failure

    -p, --package <SPEC>...
            Package to run tests for

        --workspace
            Test all packages in the workspace [aliases: all]

        --exclude <SPEC>...
            Exclude packages from the test

        --release
            Build artifacts in release mode, with optimizations

        --features <FEATURES>...
            Space or comma separated list of features to activate

        --all-features
            Activate all available features

        --no-default-features
            Do not activate the `default` feature

        --target <TRIPLE>
            Build for the target triple

            When this option is used, coverage for proc-macro and build script
            will not be displayed because cargo does not pass RUSTFLAGS to them.

        --manifest-path <PATH>
            Path to Cargo.toml

    -v, --verbose
            Use verbose output (-vv very verbose/build.rs output)

        --color <WHEN>
            Coloring [possible values: auto, always, never]

        --frozen
            Require Cargo.lock and cache are up to date

        --locked
            Require Cargo.lock is up to date

    -Z <FLAG>...
            Unstable (nightly-only) flags to Cargo

    -h, --help
            Prints help information

    -V, --version
            Prints version information

By default, only the summary is printed to stdout.

cargo llvm-cov

With html report (the report will be generated to target/llvm-cov directory):

cargo llvm-cov --html
open target/llvm-cov/index.html

or

cargo llvm-cov --open

With plain text report (if --output-path is not specified, the report will be printed to stdout):

cargo llvm-cov --text | less -R

With json report (if --output-path is not specified, the report will be printed to stdout):

cargo llvm-cov --json --output-path cov.json

With lcov report (if --output-path is not specified, the report will be printed to stdout):

cargo llvm-cov --lcov --output-path lcov.info

Continuous Integration

Here is an example of GitHub Actions workflow that uploads coverage to Codecov.

name: Coverage

on: [pull_request, push]

jobs:
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/[email protected]
      - name: Install Rust
        run: rustup toolchain install nightly --component llvm-tools-preview
      - name: Install cargo-llvm-cov
        run: curl -LsSf https://github.com/taiki-e/cargo-llvm-cov/releases/download/v0.1.0-alpha.5/cargo-llvm-cov-x86_64-unknown-linux-gnu.tar.gz | tar xzf - -C ~/.cargo/bin
      - name: Generate code coverage
        run: cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info
      - name: Upload coverage to Codecov
        uses: codecov/[email protected]
        with:
          token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos
          files: lcov.info
          fail_ci_if_error: true

NOTE: Currently, only line coverage is available on Codecov. This is because -Z instrument-coverage does not support branch coverage and Codecov does not support region coverage. See also #8, #12, and #20.

Known limitations

  • Due to a bug of -Z instrument-coverage, some files may be ignored. There is a known workaround for this issue, but note that the workaround is likely to cause another problem. See rust-lang/rust#86177 and #26 for more.
  • Branch coverage is not supported yet. See #8 and rust-lang/rust#79649 for more.
  • Support for doc tests is unstable and has known issues. See #2 and rust-lang/rust#79417 for more.

See also the code-coverage-related issues reported in rust-lang/rust.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Issues

Collection of the latest Issues

svenstaro

svenstaro

C-enhancement
0

Hey! This is currently my favorite Rust coverage tool but I'm missing Cobertura output for GitLab. Currently, the only way seems to be to use the unmaintained Python package lcov-to-cobertura-xml but that seems annoying to me. Any chance we can get native GitLab-compatible Cobertura output in this tool? :)

smoelius

smoelius

4

Would you be open to using cargo's --target-dir command line option instead of the CARG_TOARGET_DIR environment variable? https://github.com/taiki-e/cargo-llvm-cov/blob/4ab165ff683eb8a1968919ab645aac7c3558850e/src/main.rs#L184 The problem with using the environment variable is that it "infects" subordinate cargo invocations. I would expect that in most situations, one would only want to set the target directory for the initial invocation.

I could submit a PR if you like.

YJDoc2

YJDoc2

C-enhancement
3

I do not know much about how the code coverage data generation works, so sorry if this is a stupid question, but can we use external programs that run our binary to generate coverage data?

To elaborate, I am contributing to https://github.com/containers/youki/ , where we use cargo-llvm-cov in CI to generate coverage report. As of now, we generate coverage using unit tests, which are Rust tests (marked with #[tests] attribute).

Now to validate that the runtime itself is consistent with OCI spec, OCI has validation tests which run youki binary and test it on various conditions : https://github.com/opencontainers/runtime-tools

My question is : is it possible to generate coverage data from those tests, and use that as well to find total coverage?

Thanks for you time.

dbr

dbr

1

This tool has been extremely useful, thanks! By far the simplest to get working tool for the instrument-coverage (I had some trouble getting grcov to work)

One thing I think could be improved is the command line args - mainly about generating reports (where quite a lot of the arguments are mutually exclusive, e.g --no-run and any of the arguments like --lib, --html and --output-file`)

I think to steal some ideas from coverage.py could work, specifically to have a report subcommand.

Main downside of this is obviously a common "run tests and open report" usage becomes two commands, so instead of just cargo llvm-cov --html --open it would become cargo llvm-cov && cargo llvm-cov html --open.

However the upsides are:

  1. the available options become smaller and more organised
  2. maybe be a tiny bit more intuitive to use (e.g first few times I accidentally reran my testsuite because I assumed cargo llvm-cov --open would only open the report)
  3. If gathering and reporting are two separate actions, it would be (if it's not already?) possible to gather coverage from multiple runs and report on them in one - e.g cargo llvm-cov test && cargo llvm-cov run --append-cov && cargo llvm-cov run --example blah --append-cov && cargo llvm-cov html which would run the tests and two executable, and generate a report of the combined coverage

I imagine the help output could look something like:

Example help output

Warwolt

Warwolt

0

First of all, really nice project! I got it installed and running in my Windows environment with zero hassle. 👏

I've only used gcov in C++ based projects before, and so I've only seen that classical blocky html report. Gcov seems to render covered lines in green, and non-covered lines in red, in their html reports. As far as I can tell for the rust code that I've tested cargo-llvm-cov with, I'm only getting red lines.

image image

Is it possible for cargo-llvm-cov to generate html reports that contain these green lines? Or does it already and I'm just perhaps having a config issue?

larsluthman

larsluthman

C-enhancement
2

First, thanks for a very useful tool!

It would be nice if there was an command line option to merge the coverage of different instantiations of generic functions in the summary of the coverge of a file, so that it would show 100% coverage if the tests covered all of a generic function, even if different regions of the function were covered by different instantiations. Consider the following source file:

With cargo-llvm-cov 0.1.0-alpha.4, the report will show a region coverage of 88.89% (8/9), even though all regions of the generic function are marked as covered. The current rule seems to be that there needs to be at least one instantiation with 100% coverage for the generic function to be counted as 100% covered. This is probably a good choice for some situations, but sometimes you have a generic function that is difficult to cover completely with a single instantiation and for those cases it would be nice to have an option to merge the coverage of all instantiations when computing the summary.

taiki-e

taiki-e

C-enhancement
7

STATUS:

#40 fixed all of the errors and many of the warnings

The remaining warnings are probably the column offset errors mentioned in rust-lang/rust#79417 (comment), and it's not clear whether we can fix them on our side at this time.


This is currently available via --doctests flag as an unstable feature.

Refs:

Versions

Find the latest versions by id

v0.1.13 - Dec 14, 2021

  • Support custom-built rust toolchain. (#111, thanks @tofay)

v0.1.12 - Nov 15, 2021

  • Exclude CARGO_HOME and RUSTUP_HOME used in the official rust docker image from reports. (#105)

v0.1.10 - Oct 23, 2021

  • Fix a compatibility issue with cc. (#98)

v0.1.9 - Oct 13, 2021

  • Distribute statically linked binary on Windows MSVC. (#95)

v0.1.8 - Oct 04, 2021

  • Fix an issue where some files were incorrectly ignored in reports. (#94, thanks @larsluthman)

v0.1.7 - Sep 19, 2021

  • Add --failure-mode option. (#91, thanks @smoelius)

v0.1.3 - Aug 26, 2021

v0.1.0-alpha.2 - Feb 12, 2021

  • Add --text option to output full report in plain text. (#3, thanks @romac)

v0.1.0-alpha.1 - Jan 23, 2021

Initial release

Information - Updated Jan 07, 2022

Stars: 34
Forks: 2
Issues: 9

A fantasy deathcrawl in Rust

To run, with Rust compiler and Cargo package manager installed:

A fantasy deathcrawl in Rust

cargo-ndk - Build Rust code for Android

This cargo extension handles all the environment configuration needed for successfully building libraries

cargo-ndk - Build Rust code for Android

quest-hook-template

A template for writing mods for Quest il2cpp games in Rust using cargo generate to clone the template:

quest-hook-template

Command line json text parsing and processing utility

parsing json compliant with rust and cargo

Command line json text parsing and processing utility

Clone this repo: git clone

If you don't have Rust and cargo-make installed,

Clone this repo: git clone

Rustup: the Rust installer and version management tool

To test that you have Rust and Cargo installed, you can run this in your terminal of choice: cargo --version

Rustup: the Rust installer and version management tool

Address generator in Rust

If you have Rust: cargo install gemgen

Address generator in Rust

First, complete the basic Rust setup instructions

Use Rust's native cargo command to build and launch the template node:

First, complete the basic Rust setup instructions

Use one mouse on multiple computers

In France, we call that &quot;flemme&quot; (install cargo for rust)

Use one mouse on multiple computers

cargo rssc - Rust scripts for crates building

will copy the template_basic into scripts_rssc folder

cargo rssc - Rust scripts for crates building

NES Emulator in Rust-WASM

Requires Rust with cargo, nodejs, and wasm-pack

NES Emulator in Rust-WASM
Facebook Instagram Twitter GitHub Dribbble
Privacy