nextest-rs/nextest

Nextest is a next-generation test runner for Rust

For more information, check out filter expressions

Nextest

. For more information, check out the website.

This repository contains the source code for:

Minimum supported Rust version

The minimum supported Rust version is Rust 1.59. At least the last 3 versions of stable Rust are supported at any given time; see the stability policy for more details.

While a crate is pre-release status (0.x.x) it may have its MSRV bumped in a patch release. Once a crate has reached 1.x, any MSRV bump will be accompanied with a new minor version.

Contributing

See the CONTRIBUTING file for how to help out.

License

This project is available under the terms of either the Apache 2.0 license or the MIT license.

This project is derived from diem-devtools. Upstream source code is used under the terms of the Apache 2.0 license and the MIT license.

Issues

Collection of the latest Issues

sunshowers

sunshowers

Comment Icon0

The status reporter (nextest-runner/src/reporter.rs) is the one major component of nextest for which there isn't as much coverage as one would ideally want to have. Ensure it's tested so that issues like #430 don't arise again.

Guiguiprim

Guiguiprim

Comment Icon3

I have a use case that is maybe half a hack.

In more details:

  • in my workspace I have some small crates that only contains some small helpers binaries used by my project tests (or when doing some manual testing).
  • cargo build at the workspace level builds all those binaries
  • cargo test works, as the binaries have already being built and can be found
  • cargo test without cargo build before fails, as the binaries are missing

Currently cargo nextest archive does not include those binaries even if cargo build was called before. Is it possible to include in the archive all the binaries in the target folder ? I guess the exact list would be hard to get other than just listing the folder content...

As a workaround, I can:

  • Not use the archive feature (which is a be sad)
  • Add empty integration tests in the crates with the helpers binaries (at the cost of more build and link time) to force cargo test to build them

So the question is: Is it reasonable to try to add the binaries inside the archive, or is my use-case too sketchy (which I would understand) ?

sunshowers

sunshowers

enhancement
Comment Icon0

Nextest reserves the "default-" namespace for profiles. Produce a warning if an unknown profile name starting with "default-" is encountered.

This will involve:

  1. looking through default-config.toml to gather the set of "default-" profiles known to nextest.
  2. adding a method on NextestConfig which uses the set from step 1 to return "default-" profiles that aren't known to nextest
  3. for unknown "default-" profiles, producing a log::warn! explaining this (do this in cargo-nextest)

This is a great first issue, roughly 3-4 hours of work. Please comment on this issue if you'd like to work on this!

sunshowers

sunshowers

enhancement
Comment Icon0

Cargo profiles have the inherits setting which lets some profiles inherit from others.

Nextest profiles should have the inherits setting as well. Steps to do:

  • Add "inherits" to the [profile] data model. Treat an unspecified "inherits" as inheriting from the default profile.
  • Check for cycles in the inherits graph (can use petgraph toposort for this, storing profile names in a GraphMap)
  • Implement support for inherits -- in NextestProfile, store a chain of profiles rather than just one profile + default.

This is a really good first issue and will probably involve 1-2 days of work including tests -- please comment here if you'd like to work on it!

sunshowers

sunshowers

enhancement
Comment Icon0

This might be useful for larger deployments.

Need to see whether it makes sense to automatically update to the specific version -- that might be "too much" for us to do.

lcmgh

lcmgh

enhancement
Comment Icon1

Hi!

I'm using the great archived test features in order to exec. my tests on a temporary CICD spawned pod in Kubernetes to test against other resources in there.

Two questions:

  1. Is there a way to run those archived tests with tarpaulin to gather code coverage?
  2. I run my remote tests with ./cargo-nextest nextest run --archive-file /app/test-archive.tar.zst (Cargo not installed). How would I run these tests and gather code coverage with llvm-cov?

Thanks

steveeJ

steveeJ

enhancement
Comment Icon12

Note from maintainer: Here's an outline of the work required for this. Please dive in if you're interested!


we're having issues running cargo nextest in a nix-shell on a macos github actions runner. we run the same steps on an ubuntu runner in parallel which succeeds, so this seems to be macos specific.

it may or may not have something to do that the tested source code is under /tmp, which on this macos runner has an implicit /private/ prefix.

some references for context:

please let me know if i can provide further debug output to help us understand what's happening.

johamster

johamster

question
Comment Icon10

Hi,

first of all thanks for building nextest it is great (except for what follows here).

We observed a performance regression in our test execution times from approx. 20-30s to >200s. So far, we don't have an explanation. We observed it yesterday but it could have gone unnoticed for a few days.

It only happens on Mac with M1 (pro) chips. Linux on x86 machines (e.g., our CI) is not affected.

So far we tried the following without success:

  • revert to an older revision of the code to exclude recent changes in one of our dependency crates.
  • tested cargo-nextest in different revisions (from 0.9.0 - 0.9.16).
  • cargo clear.
  • Update to MacOS 12.4 (we first noticed on 12.3.1).

Further, we observed that nextest processes are spawned as expected. However we observed the following:

  • Test execution is really slow. Tests which should run sub-second take significantly longer.
  • The related processes show up in the activity monitor with almost no CPU utilization. We don't know if they are waiting for something or if the activity monitor is just too slow and they die before proper measurements could be done.
  • We observed that once we start nextest the following service shows up in activity monitor: parentalcontrolsd with significant CPU utilization. Not sure if that is expected.
  • It does not seem to be sporadic since it occurred on 4/4 tested Mac M1 devices. We don't know if Mac on x86 is affected.

Any ideas? We are happy to help.

Kind regards, Johannes

davemilter

davemilter

Comment Icon3

May be I missed something, but I can non find any mention for this. It would be nice to get from cargo list of crates that would be rebuilt by cargo build, and then run tests only for affected crates in workspace.

sunshowers

sunshowers

A-filter-expressions
Comment Icon0

Add filtering for packages (-p can be overloaded to mean this).

This would accept the package, deps and rdeps functions, but not test. The point of this would be to possibly build a smaller set of tests (at the cost of a different feature unification result, plus the inability to reuse builds).

This doesn't block the full release of filter expressions.

Guiguiprim

Guiguiprim

Comment Icon7

My next maybe weird feature request.

This is somehow related to https://github.com/nextest-rs/nextest/issues/27 or at least could be a first step for it.

My initial use case

I have a bunch of tests (across multiple binaries) who need to uniquely acquire a system resource. Currently I run the tests with -j1 which is sub-optimal. I could run my tests in two passes: cargo nextest -j1 <some filtering> and cargo nextest <opposite filtering>, but it's not ideal too and force to maintain two filtering (hopefully without missing some tests).

Having a way of defining test groups with custom execution configuration would be great.

Proposed feature

Adding groups support !

It could allow:

  • only running predefined groups of tests
  • running differently different groups of tests
  • more ?

Defining groups

I think it could go inside ./config/nextest.conf. Maybe like:

Constructing groups

Constructing groups could be as easy as:

  • Applying TestBuildFilter to get the tests pool
  • For every TestGroupConfig construct a group by taking out of the pool the matching tests
  • All the remaining tests are put in a unnamed group with the config give to the CLI

This would give an importance to the group definitions order while also avoiding any duplication.

Running tests

Here things can be as complex as we want.

  • In the first version I would go with running groups one after the other.
  • In following version we could add more complex scheduling scheme
    • If we have 3 groups with -j1 maybe they are allowed to run concurrently
    • ...

New CLI options

With all of this, we could probably add some options (for both list and run): --group <NAME> and --unamed-group/--not-in-group (or any better name) to only run tests within some groups.

jszwedko

jszwedko

help wanted
Comment Icon7

Hi all!

I'm a big fan of what this project is doing.

I noticed when trying to integrate this into https://github.com/vectordotdev/vector that it fails to run test binaries built from criterion benchmarks which don't support the same --format flag that normal test binaries support:

Running --help on the test binary:

I was just curious to get thoughts on handling this. Should I stick with normal cargo test --benches for that target for now and use nextest for the other targets?

sunshowers

sunshowers

help wanted
Comment Icon1

Antivirus and Gatekeeper checks can cause performance issues with nextest on Windows and macOS, respectively.

I've added a note about them to the nextest site, but I'm going to leave this issue open as a catch-all in case people still have performance issues afterwards (I've definitely seen some reports on macOS even after the terminal was added to Developer Tools).

tailhook

tailhook

Comment Icon4

Hi! Nextest looks pretty great!

One common thing I have in many crates is running tests with different feature sets. Would be nice to be able to merge those tests into the single set of tests instead of running sequentially via separate commands, if it's at all possible.

TomPridham

TomPridham

enhancement
Comment Icon8

first off, really like this so far. would it be possible to group related tests? right now tests are output in the order the run(i assume), which means that they can get printed out of order in relation to their mod. it would be helpful to me to have all tests in a mod grouped together after they have finished running to make comparing times and looking at failures easier. current behavior

desired behavior

sunshowers

sunshowers

enhancement
Comment Icon5

Currently, nextest runs every test in its own process. Should running tests within the same process be supported?

  • How should test binaries be marked this way? package.metadata is one solution.
  • How does concurrency control work? Rust's libtest doesn't support the jobserver protocol, so it's hard to communicate with it. One option is to run them serially at the end, once all the regular tests are run.
  • How can test timing, stdout/stderr for individual tests, and other results be obtained? libtest's --format json is nightly-only. Maybe nextest can just give up on reporting individual statuses for these tests.
sunshowers

sunshowers

enhancement
Comment Icon4

Currently, nextest doesn't support Rust doctests. This is because doctests are not exposed in stable Rust the way regular test binaries are, and are instead treated as special by cargo test.

One nightly-only fix that might work is to collect doctest executables through -Z unstable-options --persist-doctests. However, this isn't a stable approach so it must be used with care and must be exposed

Note: You can run cargo test --doc as a separate step from cargo nextest run. This will not incur a performance penalty: cargo nextest run && cargo test --doc will not cause any more builds than a plain cargo test that runs doctests.

sunshowers

sunshowers

enhancement
Comment Icon0

This needs to be done with a little care:

  • The initial, simpler use requires keeping track of which Cargo arguments the build was run with -- this is relatively simple.
  • An extended use case to solve is "what if the user wants to grow the set of tests or binaries that are run?" This is going to require us keeping track of the exact set of binaries run and Cargo arguments passed. Probably worth discussing with some folks before doing so.

Overall this is a stateful operation, kind of like a source control bisect.

sunshowers

sunshowers

enhancement
Comment Icon0

Add support to run a single test multiple times in parallel. This is a somewhat different mode of operation from nextest as usual (the runner should be fed the same test multiple times, and should maybe not run other tests at the same time), but is worth doing as a future improvement.

Versions

Find the latest versions by id

nextest-runner-0.22.1 - Jul 31, 2022

Fixed

  • Reverted indicatif to 0.16.2 to fix regression where nextest no longer produced any output if stderr wasn't a terminal.

cargo-nextest-0.9.33 - Jul 31, 2022

Fixed

  • Fixed regression in cargo-nextest 0.9.32 where it no longer produced any output if stderr wasn't a terminal.

nextest-runner-0.22.0 - Jul 30, 2022

Changed

  • Progress bar library indicatif updated to 0.17.0.

nextest-filtering-0.2.1 - Jul 30, 2022

Internal

  • Evaluation now uses a stack machine via the recursion crate. Thanks Inanna for your first contribution!

cargo-nextest-0.9.32 - Jul 30, 2022

Added

  • cargo nextest run now has a new --no-run feature to build but not run tests. (This was previously achievable with cargo nextest list -E 'none()', but is more intuitive this way.)
  • Pre-built binaries are now available for i686 Windows. Thanks Guiguiprim!

Internal improvements

  • Filter expression evaluation now uses a stack machine via the recursion crate. Thanks Inanna for your first contribution!

quick-junit-0.3.0 - Jul 28, 2022

Added

  • Report contains a new uuid field with a unique identifier for a particular run. This is an extension to the JUnit spec.

nextest-runner-0.21.0 - Jul 28, 2022

Changed

See the changelog for cargo-nextest 0.9.31.

cargo-nextest-0.9.31 - Jul 28, 2022

Added

  • Nextest sets a new NEXTEST_RUN_ID environment variable with a UUID for a test run. All tests run within a single invocation of cargo nextest run will set the same run ID. Thanks mitsuhiko for your first contribution!

nextest-runner-0.20.0 - Jul 25, 2022

Changed

See the changelog for cargo-nextest 0.9.30.

cargo-nextest-0.9.30 - Jul 25, 2022

Fixed

  • Fixed target runners specified as relative paths.

  • On Unix, cargo-nextest's performance had regressed (by 3x on clap) due to the change introduced in version 0.9.29 to put each test process into its own process group. In this version, this regression has been fixed, but only if you're using the pre-built binaries or building on Rust 1.64+ (currently in nightly).

    Note to distributors: to fix this regression while building with stable Rust 1.62, set the following environment variables:

    • RUSTC_BOOTSTRAP=1
    • RUSTFLAGS='--cfg process_group --cfg process_group_bootstrap_hack'

    This is temporary until the process_set_process_group feature is stabilized in Rust 1.64.

cargo-nextest-0.9.30-rc.3 - Jul 25, 2022

This is a test release.

cargo-nextest-0.9.30-rc.2 - Jul 25, 2022

This is a test release.

cargo-nextest-0.9.30-rc.1 - Jul 25, 2022

This is a test release.

nextest-runner-0.19.0 - Jul 24, 2022

Changed

See the changelog for cargo-nextest 0.9.29.

cargo-nextest-0.9.29 - Jul 24, 2022

Added

  • On Unix, each test process is now put into its own process group. If a test times out or Ctrl-C is pressed, the entire process group is signaled. This means that most subprocesses spawned by tests are also killed.

    However, because process groups aren't nested, if a test creates a process group itself, those groups won't be signaled. This is a relatively uncommon situation.

  • On Windows, each test process is now associated with a job object. On timeouts, the entire job object is terminated. Since job objects are nested in recent versions of Windows, this should result in all subprocesses spawned by tests being killed.

    (On Windows, the Ctrl-C behavior hasn't changed. Nextest also doesn't do graceful shutdowns on Windows yet, though this may change in the future.)

  • Nextest can now parse Cargo configs specified via the unstable --config option.

  • Nextest now publishes binaries for aarch64-unknown-linux-gnu (#398) and x86_64-unknown-linux-musl (#399). Thanks messense and Teymour for your first contributions!

Fixed

  • Per-test overrides are now additive across configuration files (including tool-specific configuration files).

cargo-nextest-0.9.29-rc.1 - Jul 24, 2022

This is a test release to ensure that releasing Linux aarch64 and musl binaries works well.

nextest-runner-0.18.0 - Jul 22, 2022

Changed

See the changelog for cargo-nextest 0.9.27.

cargo-nextest-0.9.28 - Jul 22, 2022

This is a quick hotfix release to ensure that the right tokio features are enabled under default-no-update.

cargo-nextest-0.9.27 - Jul 22, 2022

This is a major architectural rework of nextest. We've tested it thoroughly to the best of our ability, but if you see regressions please report them!

If you encounter a regression, you can temporarily pin nextest to the previous version in CI. If you're on GitHub Actions and are using taiki-e/install-action, use this instead:

Added

  • Nextest now works with the Miri interpreter. Use cargo miri nextest run to run your tests with Miri.
  • Nextest now detects some situations where tests leak subprocesses. Previously, these situations would cause nextest to hang.
  • Per-test overrides now support slow-timeout and the new leak-timeout config parameter.
  • A new option --tool-config-file allows tools that wrap nextest to specify custom config settings, while still prioritizing repository-specific configuration.

Changed

  • Major internal change: The nextest list and run steps now use Tokio. This change enables the leak detection described above.
  • The list step now runs list commands in parallel. This should result in speedups in most cases.

Fixed

  • Nextest now redirects standard input during test runs to /dev/null (or NUL on Windows). Most tests do not read from standard input, but if a test does, it will no longer cause nextest to hang.
  • On Windows, nextest configures standard input, standard output and standard error to not be inherited. This prevents some kinds of test hangs on Windows.
  • If a dynamic library link path doesn't exist, nextest no longer adds it to LD_LIBRARY_PATH or equivalent. This should have no practical effect.
  • Archiving tests now works even if the target directory is not called "target".

nextest-runner-0.17.0 - Jul 14, 2022

Changed

  • nextest-metadata updated to 0.5.0.

nextest-metadata-0.5.0 - Jul 14, 2022

(This change was included in 0.4.4, which should have been a breaking change.)

Changed

  • RustTestSuiteSummary::testcases renamed to test_cases.

cargo-nextest-0.9.26 - Jul 14, 2022

This is a quick hotfix release to update the version of nextest-metadata, to which a breaking change was accidentally committed.

nextest-runner-0.16.0 - Jul 13, 2022

See the changelog for cargo-nextest 0.9.25.

nextest-metadata-0.4.4 - Jul 13, 2022

Added

  • RustTestSuiteSummary has a new field status, which is a newtype over strings:
    • "listed": the test binary was executed with --list to gather the list of tests in it.
    • "skipped": the test binary was not executed because it didn't match any expression filters.

nextest-filtering-0.2.0 - Jul 13, 2022

Added

  • The expression language supports several new predicates:
    • kind(name-matcher): include all tests in binary kinds (e.g. lib, test, bench) matching name-matcher.
    • binary(name-matcher): include all tests in binary names matching name-matcher.
    • platform(host) or platform(target): include all tests that are built for the host or target platform, respectively.
  • It is now possible to evaluate a query without knowing the name of the test. The result is evaluated as a three-valued logic (Kleene K3), and is returned as an Option<bool> where None indicates the unknown state.

Changed

  • The evaluator now takes a TestQuery struct, making it easier to add more parameters in the future.
  • MSRV updated to Rust 1.59.

cargo-nextest-0.9.25 - Jul 13, 2022

This is a major release with several new features.

Filter expressions

Filter expressions are now ready for production. For example, to run all tests in nextest-runner and all its transitive dependencies within the workspace:

This release includes a number of additions and changes to filter expressions.

Added

  • The expression language supports several new predicates:
    • kind(name-matcher): include all tests in binary kinds (e.g. lib, test, bench) matching name-matcher.
    • binary(name-matcher): include all tests in binary names matching name-matcher.
    • platform(host) or platform(target): include all tests that are built for the host or target platform, respectively.

Changed

  • If a filter expression is guaranteed not to match a particular binary, it will not be listed by nextest. (This allows platform(host) and platform(target) to work correctly.)

  • If both filter expressions and standard substring filters are passed in, a test must match filter expressions AND substring filters to be executed. For example:

This will execute only the tests in nextest-runner that match test_foo or test_bar.

Per-test overrides

Nextest now supports per-test overrides. These overrides let you customize settings for subsets of tests. For example, to retry tests that contain the substring test_e2e 3 times:

Currently, only retries are supported. In the future, more kinds of customization will be added.

Other changes

  • A new environment variable NEXTEST_RETRIES controls the number of retries tests are run with. In terms of precedence, this slots in between the command-line --retries option and per-test overrides for retries.
  • cargo nextest list now hides skipped tests and binaries by default. To print out skipped tests and binaries, use cargo nextest list --verbose.
  • The Machine-readable output for cargo nextest list now contains a new "status" key. By default, this is set to "listed", and for binaries that aren't run because they don't match expression filters this is set to "skipped".
  • The --platform-filter option is deprecated, though it will keep working for all versions within the nextest 0.9 series. Use -E 'platform(host)' or -E 'platform(target)' instead.
  • cargo nextest run -- --skip and --exact now suggest using a filter expression instead.

nextest-runner-0.15.0 - Jul 02, 2022

Added

  • New config option profile.<profile-name>.test-threads controls the number of tests run simultaneously. This option accepts either an integer with the number of threads, or the string "num-cpus" (default) for the number of logical CPUs.

Fixed

  • Within JUnit XML, test failure descriptions (text nodes for <failure> and <error> tags) now have invalid ANSI escape codes stripped from their output.

cargo-nextest-0.9.24 - Jul 02, 2022

Added

  • New config option profile.<profile-name>.test-threads controls the number of tests run simultaneously. This option accepts either an integer with the number of threads, or the string "num-cpus" (default) for the number of logical CPUs. As usual, this option is overridden by --test-threads and NEXTEST_TEST_THREADS, in that order.
  • The command-line --test-threads option and the NEXTEST_TEST_THREADS environment variable now accept num-cpus as their argument.
  • nextest now works with cargo binstall (#332). Thanks [Remoun] for your first contribution!

Fixed

  • Within JUnit XML, test failure descriptions (text nodes for <failure> and <error> tags) now have invalid ANSI escape codes stripped from their output.

nextest-runner-0.14.0 - Jun 26, 2022

Added

  • On Windows, nextest now detects tests that abort due to e.g. an access violation (segfault) and prints their status as "ABORT" rather than "FAIL", along with an explanatory message on the next line.
  • Improved JUnit support: nextest now heuristically detects stack traces and adds them to the text node of the <failure> element (#311).

nextest-metadata-0.4.3 - Jun 26, 2022

Added

Information - Updated Aug 09, 2022

Stars: 1.0K
Forks: 36
Issues: 26

utest standard testing for Rust

Make sure your not using this for testing no_std code as it relies on the unstable branch

utest standard testing for Rust

K9 - Rust Testing Library

Snapshot testing + better assertions

K9 - Rust Testing Library

Spec &quot;it&quot; for Rust testing

The test output is like the following

Spec &quot;it&quot; for Rust testing

Shuttle is a library for testing concurrent Rust code

It is an implementation of a number of

Shuttle is a library for testing concurrent Rust code

Let's make a web service and client in Rust

So I'm working on this project Rust regression testing

Let's make a web service and client in Rust

Simple golden file testing for Rust

Add the following to your Cargo manifest

Simple golden file testing for Rust
Http

329

HTTP mocking to test Rust applications

wiremock provides HTTP mocking to perform black-box testing of Rust applications that

HTTP mocking to test Rust applications

just testing rust

cli + advanced cli features

just testing rust
Facebook Instagram Twitter GitHub Dribbble
Privacy