awslabs/shuttle

Shuttle is a library for testing concurrent Rust code

It is an implementation of a number of

Shuttle

. randomized concurrency testing techniques, including A Randomized Scheduler with Probabilistic Guarantees of Finding Bugs.

Getting started

Consider this simple piece of concurrent code:

use std::sync::{Arc, Mutex};
use std::thread;

let lock = Arc::new(Mutex::new(0u64));
let lock2 = lock.clone();

thread::spawn(move || {
    *lock.lock().unwrap() = 1;
});

assert_eq!(0, *lock2.lock().unwrap());

There is an obvious race condition here: if the spawned thread runs before the assertion, the assertion will fail. But writing a unit test that finds this execution is tricky. We could run the test many times and try to "get lucky" by finding a failing execution, but that's not a very reliable testing approach. Even if the test does fail, it will be difficult to debug: we won't be able to easily catch the failure in a debugger, and every time we make a change, we will need to run the test many times to decide whether we fixed the issue.

Randomly testing concurrent code with Shuttle

Shuttle avoids this issue by controlling the scheduling of each thread in the program, and scheduling those threads randomly. By controlling the scheduling, Shuttle allows us to reproduce failing tests deterministically. By using random scheduling, with appropriate heuristics, Shuttle can still catch most (non-adversarial) concurrency bugs even though it is not an exhaustive checker.

A Shuttle version of the above test just wraps the test body in a call to Shuttle's check_random function, and replaces the concurrency-related imports from std with imports from shuttle:

use shuttle::sync::{Arc, Mutex};
use shuttle::thread;

shuttle::check_random(|| {
    let lock = Arc::new(Mutex::new(0u64));
    let lock2 = lock.clone();

    thread::spawn(move || {
        *lock.lock().unwrap() = 1;
    });

    assert_eq!(0, *lock2.lock().unwrap());
}, 100);

This test detects the assertion failure with extremely high probability (over 99.9999%).

Shuttle is inspired by the Loom library for testing concurrent Rust code. Shuttle focuses on randomized testing, rather than the exhaustive testing that Loom offers. This is a soundness—scalability trade-off: Shuttle is not sound (a passing Shuttle test does not prove the code is correct), but it scales to much larger test cases than Loom. Empirically, randomized testing is successful at finding most concurrency bugs, which tend not to be adversarial.

License

This project is licensed under the Apache-2.0 License.

Security

See CONTRIBUTING for more information.

Issues

Collection of the latest Issues

jamesbornholt

jamesbornholt

Comment Icon2

When a thread panics in a Shuttle test, generator produces an error that gets printed to stdout via tracing. This error is a red herring: it mentions generator::gen_impl and so suggests that maybe something went wrong in that code, when really it's just that a Shuttle thread panicked and that panic is being propagated. It would be nice to suppress this error so we don't lead users down the wrong debugging path.

jamesbornholt

jamesbornholt

Comment Icon0

#38 adds support for tracking causality using vector clocks. We think it might be possible for these vector clocks to replace epochs in the implementation of Condvar, since they capture the same information (an ordering between operations in the same condvar). This would hopefully make the Condvar implementation simpler and more obviously correct.

jamesbornholt

jamesbornholt

Comment Icon0

This test panics in the Drop impl of PoolItem:

The problem is that ContinueAfter makes us stop the test early, while some PoolItems are still alive. During cleanup we drop the Generator for each thread, which tries to safely clean up by unwinding their stacks, including dropping the PoolItem. But PoolItem's drop tries to acquire a lock, which is no longer allowed once ContinueAfter has triggered.

I'm not totally sure what to do here:

  • We could just leak the continuation's stack...
  • We could do something smarter during teardown, allowing threads to continue running until termination or deadlock, under the assumption that the only code that can run once we drop the continuations is cleanup code
Versions

Find the latest versions by id

v0.1.0 - Apr 11, 2022

  • Implement Condvar::wait_while and Condvar::wait_timeout_while (#59)
  • Remove implicit Sized bounds on Mutex and RwLock (#62)
  • Dependency updates (#58, #60)

v0.0.7 - Sep 21, 2021

  • Fix a number of issues in support for async tasks (#50, #51, #52, #54)
  • Improve error messages when using Shuttle primitives outside a Shuttle test (#42)
  • Add support for thread local storage (the thread_local! macro) (#43, #53)
  • Add support for Once cells (#49)
  • Simplify some dependencies to improve build times (#55)
  • Move context_switches and my_clock functions into a new current module (#56)

v0.0.6 - Jul 08, 2021

  • Add support for std::sync::atomic (#33)
  • Add shuttle::context_switches to get a logical clock for an execution (#37)
  • Track causality between threads (#38)
  • Better handling for double panics and poisoned locks (#30, #40)
  • Add option to not persist failures (#34)

v0.0.5 - Jun 11, 2021

  • Fix a performance regression with tracing introduced by #24 (#31)
  • Include default features for the rand crate to fix compilation issues (#29)

v0.0.4 - Jun 02, 2021

  • Add a timeout option to run tests for a fixed amount of time (#25)
  • Include task ID in all tracing log output (#24)
  • Implement thread::current (#23)

v0.0.3 - Apr 14, 2021

  • Update for Rust 1.51 (#11)
  • Add option to bound how many steps a test runs on each iterations (#14)
  • Remove option to configure the maximum number of threads/tasks (#16, #19)
  • Make yield_now a hint to the scheduler to allow validating busy loops (#18)
  • Add ReplayScheduler::new_from_file (#20)

v0.0.2 - Mar 19, 2021

  • Add Default impl to RwLock (#7)
  • Add option to persist schedules to a file (#4)

v0.0.1 - Mar 03, 2021

Initial release of Shuttle

Information - Updated Apr 28, 2022

Stars: 239
Forks: 14
Issues: 3

Repositories & Extras

macOS/iOS Security framework for Rust

MIT license (LICENSE-MIT or

macOS/iOS Security framework for Rust
Actix

2.6K

Curated examples using the Actix ecosystem

Zero2prod : Source code of zero to production series Triox : A free file hosting server that focuses on speed, reliability and security

Curated examples using the Actix ecosystem

binserve :zap::crab:

A blazingly fast static web server with routing, templating, and security in a single binary you can set up with zero code

binserve :zap::crab:

libdiffuzz: security-oriented alternative to Memory Sanitizer

This is a drop-in replacement for OS memory allocator that can be used to detect uses of uninitialized memory

libdiffuzz: security-oriented alternative to Memory Sanitizer

Transparent endpoint security

Block and detect advanced attacks

Transparent endpoint security

Rust Language Security

execrices: RUSTSEC-2021-0001

Rust Language Security

No security audit has been performed

There are currently 3 interesting things

No security audit has been performed

security-keys-rust

Many thanks to the authors of the openpgp-card Rust crate

security-keys-rust

Owlyshield open source security platform

An OSS security platform written in rust with security threat detection

Owlyshield open source security platform

No security audit has been performed

There are currently 3 interesting things

No security audit has been performed

weggli is a fast and robust semantic search tool for C and C++ codebases

It is designed to help security researchers identify interesting functionality in large codebases

weggli is a fast and robust semantic search tool for C and C++ codebases

No security audit has been performed

There are currently 5 interesting things

No security audit has been performed
Facebook Instagram Twitter GitHub Dribbble
Privacy