lukemathwalker/cargo-chef

Cache the dependencies of your Rust project and speed up your Docker builds

You can install cargo-chef from all the available tags on Dockerhub

cargo-chef

.


Table of Contents

  1. How to install
  2. How to use
  3. Benefits vs Limitations
  4. License

How To Install

You can install cargo-chef from crates.io with

cargo install cargo-chef

How to use

:warning: cargo-chef is not meant to be run locally
Its primary use-case is to speed up container builds by running BEFORE the actual source code is copied over. Don't run it on existing codebases to avoid having files being overwritten.

cargo-chef exposes two commands: prepare and cook:

cargo chef --help

cargo-chef

USAGE:
    cargo chef <SUBCOMMAND>

SUBCOMMANDS:
    cook       Re-hydrate the minimum project skeleton identified by `cargo chef prepare` and
               build it to cache dependencies
    prepare    Analyze the current project to determine the minimum subset of files (Cargo.lock
               and Cargo.toml manifests) required to build it and cache dependencies

prepare examines your project and builds a recipe that captures the set of information required to build your dependencies.

cargo chef prepare --recipe-path recipe.json

Nothing too mysterious going on here, you can examine the recipe.json file: it contains the skeleton of your project (e.g. all the Cargo.toml files with their relative path, the Cargo.lock file is available) plus a few additional pieces of information.
In particular it makes sure that all libraries and binaries are explicitly declared in their respective Cargo.toml files even if they can be found at the canonical default location (src/main.rs for a binary, src/lib.rs for a library).

The recipe.json is the equivalent of the Python requirements.txt file - it is the only input required for cargo chef cook, the command that will build out our dependencies:

cargo chef cook --recipe-path recipe.json

If you want to build in --release mode:

cargo chef cook --release --recipe-path recipe.json

You can leverage it in a Dockerfile:

FROM lukemathwalker/cargo-chef:latest-rust-1.53.0 as planner
WORKDIR app
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM lukemathwalker/cargo-chef:latest-rust-1.53.0 as cacher
WORKDIR app
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

FROM rust:1.53.0 as builder
WORKDIR app
COPY . .
# Copy over the cached dependencies
COPY --from=cacher /app/target target
COPY --from=cacher $CARGO_HOME $CARGO_HOME
RUN cargo build --release --bin app

FROM rust:1.53.0 as runtime
WORKDIR app
COPY --from=builder /app/target/release/app /usr/local/bin
ENTRYPOINT ["/usr/local/bin/app"]

We are using four stages: the first computes the recipe file, the second caches our dependencies, the third builds the binary and the fourth is our runtime environment.
As long as your dependencies do not change the recipe.json file will stay the same, therefore the outcome of cargo cargo chef cook --release --recipe-path recipe.json will be cached, massively speeding up your builds (up to 5x measured on some commercial projects).

Pre-built images

We offer lukemathwalker/cargo-chef as a pre-built Docker image equipped with both Rust and cargo-chef.

The tagging scheme is <cargo-chef version>-rust-<rust version>.
For example, 0.1.22-rust-1.53.0.
You can choose to get the latest version of either cargo-chef or rust by using:

  • latest-rust-1.53.0 (use latest cargo-chef with specific Rust version);
  • 0.1.22-rust-latest (use latest Rust with specific cargo-chef version).
    You can find all the available tags on Dockerhub.

:warning: You must use the same Rust version in all stages
If you use a different Rust version in one of the stages caching will not work as expected.

Without the pre-built image

If you do not want to use the lukemathwalker/cargo-chef image, you can simply install the CLI within the Dockerfile:

FROM rust:1.53.0 as planner
WORKDIR app
# We only pay the installation cost once, 
# it will be cached from the second build onwards
RUN cargo install cargo-chef 
COPY . .
RUN cargo chef prepare  --recipe-path recipe.json

FROM rust:1.53.0 as cacher
WORKDIR app
RUN cargo install cargo-chef
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

FROM rust:1.53.0 as builder
WORKDIR app
COPY . .
# Copy over the cached dependencies
COPY --from=cacher /app/target target
COPY --from=cacher $CARGO_HOME $CARGO_HOME
RUN cargo build --release --bin app

FROM rust:1.53.0 as runtime
WORKDIR app
COPY --from=builder /app/target/release/app /usr/local/bin
ENTRYPOINT ["/usr/local/bin/app"]

Benefits vs Limitations

cargo-chef has been tested on a few OpenSource projects and some of commercial projects, but our testing has definitely not exhausted the range of possibilities when it comes to cargo build customisations and we are sure that there are a few rough edges that will have to be smoothed out - please file issues on GitHub.

Benefits of cargo-chef:

A common alternative is to load a minimal main.rs into a container with Cargo.toml and Cargo.lock to build a Docker layer that consists of only your dependencies (more info here). This is fragile compared to cargo-chef which will instead:

  • automatically pick up all crates in a workspace (and new ones as they are added)
  • keep working when files or crates are moved around, which would instead require manual edits to the Dockerfile using the "manual" approach
  • generate fewer intermediate Docker layers (for workspaces)

Limitations and caveats:

  • cargo cook and cargo build must be executed from the same working directory. If you examine the *.d files under target/debug/deps for one of your projects using cat you will notice that they contain absolute paths referring to the project target directory. If moved around, cargo will not leverage them as cached dependencies;
  • cargo build will build local dependencies (outside of the current project) from scratch, even if they are unchanged, due to the reliance of its fingerprinting logic on timestamps (see this long issue on cargo's repository);

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Issues

Collection of the latest Issues

EdmundsEcho

EdmundsEcho

Comment Icon1

Thank you for putting this together. The time watching github download and compile the rust binaries is painful. Thank you for reducing the pain!

In part to better understand and perhaps generally help, I have the following questions:

Including --bin option in chef cook - could it help?

I have a workspace. One of the members is a bin. -- Is there any benefit to having chef cook called with the specific bin in the event I only want to build that bin using cargo build? In contrast for instance, right now we have to reflect the --target <value> in each call to chef cook and cargo build. This is not the case for the specific bin we plan to build. It is not an option right now, for which there might be a good reason, so I wanted to ask.

Might using --target-dir frustrate the cache benefits of chef?

In my working with chef I get the feeling that I prevent the proper functioning of chef if I set the ---target-dir feature (something that can also be set using ENV CARGO_TARGET_DIR) to a non-default value. Said differently, does where I build the final artifact matter to chef? The cargo docs suggest using this feature to help with caching issues, perhaps "two rights make a wrong" in this case :))?

The cargo index always gets updated

When I change something in the runtime layer that has nothing to do with the build layers, the cargo index at the very top of the process insists on updating. Is that related to what used to be an issue for workspaces? (https://github.com/rust-lang/cargo/pull/9393)

itamarst

itamarst

enhancement
Comment Icon2

cargo chef prepare is failing for me because it's looking for a dev dependency it can't find during a Docker build. Since I am going to build in release mode, looking for dev dependencies seems unnecessary.

I can work around it, but it would be nice if cargo chef prepare had a --release flag corresponding to cargo chef cook --release which just skipped any dev dependencies.

ThomWright

ThomWright

Comment Icon4

Problem

I'm working through Zero To Production, and early on decided to use a rust-toolchain.toml file to make it easy to re-use the same toolchain whenever I'm working on the repo.

I found that when using chargo-chef I wasn't getting nice quick builds, I assume because everything was getting built with an unexpected toolchain.

Workaround

Adding the rust-toolchain.toml file to .dockerignore solved the issue.

Proposed solution

I don't know whether this is something worth "fixing" inside cargo-chef, but it might be worth adding something to the documentation to warn people about this problem.

I expect that making Docker ignore the file is the correct thing to do here. In my case though, I'd forgotten I had the file, and didn't consider that it might be interfering with Docker builds. I expect I won't be the only one this happens to!

Reproduction

You can use my repo here to reproduce - comment out the appropriate line in .dockerignore to see the problem.

More details

For reference, the content of my rust-toolchain.toml file is:

And the relevant output from docker build --tag zero2prod --file Dockerfile . when the toolchain file wasn't ignored:

z4f1r0v

z4f1r0v

Comment Icon3

Hello,

I'm following the familiar Zero To Production book and am facing a problem building a docker image from one of the versions of the Dockerfile. Here it is:

Building the image gives me:

I'm running this on a M1 Mac. From searching about the topic it seems like this problem is related to my machine. Still decided to ask since I'm not too familiar with the workings of cargo-chef. I would appreciate any hints!

Veetaha

Veetaha

bug
Comment Icon5

Suppose we have a crate foo

and we have a crate bar

This project layout where a local crate depends on other local crate that is renamed doesn't work with the logic here: https://github.com/LukeMathWalker/cargo-chef/blob/30e5b862483790f13c337acacf2c60f6cde7ac43/src/skeleton/version_masking.rs#L3-L22

cargo chef replaces the versions of all local crates with 0.0.1, however it doesn't modify the version requirement in bar's [dependencies] for foo2, so cargo chef cook fails with

Leandros

Leandros

bug
Comment Icon8

I'm building a project which has the same library as a dependency several times, due to being required by my dependencies in several versions.

This results in an error from cargo-chef:

getrandom is indeed multiple times in my lockfile due to it being required by several versions of rand

colelawrence

colelawrence

waiting for information
Comment Icon5

I'm not exactly sure if this is happening as a result of cargo chef, but it seems to be based on my investigations into our dive results. image If this sounds like something that isn't a dumb mistake on my part, I can work on making a repro.

KoltesDigital

KoltesDigital

waiting for information
Comment Icon3

Hi,

I'm building a Docker image using Alpine. I'm taking the path of building from Alpine instead of cross-compiling. In order to statically link libraries, I believe I must add RUSTFLAGS='-C target-feature=-crt-static' to cargo build commands, otherwise the built app throws SIGSEGV. I noticed that doing so, packages are recompiled by cargo build, while before I added RUSTFLAGS it was using cached builds as expected.

Reproduction steps:

(Update: I know this must not be done this way because cargo chef will erase main.rs, but this does show the bug.)

In spite of cargo chef forwarding the environment variables, the last line recompiles all dependencies. Do you have any idea why?

Besides, I'm surprised you don't mention this flag in your Alpine section of the README. Don't you need it?

benwis

benwis

waiting for information
Comment Icon8

Hello! I'm working on deploying my workspace app with Docker and cargo chef, but I'm getting a panic trying to build it. Not sure what would be useful for you to help me debug it, let me know if I can provide anything.

Cargo.toml for main crate

Cargo.toml for tests crate

Thank you for any help you can provide!

prestontw

prestontw

Comment Icon0

(The repo this issue is based on is private. I will describe the issue first, then try to recreate in a public minimal repo for further clarity and reproducibility.)

I have a project directory with structure

Each of the services has individual Cargo.toml files and its own Dockerfile.

I've tried adding cargo-chef to one of these services (currently, we are either not being cache friendly or using a cargo new approach), say, serviceZ. cargo-chef does excellently at caching external dependencies for the entire workspace, but because it skelefies libA, libB, and libC, it has to rebuild these local library dependencies for each change in serviceZ. (The actual repo is too large to individually specify what dependencies to copy over and doing so would lead to keeping dependencies between the project and the Cargo.toml file in sync.) These end up consuming most of the compilation time: using cargo chef can bring subsequent runs from 404s down to 270s, but using a cargo new approach brings subsequent runs down from 404s to 70s. I can also visually see that we are recompiling local dependencies when running cargo build in the cargo chef approach while only serviceZ is compiled in the cargo new approach.

This is related to https://github.com/LukeMathWalker/cargo-chef/issues/74 and https://github.com/LukeMathWalker/cargo-chef/issues/64 in that I want very fine control over what I skelefy.

A quick aside: the cargo new approach suffers from changes in serviceZ invalidating the entire COPY src src line, leading to discarding the dependency cache on source changes. The way I've gotten around it is to separate this process into several stages, similar to the approach in cargo chef, but instead of sharing the recipe.json file between stages, I'm sharing an entire directory that should be the same between runs:

prestontw

prestontw

enhancement
Comment Icon4

Howdy! I work in a medium-large workspace that includes patches to a couple of dependencies (can provide a minimal example if that would be helpful). When trying to use cargo chef cook, it doesn't seem like the patches are included or applied---I'm seeing compilation errors.

Would it be possible to add rust-patches support to cargo-chef?

dpc

dpc

help wanted
Comment Icon6

I know, I have a most complicated build that ever existed... :D

For complicated reasons, that I can't get into RN my project structure is more or less:

The client and server are a different workspace. But they both include the api as path = "../api".

So I'm running cargo-chef inside both server and client, but then cargo chef cook gets totally confused, and says:

Even though path ... should be ... Hmmm... /app/api/Cargo.toml . Seems like some path concatenation somewhere where wrong WRT ...

Versions

Find the latest versions by id

v0.1.35 - Mar 22, 2022

v0.1.34 - Feb 27, 2022

v0.1.33 - Nov 24, 2021

v0.1.32 - Nov 13, 2021

v0.1.31 - Sep 04, 2021

v0.1.30 - Sep 03, 2021

v0.1.29 - Sep 02, 2021

v0.1.28 - Aug 29, 2021

v0.1.27 - Aug 29, 2021

Information - Updated Apr 16, 2022

Stars: 337
Forks: 37
Issues: 10

quest-hook-template

A template for writing mods for Quest il2cpp games in Rust using cargo generate to clone the template:

quest-hook-template

Command line json text parsing and processing utility

parsing json compliant with rust and cargo

Command line json text parsing and processing utility

Clone this repo: git clone

If you don't have Rust and cargo-make installed,

Clone this repo: git clone

Rustup: the Rust installer and version management tool

To test that you have Rust and Cargo installed, you can run this in your terminal of choice: cargo --version

Rustup: the Rust installer and version management tool

Address generator in Rust

If you have Rust: cargo install gemgen

Address generator in Rust

First, complete the basic Rust setup instructions

Use Rust's native cargo command to build and launch the template node:

First, complete the basic Rust setup instructions

cargo rssc - Rust scripts for crates building

will copy the template_basic into scripts_rssc folder

cargo rssc - Rust scripts for crates building

NES Emulator in Rust-WASM

Requires Rust with cargo, nodejs, and wasm-pack

NES Emulator in Rust-WASM

Terraformer Engine

A 3D game engine but completely in rust/std/cargo

Terraformer Engine

Rustup: the Rust installer and version management tool

To test that you have Rust and Cargo installed, you can run this in your terminal of choice: cargo --version

Rustup: the Rust installer and version management tool
Facebook Instagram Twitter GitHub Dribbble
Privacy