google/cargo-raze

cargo-raze: Bazel BUILD generation for Rust Crates

An experimental support Cargo plugin for distilling a workspace-level

cargo-raze: Bazel BUILD generation for Rust Crates

An experimental support Cargo plugin for distilling a workspace-level Cargo.toml into BUILD targets that code using rules_rust can depend on directly.

Disclaimer

This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

Overview

This project synthesizes the dependency resolution logic and some of the functionality of Cargo such as features and build scripts into executable rules that Bazel can run to compile Rust crates. Though the standard rules_rust rules can be used to compile Rust code from scratch, the fine granularity of the dependency ecosystem makes transforming dependency trees based on that ecosystem onerous, even for code with few dependencies.

Usage

cargo-raze can generate buildable targets in one of two modes: Vendoring, or Non-Vendoring. In the vendoring mode, developers use the common cargo vendor subcommand to retrieve the dependencies indicated by their workspace Cargo.toml into directories that cargo-raze then populates with BUILD files. In the non-vendoring mode, cargo-raze generates a flat list of BUILD files, and a workspace-level macro that can be invoked in the WORKSPACE file to pull down the dependencies automatically in similar fashion to Cargo itself.

In both cases, the first step is to decide where to situate the Cargo dependencies in the workspace. This library was designed with monorepos in mind, where an organization decides upon a set of dependencies that everyone points at. It is intended that stakeholders in the dependencies collaborate to upgrade dependencies atomically, and fix breakages across their codebase simultaneously. In the event that this isn't feasible, it is still possible to use cargo-raze in a decentralized scenario, but it's unlikely that such decoupled repositories would interact well together with the current implementation.

Regardless of the approach chosen, the rust_rules should be brought in to the WORKSPACE. Here is an example:

Generate a Cargo.toml

For Bazel only projects, users should first generate a standard Cargo.toml with the dependencies of interest. Take care to include a [lib] directive so that Cargo does not complain about missing source files for this mock crate. Here is an example:

Once the standard Cargo.toml is in place, add the [package.metadata.raze] directives per the next section.

Using existing Cargo.toml

Almost all canonical cargo setups should be able to function inplace with cargo-raze. Assuming the Cargo workspace is now nested under a Bazel workspace, Users can simply add RazeSettings to their Cargo.toml files to be used for generating Bazel files

Cargo workspace projects

In projects that use cargo workspaces uses should organize all of their raze settings into the [workspace.metadata.raze] field in the top level Cargo.toml file which contains the [workspace] definition. These settings should be identical to the ones seen in [package.metadata.raze] in the previous section. However, crate settings may still be placed in the Cargo.toml files of the workspace members:

Remote Dependency Mode

In Remote mode, a directory similar to the vendoring mode is selected. In this case, though, it contains only BUILD files, a vendoring instruction for the WORKSPACE, and aliases to the explicit dependencies. Slightly different plumbing is required.

This tells Raze not to expect the dependencies to be vendored and to generate different files.

Generate buildable targets

First, install cargo-raze.

Next, execute cargo raze from within the cargo directory

Finally, invoke the remote library fetching function within your WORKSPACE:

This tells Bazel where to get the dependencies from, and how to build them: using the files generated into //cargo.

Note that this method's name depends on your gen_workspace_prefix setting.

You can depend on any explicit dependencies in any Rust rule by depending on //cargo:your_dependency_name.

Vendoring Mode

In Vendoring mode, a root directly is selected that will house the vendored dependencies and become the gateway to those build rules. //cargo is conventional, but //third_party/cargo may be desirable to satisfy organizational needs. Vendoring directly into root isn't well supported due to implementation-specific idiosyncracies, but it may be supported in the future. From here forward, //cargo will be the assumed directory.

Generate buildable targets (vendored)

First, install the required tools for vendoring and generating BUILDable targets.

Following that, vendor your dependencies from within the cargo/ directory. This will also update your Cargo.lock file.

Finally, generate your BUILD files, again from within the cargo/ directory

You can now depend on any explicit dependencies in any Rust rule by depending on //cargo:your_dependency_name.

Using cargo-raze through Bazel

Cargo-raze can be built entirely in Bazel and used without needing to setup cargo on the host machine. To do so, simply add the following to the WORKSPACE file in your project:

With this in place, users can run the @cargo_raze//:raze target to generate new BUILD files. eg:

Note that users using the vendored genmode will still have to vendor their dependencies somehow as cargo-raze does not currently do this for you.

Handling Unconventional Crates

Some crates execute a "build script", which, while technically unrestricted in what it can do, usually does one of a few common things.

All options noted below are enumerated in the src/settings.rs file.

Crates that generate files using locally known information

In some cases, a crate uses only basic information in order to generate a Rust source file. These build-scripts rules can actually be executed and used within Bazel by including a directive in your Cargo.toml prior to generation:

This setting tells cargo-raze to generate a rust_binary target for the build script and to direct its generated (OUT_DIR-style) outputs to the parent crate.

Crates that depend on certain flags being determined by a build script

Some build scripts conditionally emit directives to stdout that Cargo knows how to propagate. Unfortunately, its not so simple to manage build-time generated dependency information, so if the flags are statically known (perhaps, since the compilation target is statically known), they can be provided from within the Cargo.toml, in the following manner

Flags provided in this manner are directly handed to rustc. It may be helpful to refer to the build-script section of the documentation to interpret build scripts and stdout directives that are encountered, available here: https://doc.rust-lang.org/cargo/reference/build-scripts.html

Crates that need system libraries

There are two ways to provide system libraries that a crate needs for compilation. The first is to vendor the system library directly, craft a BUILD rule for it, and add the dependency to the corresponding -sys crate. For openssl, this may in part look like:

In some cases, directly wiring up a local system dependency may be preferable. To do this, refer to the new_local_repository section of the Bazel documentation. For a precompiled version of llvm in a WORKSPACE, this may look something like:

In a few cases, the sys crate may need to be overridden entirely. This can be facilitated by removing and supplementing dependencies in the Cargo.toml, pre-generation:

Crates that supply useful binaries

Some crates provide useful binaries that themselves can be used as part of a compilation process: Bindgen is a great example. Bindgen produces Rust source files by processing C or C++ files. A directive can be added to the Cargo.toml to tell Bazel to expose such binaries for you:

Cargo-raze prefixes binary targets with cargo_bin_, as although Cargo permits binaries and libraries to share the same target name, Bazel disallows this.

Crates that only provide binaries

Currently, cargo does not gather metadata about crates that do not provide any libraries. This means that these specifying them in the [dependencies] section of your Cargo.toml file will not result in generated Bazel targets. Cargo-raze has a special field to handle these crates when using genmode = "Remote":

In the snippet above, the wasm-bindgen-cli crate is defined as binary dependency and Cargo-raze will ensure metadata for this and any other crate defined here are included in the resulting output directory. Lockfiles for targets specified under [package.metadata.raze.binary_deps] will be generated into a lockfiles directory inside the path specified by workspace_path.

Note that the binary_deps field can go in workspace and package metadata, however, only one definition of a binary dependency can exist at a time. If you have multiple packages that depend on a single binary dependency, that definition needs to be be moved to the workspace metadata.

Build scripts by default

Setting default_gen_buildrs to true will cause cargo-raze to generate build scripts for all crates that require them:

This setting is a trade-off between convenience and correctness. By enabling it, you should find many crates work without having to specify any flags explicitly, and without having to manually enable individual build scripts. But by turning it on, you are allowing all of the crates you are using to run arbitrary code at build time, and the actions they perform may not be hermetic.

Even with this setting enabled, you may still need to provide extra settings for a few crates. For example, the ring crate needs access to the source tree at build time:

If you wish to disable the build script on an individual crate, you can do so as follows:

FAQ

Why choose Bazel to build a Rust project?

Bazel ("fast", "correct", choose two) is a battle-tested build system used by Google to compile incredibly large, multilingual projects without duplicating effort, and without compromising on correctness. It accomplishes this in part by limiting what mechanisms a given compilation object can use to discover dependencies and by forcing buildable units to express the complete set of their dependencies. It expects two identical sets of build target inputs to produce a byte-for-byte equivalent final result.

In exchange, users are rewarded with a customizable and extensible build system that compiles any kind of compilable target and allows expressing "unconventional dependencies", such as Protobuf objects, precompiled graphics shaders, or generated code, while remaining fast and correct.

Its also probable (though not yet demonstrated with benchmarks) that large applications built with Bazel's strengths in mind: highly granular build units, will compile significantly faster as they are able to cache more aggressively and avoid recompilation of as much code while iterating.

Why try to integrate Cargo's dependencies into this build tool?

For better or worse, the Rust ecosystem heavily depends on Cargo crates in order to provide functionality that is often present in standard libraries. This is actually a fantastic thing for the evolution of the language, as it describes a structured process to stabilization (experimental crate -> 1.0 crate -> RFC -> inclusion in stdlib), but it means that people who lack access to this ecosystem must reinvent many wheels.

Putting that aside there are also fantastic crates that help Rust developers interact with industry standard systems and libraries which can greatly accelerate development in the language.

Why not build directly with Cargo / Why generate rustc invocations?

Though the burden of emulating Cargo's functionality (where possible at all!) is high, it appears to be the only way to maintain the guarantees (correctness, reproducibility) that Bazel depends on to stay performant. It is possible and likely with inflight RFCs that Cargo will become sufficiently flexible to allow it to be used directly for compilation but at this point in time it appears that maintaining a semblance of feature parity is actually easier than avoiding all of the sharp edges introduced by treating Cargo like the Rust compiler.

What is buildable right now with Bazel, and what is not?

With a little bit of elbow grease it is possible to build nearly everything, including projects that depend on openssl-sys. Many sys crates will require identifying the system library that they wrap, and either vendoring it into the project, or telling Bazel where it lives on your system. Some may require minor source tweaks, such as eliminating hardcoded cargo environment variable requirements. Fixes can be non-trivial in a few cases, but a good number of the most popular crates have been built in an example repo, available at https://github.com/acmcarther/cargo-raze-crater

Example Repos

See these examples of providing crate configuration:

Using vendored mode:

  • hello-cargo-library
  • complicated-cargo-library
  • non-cratesio

Using remote mode:

  • complicated-example
  • non-cratesio

Compiling OpenSSL:

  • openssl

The [package.metadata.raze] section is derived from a struct declared in impl/src/settings.rs.

Issues

Collection of the latest Issues

jgao54

jgao54

Comment Icon1

Currently Bazel will throw an error while parsing a build file if it identifies the same dep appearing in two select expressions being added together.

I encountered this issue with the raze-generated BUILD.sha2-0.9.5.bazel file:

This will result in the following error during parsing of the build file:

snowp

snowp

Comment Icon1

Using 0.15.0 cargo-raze to pull in axum (as a transitive dep of tonic) I run into the following compilation issue:

I was able to fix this locally by applying the following patch:

I'm fairly new to raze so I could be missing something, but in general I'd expect this to "just work"

joell

joell

Comment Icon0

In cargo raze 0.12.0, when generating Bazel rules for a workspace of multiple packages, the BUILD.bazel file for each package has alias rules for just the dependencies of that package.

In cargo raze 0.13.0, this behavior changed to add aliases for all the dependencies of every package in the workspace.

This causes Bazel alias conflicts when two packages within the workspace depend on different major versions of a crate (e.g., clap 2.x vs 3.x). In such cases, the Bazel project can no longer build.

It is possible that issue #474 might be reflecting this deeper problem.

maghoff

maghoff

Comment Icon1

When pulling in value-bag as a dependency with cargo-raze-0.15.0, an alias from Cargo.toml gets included in the generated rust_library rule, but it should apply to the generated cargo_build_script rule.

The dependency is declared in value-bag's Cargo.toml: https://github.com/sval-rs/value-bag/blob/v1.0.0-alpha.8/Cargo.toml#L100-L102

The generated bazel rules:

This does not build correctly, as build.rs expects a crate named rustc, which is an alias of version_check. Moving the aliases section from the rust_library rule to the cargo_build_script rule fixes this:

Cargo-raze should generate aliases for build-dependencies in the cargo_build_script, but they are currently generated in the rust_library rule.

jgao54

jgao54

Comment Icon2

Hello, I have an existing project that uses cargo workspace and I am converting it to build with Bazel; and I am using cargo-raze (v0.15.0) in Remote mode.

There are two packages: A and B. A's Cargo.toml contains:

B's Cargo.toml contains:

After running cargo raze, both versions are populated in the generated BUILD.bazel file with the same name:

Because of this, the project failed to build with:

Temporary workaround is to manually edit the BUILD.bazel file and remove one of the aliases.

I also tried to rename clap in A's cargo.toml (i.e. clap3 = {package = "clap", version = "3.0.14"}), however it did not affect the generated BUILD.bazel file.

(I saw https://github.com/google/cargo-raze/pull/282 and https://github.com/google/cargo-raze/pull/425, and thought this was fixed, but not familiar enough with cargo-raze to deduce why it didn't propagate the renaming)

maghoff

maghoff

Comment Icon5

I am unable to add clap as a dependency because it includes its README.md as part of its build, and this is not declared as an input to the build rules generated by raze. I expect to be able to use clap 🙂 More details:

Adding clap as a dependency by adding the following to Cargo.toml:

gives me the following error during build:

The line in question on GitHub: https://github.com/clap-rs/clap/blob/v3.0.14/src/lib.rs#L8

I can work around the problem by adding README.md to the data of the generated build rules, but that is a hack. Maybe #287 is the correct solution to this issue?

(By the way, clap_derive exposes the same issue)

tp-woven

tp-woven

Comment Icon0

Hi all,

I am trying to set up a simple Rust build for a small project. I tried following the instructions in the README, but no matter what I do, Bazel doesn't seem to like it.

When installing cargo-raze and running it "manually": When I try to run any Bazel command, it complains that it can't find I tried adding it to my WORKSPACE manually, and helps it get a little further along, but then when I try to add a dependency to any of my targets, attempting to build that target results in Bazel complaining it can't find its repository (i.e. This error happens both when using aliases and when using (the experimental) all_crate_deps().

When attempting to run cargo-raze through Bazel: Trying to bazel run @cargo_raze//:raze it complains about @bazel_skylib again. Adding that to my WORKSPACE manually causes it to complaining about @rules_foreign_cc (similar to issue #423).

I also tried to build some examples from the examples folder, but:

  • When running cargo-raze in any of the example folders, it doesn't appear to do anything, and no new files are created.
  • Attempting to build an example fails with error: error: linking with external/local_config_cc/cc_wrapper.sh failed: exit code: 1

(As a side note, it is a little unclear to me how the examples work - they contain some things like cargo folders and various .bzl and BUILD/BUILD.bazel files, and it is not clear which of those are supposed to be authored by the "user" and which should be generated by cargo-raze.)

Platform: macOS Monterey (Intel) Bazel version: 4.2.1 "Installed" cargo-raze version: 0.12.0 "Bazel" cargo-raze version: 0.12.0, 0.14.1

Dig-Doug

Dig-Doug

Comment Icon0

When trying to use reqwest, I'm seeing an issue where dependencies for a wasm32 target are missing from the generated BUILD file. As a result, when building for wasm I get errors related to missing dependencies, e.g. js-sys.

Example BUILD

Cargo.toml

If you look at the BUILD, you'll see it has one set of dependencies from the crate's config: [target.'cfg(not(target_arch = "wasm32"))'.dependencies] but is missing the wasm32 specific ones in [target.'cfg(target_arch = "wasm32")'.dependencies]

bcmyers

bcmyers

Comment Icon0

Building and running cargo raze directly with bazel like so

appears to depend on the --legacy_external_runfiles bazel flag being set to True.

Folks who have this turned off in their .bazelrc like so

experience this not very nice error message.

You can get around this by running the following command instead.

But this is not well documented.

So two questions:

  • Is it worth adding information about this in the documentation?
  • Is it worth modifying the code so that building/running cargo raze directly with bazel does not rely on --legacy_external_runfiles?

Bazel version: 4.2.0 Cargo raze version: 0.14.0 Rules Rust version: c0bdb55ec101133a051bfcf13ff3295f60055739

rbtcollins

rbtcollins

Comment Icon1

We had a workspace Cargo.toml with this:

This didn't customise the rust_library target in the crate BUILD...bazel file.

This did match though:

I don't have time to track it down, but my guess is the pre-release crate version.

PiotrSikora

PiotrSikora

Comment Icon1

errno crate lists multiple targets that depend on libc crate (Cargo.toml):

which gets translated by cargo raze into this (BUILD.errno-0.2.8.bazel):

However, such duplicates are disallowed in Bazel and bazel query treats it as an error (see: https://github.com/bazelbuild/bazel/issues/13785):

This could be solved by either generating a single select (as suggested here: https://github.com/google/cargo-raze/pull/437#issuecomment-901776797) or by combining all select conditions that evaluate into the same value into one (although, that would work only when dependencies are exactly the same and not simply overlapping). cc @illicitonion

XAMPPRocky

XAMPPRocky

Comment Icon4

Hello, we've using cargo-raze in a large mono repo-repo for awhile, and we're trying to move towards using the newer [workspace] support in cargo-raze in order to be able to support path dependencies.

However we ran into an issue with using our custom registry, because it seems like cargo-raze doesn't pick up the .cargo/config, and currently cargo-raze only allows replacing a single registry, and doesn't provide an option to have multiple registries as we use dependencies from both crates.io and our own private registry.

Would it be possible to add support for multiple registries to cargo-raze? I'd be willing to contribute the feature, but would appreciate some guidance on the codebase.

briansmith

briansmith

Comment Icon0

Sorry I didn't capture the error, but one of the dependencies of cargo-raze fails to build if cargo install cargo-raze is used. Instead cargo install --locked works. Perhaps the README.md should be updated to recommend the use of --locked and/or a daily CI job should ensure that cargo install without --locked works each day.

weixiao-huang

weixiao-huang

Comment Icon1

I want to generate BUILD files for a binary crate ONLY from git instead of crate.io, so I specify binary_deps below:

However, when I run cargo raze -v, I got error below

So how should I generate the binary_deps from a git repository?

rbtcollins

rbtcollins

Comment Icon1

This is similar - perhaps a dupe of #399

Background: cargo does not permit specifying a crate in git in a git repository that uses workspace configuration; instead the whole workspace is specified, and then cargo picks out the desired crate by introspection.

Consider this example: example.zip

This uses the protobuf and protoc-rust crates, but fails at compile time:

I'm not 100% sure - still learning bazel - but I think what is happening is that the protobuf-test crate build.rs is being compiled instead of the protobuf crate build.rs. I think this because if you look at the -test build.rs, it outputs build instructions for that file that is being complained about.

(Note - I didn't add local targets as they aren't needed to show the problem. bazel query ...:* - bazel build @raze__protoc_rust__3_0_0_pre//:protoc_rust will fail with the correct error)

Information - Updated Jun 20, 2022

Stars: 383
Forks: 100
Issues: 110

Repositories & Extras

DEPRECATED in favor of whatlang, which is native Rust and

DEPRECATED in favor of cld2 library from the Chromium project

DEPRECATED in favor of whatlang, which is native Rust and
CLI

104

Project details:

This project consists of two parts: the library that can be used to detect the

Project details:

A Rust wrapper library for devkitPro's libogc

A Rust wrapper library for devkitPro's Wii testing project for instructions on how to use this library

A Rust wrapper library for devkitPro's libogc

A fast math library for Rust written to support my machine learning project sickml

A fast math library for Rust written to support my machine learning project

A fast math library for Rust written to support my machine learning project sickml

Woff decoder library to convert WOFF file to SFNT

Of course you can use this library only with rust but project also includes 'C/C++' header file with wrapper functions to call code from Rust...

Woff decoder library to convert WOFF file to SFNT

A Rust implementation of the telemetry APIs of modern F1 video games

This project implements a client library for the telemetry APIs that are

A Rust implementation of the telemetry APIs of modern F1 video games

Library crate for common tasks when building rust projects

Intended for use with cargo-auto

Library crate for common tasks when building rust projects

A rust version of my CustomLibrary

Also works as a "standard" library of sorts for my projects

A rust version of my CustomLibrary

This project has reached the end of its development as a cryptographic library

Feel free to browse the code, and feel free to use it, but it will

This project has reached the end of its development as a cryptographic library

Bitcoin's libbitcoinconsensus with Rust bindings

This project builds the libbitcoinconsensus library from Bitcoin's C++ sources using cargo and provides Rust bindings to its API

Bitcoin's libbitcoinconsensus with Rust bindings
Facebook Instagram Twitter GitHub Dribbble
Privacy