stepancheg/rust-protobuf

Protobuf implementation in Rust

Has runtime library for generated code

rust-protobuf

Protobuf implementation in Rust.

  • Written in pure rust
  • Generate rust code
  • (Coded{Input|Output}Stream impl)
  • Supports both Protobuf versions 2 and 3

List of crates

rust-protobuf — repository provides multiple crates:

  • protobuf — protobuf runtime
  • protobuf-codegen — protobuf codegen engine and protoc-gen-rust plugin for protoc command
  • protoc — programmatically work with protoc command
  • protoc-rust — codegen which can be invoked programmatically using protoc binary (e. g. from build.rs)
  • protobuf-codegen-pure — pure rust codegen
  • protoc-bin-vendoredprotoc binary packaged as crate, can be used with protoc or protoc-rust crates

About versions and branches

  • 2.*.* is the latest stable version. 2.*.* versions follow semver conventions
  • versions below 2 are no longer supported

See CHANGELOG.md for a list of changes and compatility issues between versions.

How to generate rust code

There are several ways to generate rust code from .proto files:

  • Invoke protoc programmatically with protoc-rust crate (protoc-rust crate). Reliable, but depends on protoc binary.
  • Use pure rust protobuf parser and code generator (protobuf-codegen-pure crate). Convenient, but may generate incorrect code or fail in complex definitions.
  • Use protoc-gen-rust plugin for Google's protoc command. This is useful to generate code manually or when using external build system.

Generated code

Have a look at generated files, used internally in rust-protobuf:

  • descriptor.rs for descriptor.proto (that is part of Google protobuf)

Rustdoc

docs.rs hosts rustdoc for protobuf.

Getting help

Feel free to open an issue if you need help with rust-protobuf.

Copy-on-write

Rust-protobuf can be used with bytes crate.

To enable Bytes you need to:

  1. Enable with-bytes feature in rust-protobuf:
[dependencies]
protobuf = { version = "2", features = ["with-bytes"] }
  1. Enable bytes option

with Customize when codegen is invoked programmatically:

With stable rust-protobuf:

protoc_rust::run(protoc_rust::Args {
    ...
    customize: Customize {
        carllerche_bytes_for_bytes: Some(true),
        carllerche_bytes_for_string: Some(true),
        ..Default::default()
    },
 });

With rust-protobuf from master:

protoc_rust::Args::new()
    ...
    .customize(Customize {
        carllerche_bytes_for_bytes: Some(true),
        carllerche_bytes_for_string: Some(true),
        ..Default::default()
    })
    .run()?;

or in .proto file:

import "rustproto.proto";

option (rustproto.carllerche_bytes_for_bytes_all) = true;
option (rustproto.carllerche_bytes_for_string_all) = true;

With these options enabled, fields of type bytes or string are generated as Bytes or Chars respectively. When CodedInputStream is constructed from Bytes object, fields of these types get subslices of original Bytes object, instead of being allocated on heap.

serde_derive support

(Only in master, not released yet)

Rust-protobuf can be used with serde.

To enable serde you need to:

  1. Enable serde option

with Customize when codegen is invoked programmatically:

with stable rust-protobuf:

protoc_rust::run(protoc_rust::Args {
    ...
    customize: Customize {
        serde_derive: Some(true),
        ..Default::default()
    },
});

with rust-protobuf from master:

protoc_rust::Args::new()
    ...
    .customize(Customize {
        serde_derive: Some(true),
        ..Default::default()
    })
    .run()?;

or in .proto file:

import "rustproto.proto";

option (rustproto.serde_derive_all) = true;

You may now Serialize and Deserialize messages:

let my_message = MyMessage::new();
serde_json::to_string(&my_message).unwrap();

Related projects

  • quick-protobuf — alternative protobuf implementation in Rust
  • prost — another protobuf implementation in Rust
  • serde-protobuf
  • grpc-rust — implementation of gRPC based on this library
  • grpc-rs — another gRPC implementation for Rust
Issues

Collection of the latest Issues

0liCom

0liCom

Comment Icon2

I'm developing a network application that uses protobuf to serialize and de-serialize messages that mostly consist of proto3 bytes fields. Since my application serves as a middle-layer to other applications, I would like to use as few copy operations during (de-)serialization of the messages with special focus on the large bytes-type payloads.

I already use the carllerche_bytes feature to achieve performant deserialization, but for serialization, there is no support for zero-copy serialization in the CodedOutputStream, since it always creates a copy in the internal buffer before flushing.

More explicitly, the CodedOutputStream::write_raw_bytes() function copies bytes to the internal buffer and CodedOutputStream::flush() calls CodedOutputStream::refresh_buffer to flush the internal buffer. When I provide some Message, that mostly consists of a large bytes field, with a Write-trait-object to serialize to, a buffer will always be created to copy this possibly very large (multiple kB) into the buffer before sending the data to the provided writer. This design prohibits me to implement network buffering myself after the serialization without creating a second copy of the transmitted bytes.

Would it be possible to provide some unbuffered CodedOutputStream implementation? I would also contribute it myself, if you would be interested in having such a feature in this crate.

nyurik

nyurik

Comment Icon2

Delta-encoding seems to be a common PBF-encoding pattern, where a repeated field is encoded as delta from the previous value rather than absolute values. Each user of the lib has to implement the same logic. Would it be possible to add encoder/decoder for such cases?

Simple case - an iterator of i64 that reads Vec<i64>, e.g. this read implementation.

For encoding, perhaps an iterator that encodes values as they pass through, so it can be converted to a vector with collect()? Thx!

stepancheg

stepancheg

Comment Icon2

Is there a demand for no_std feature?

shanicky

shanicky

Comment Icon10

Hi, stepancheg!

We recently came across a simple requirement when using the rust-protobuf library, we want to dynamically parse the protobuf file at runtime and generate the corresponding Descriptors, we searched for a while and currently use protobuf_codegen_pure::parse_and_typecheck, which works fine, but we went deeper into the code and saw protobuf_codegen_pure::model and protobuf_codegen_pure::parser, but they are both private, it feels more convenient to use this directly, we are curious about these two can be made public?

Many thanks!

scullionw

scullionw

Comment Icon1

Hi and thanks for the awesome crate! From what I've tested it is the only one that supports proto2 extensions, so thank you!

I am having issues printing out these extended messages though..

For example, I can access and print an extension to a message with after retrieving the extension message,

but I am looking for a way to automatically print it when printing the whole message. ie I would like the extension to be included in the message print:

currently on the master branch it only prints out the unknown fields as bytes:

since the generated extensions are aware of what messages they are extensions of (transit_trip_descriptor_extension knows it is an extensions of TripDescriptor):

i think this should be possible. Could someone point me in the right direction?

iitalics

iitalics

Comment Icon1

flush calls .write_all() on the underlying std::io::Write but this does not necessarily actually flush the writer. For instance, if you try using stdout as the writer you will notice that data does not get flushed.

rob-brown

rob-brown

Comment Icon1

By convention, our gRPC services are structured like this:

Unfortunately, this library generates Rust files in a single directory and use the same name as the proto file. This ultimately means the services clobber each other's files.

I can see there's an old issue regarding adding hierarchal support to the code generation. This would solve our problem. Though based on its age I don't expect it to be added any time soon. Are there other solutions or work arounds to this?

NicksWorld

NicksWorld

Comment Icon0

I received protobuf files that contained the following structure:

As far as I can tell, despite the extendee starting with .google it should be valid. I ran the same file through protoc on its own, and received no errors other than a warning that the protobuf version wasn't defined.

I got confused why it was panicking, then I noticed the following here. The code was throwing an error because .google.protobuf.FieldOptions was not equal to google.protobuf.FieldOptions.

Changing the original file to the following fixed the error, but I don't think it should be necessary to do.

Proto file: As seen above Generated file: N/A, Panics while generating protobuf-codegen-pure version: 2.14.0 Code used to build:

Protoc version: libprotoc 3.11.4 OS: Linux

Jasperav

Jasperav

Comment Icon8

Timestamp is a WellKnownType. I am confused how to fill this struct with the current time. Seconds is implemented as i64, and nano is implemented as i32. Are they both combined a time? Is there any way I can say: Give me a Timestamp where the time is equal to now (epoch time)?

jeffparsons

jeffparsons

rust-protobuf-v3
Comment Icon4

I'm working with hundreds of .proto files, so it's not practical for me to manually set up a module hierarchy to mirror the protobuf package hierarchy.

Some of these .proto files have the same name as others. So even if I did something in my build.rs like suggested here https://github.com/stepancheg/rust-protobuf/issues/324#issuecomment-476109276 then I'd still run into trouble with the output.

What I'd really like is an option in protobuf-codegen-pure (controlled via Customize) to automatically generate a directory hierarchy and Rust module hierarchy to match the protobuf package hierarchy. If you're open to the idea, I could take a swing at implementing this.

mverleg

mverleg

Comment Icon0

Using new gives an uninitialized object, and the standard Rust constructor with MyType { ... } needs one to include or 'derive' the unknown_fields and cached_size.

According to this page there's a convention to call such a method with_more_details since new is taken.

The main advantage would be that one can't forget a field, and one gets informed by the compiler if a new field is added that you don't initialize.

The generated code would be something like this:

#[derive(PartialEq, Clone, Default)]
pub struct Like {
    // message fields
    pub is_liked: bool,
    pub ts_changed: ::protobuf::SingularPtrField<::protobuf::well_known_types::Timestamp>,
    // special fields
    pub unknown_fields: ::protobuf::UnknownFields,
    pub cached_size: ::protobuf::CachedSize,
}

impl Like { pub fn new() -> Like { ::std::default::Default::default() }

<span class="hljs-keyword">pub</span> <span class="hljs-function"><span class="hljs-keyword">fn</span> <span class="hljs-title">with_more_details</span></span>(is_liked: <span class="hljs-built_in">bool</span>, ts_changed: ::protobuf::SingularPtrField&lt;::protobuf::well_known_types::Timestamp&gt;) -&gt; <span class="hljs-keyword">Self</span> {
    Like {
        is_liked,
        ts_changed,
        ..Like::new()
    }
}

<span class="hljs-comment">// ...</span>

}

Ten0

Ten0

Comment Icon8

Hello!

We are considering using gRPC in our monorepo with microservices. This means at the .protos are subject to change, but it's not an issue as long as these changes are properly taken into account in every project that uses that .proto.

My issue is :

  • The generated code creates structs that come with a ::new() constructor that sets everything to their std::default::Default values.
  • This implies that the compiler does not enforce that all the fields are set when creating the structure, like it does by default.
  • Specifically, if I update my .proto by adding a new field, this will not cause compile errors anywhere, and just leave the field with its std::default::Default value on places where I forget to update it, and probably cause a bug.

I'm trying to avoid this problem. I tried disabling the accessors generation by setting the corresponding options when generating the code in build.rs :

but this had no effect whatsoever.

The documentation also doesn't say that it removes the ::new() method, so I should assume that it wouldn't even if it worked.

Also, even if it had, it would be very bothering and unclean to have to set these fields everytime we instantiate a struct :

What is the right way to solve that problem ?

BusyJay

BusyJay

Comment Icon5

Consider following case:

A/n.proto defines a TestA message, and B/n.proto defines a TestB message. When import B/n.proto into A/m.proto, and use TestB, generated sources always import TestB via super::n::TestB. However, super::n is A/n.proto, which doesn't contain TestB, hence lead to compilation error.

Related to #189.

Information - Updated May 10, 2022

Stars: 2.0K
Forks: 305
Issues: 18

protobuf-convert

Macros for convenient serialization of Rust data structures into/from Protocol Buffers

protobuf-convert

This crate implements the Concise Binary Object Representation from RFC 7049

This crate implements the Concise Binary Object Representation from Serde, the generic serialization framework for Rust

This crate implements the Concise Binary Object Representation from RFC 7049

This crate is a Rust library for using the Serde serialization framework with

This crate is a Rust library for using the Gura file format

This crate is a Rust library for using the Serde serialization framework with

embedded-msgpack

MessagePack serialization implementation for Rust optimized for embedded environments

embedded-msgpack
JSON

300

Custom de/serialization functions for Rust's serde

Apply a prefix to each field name of a struct, without changing the de/serialize implementations of the struct using serde_with::rust::StringWithSeparator::&lt;CommaSeparator&gt;

Custom de/serialization functions for Rust's serde
JSON

108

Easy-to-use REST client for Rust programming language that provides automatic serialization and deserialization from Rust...

Easy-to-use REST client for Rust programming language that provides automatic serialization and deserialization from Rust structs

Easy-to-use REST client for Rust programming language that provides automatic serialization and deserialization from Rust...

Experimental library for generating rust code for CBOR serialization from CDDL specs

Instead of hand-writing CBOR code and writing tests to make sure it matches your CDDL spec, it's much faster to just generate the code from...

Experimental library for generating rust code for CBOR serialization from CDDL specs

Universal data serialization utilities for Rust

can not or chose not to support Serde

Universal data serialization utilities for Rust

deser: an experimental serialization and deserialization library for Rust

Fast Compile Times: deser avoids excessive monomorphization by encouraging

deser: an experimental serialization and deserialization library for Rust

XDR RS Serialize

Xdr-rs-serialize is a library for facilitating the (de)serialization of rust

XDR RS Serialize

Serde UBJSON (Universal Binary JSON)

UBJSON serialization format implementation for Rust/Serde

Serde UBJSON (Universal Binary JSON)
Facebook Instagram Twitter GitHub Dribbble
Privacy