dtolnay/typetag

Serde serializable and deserializable trait objects

This crate provides a macro for painless serialization of &dyn Trait trait

Typetag

.

This crate provides a macro for painless serialization of &dyn Trait trait objects and serialization + deserialization of Box<dyn Trait> trait objects.

Let's dive into the example and I'll explain some more below.

[dependencies]
typetag = "0.1"

Supports rustc 1.31+


Example

Suppose I have a trait WebEvent and I require that every implementation of the trait be serializable and deserializable so that I can send them to my ad-serving AI. Here are just the types and trait impls to start with:

trait WebEvent {
    fn inspect(&self);
}

#[derive(Serialize, Deserialize)]
struct PageLoad;

impl WebEvent for PageLoad {
    fn inspect(&self) {
        println!("200 milliseconds or bust");
    }
}

#[derive(Serialize, Deserialize)]
struct Click {
    x: i32,
    y: i32,
}

impl WebEvent for Click {
    fn inspect(&self) {
        println!("negative space between the ads: x={} y={}", self.x, self.y);
    }
}

We'll need to be able to send an arbitrary web event as JSON to the AI:

fn send_event_to_money_factory(event: &dyn WebEvent) -> Result<()> {
    let json = serde_json::to_string(event)?;
    somehow_send_json(json)?;
    Ok(())
}

and receive an arbitrary web event as JSON on the server side:

fn process_event_from_clickfarm(json: &str) -> Result<()> {
    let event: Box<dyn WebEvent> = serde_json::from_str(json)?;
    overanalyze(event)?;
    Ok(())
}

The introduction claimed that this would be painless but I'll let you be the judge.

First stick an attribute on top of the trait.

#[typetag::serde(tag = "type")]
trait WebEvent {
    fn inspect(&self);
}

Then stick a similar attribute on all those impl blocks too.

#[typetag::serde]
impl WebEvent for PageLoad {
    fn inspect(&self) {
        println!("200 milliseconds or bust");
    }
}

#[typetag::serde]
impl WebEvent for Click {
    fn inspect(&self) {
        println!("negative space between the ads: x={} y={}", self.x, self.y);
    }
}

And now it works as described. All in all, three lines were added!


What?

Trait objects are serialized by this library like Serde enums. Every impl of the trait (anywhere in the program) looks like one variant of the enum.

All three of Serde's tagged enum representations are supported. The one shown above is the "internally tagged" style so our two event types would be represented in JSON as:

{"type":"PageLoad"}
{"type":"Click","x":10,"y":10}

The choice of enum representation is controlled by the attribute that goes on the trait definition. Let's check out the "adjacently tagged" style:

#[typetag::serde(tag = "type", content = "value")]
trait WebEvent {
    fn inspect(&self);
}
{"type":"PageLoad","value":null}
{"type":"Click","value":{"x":10,"y":10}}

and the "externally tagged" style, which is Serde's default for enums:

#[typetag::serde]
trait WebEvent {
    fn inspect(&self);
}
{"PageLoad":null}
{"Click":{"x":10,"y":10}}

Separately, the value of the tag for a given trait impl may be defined as part of the attribute that goes on the trait impl. By default the tag will be the type name when no name is specified explicitly.

#[typetag::serde(name = "mouse_button_down")]
impl WebEvent for Click {
    fn inspect(&self) {
        println!("negative space between the ads: ({}, {})", self.x, self.y);
    }
}
{"type":"mouse_button_down","x":10,"y":10}

Conceptually all you're getting with this crate is that we build for you an enum in which every impl of the trait in your program is automatically registered as an enum variant. The behavior is the same as if you had written the enum yourself and implemented Serialize and Deserialize for the dyn Trait object in terms of the enum.

// generated (conceptually)
#[derive(Serialize, Deserialize)]
enum WebEvent {
    PageLoad(PageLoad),
    Click(Click),
    /* ... */
}

So many questions

  • Does it work if the trait impls are spread across different crates? Yes

    Serialization and deserialization both support every single impl of the trait across the dependency graph of the final program binary.

  • Does it work in non-self-describing data formats like Bincode? Yes

    All three choices of enum representation will round-trip correctly through compact binary formats including Bincode.

  • Does it support non-struct types? Yes

    The implementations of the trait can be structs, enums, primitives, or anything else supported by Serde. The Serialize and Deserialize impls may be derived or handwritten.

  • Didn't someone explain to me why this wasn't possible? Yes

    It might have been me.

  • Then how does it work?

    We use the inventory crate to produce a registry of impls of your trait, which is built on the ctor crate to hook up initialization functions that insert into the registry. The first Box<dyn Trait> deserialization will perform the work of iterating the registry and building a map of tags to deserialization functions. Subsequent deserializations find the right deserialization function in that map. The erased-serde crate is also involved, to do this all in a way that does not break object safety.


License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Issues

Collection of the latest Issues

leftadjoint

leftadjoint

Comment Icon0

I'd like to be able to iterate over the typetag inventory in order to generate a JSON schema inclusive of all of my trait's impls as of link time.

Unfortunately this doesn't seem to be supported in typetag; inventory seems to explicitly support it.

A simple example that fails:

This complains:

It seems that since the iterator is private, this might not be possible at the moment.

Is it possible to make the iterator part of the public API?

jhorstmann

jhorstmann

Comment Icon0

I was trying to serialize an abstract syntax tree for a chain of comparisons. The AST looks like the following

With about 100 nested expressions, the serialization fails with

The size of the serialized json string should be around 10-15kb, so running out of stack with a stack size of 2MB was really unexpected. Serializing a similar structure based on an enum instead of trait objects works until a nesting level of several thousands. The issue can be reproduced with the following standalone program:

dennybritz

dennybritz

Comment Icon3

I'm running into a similar issue as https://github.com/dtolnay/typetag/issues/34 where a variant is not found during deserialization when it is defined in a different module or crate. I'm opening this since the above is closed.

The workaround described in #34 does not work for me, or perhaps I don't understand it. I also can't reproduce this. I have several types defined in different crates and some of them seem to be loaded while others are not.

If this is a rustc bug (https://github.com/rust-lang/rust/issues/47384), is there a current workaround for it?

EDIT:

So far, the only workaround that I've found is to artificially use the struct:

Once I do that the type seem to become registered and I can deserialize it.

vi

vi

Comment Icon0

Is it possible to make a cfg-selected mode where you need to explicitly register all possible types manually in fn main instead of relying on intentory/ctor? Less automatic, more portable.

Or should such explicit mode be added to inventory first?

RDambrosio016

RDambrosio016

Comment Icon2

Hello!

I have structs similar to

where CstRule is typetagged as externally typed. what my goal is is to be able to parse the config from this:

where something_here and something_else are the "tags" i set for the typetag, like #[typetag::serde(name = "something_else")]. This structure of vectors does not currently work, because it ends up being a vector of objects, with each object having a single key being the name of the typetag. What i basically want to do is flatten that into an object with those keys. I also tried a HashMap<String, Box<dyn CstRule>> and flattened it, but that fails to deserialize for more than one item.

I would also like to make the deserialization case insensitive, e.g. something_else and SomethingElse both work. I tried using serde-aux's attribute for this but that does not work. And i cant exactly apply it to the enum which would be generated by typetag.

Id love to know if there is a way to do this without manually implementing deserialize 🙂

ryzhyk

ryzhyk

Comment Icon0

@dtolnay, thanks for the great serde ecosystem!

I have a program that declares several hundred struct's and enum's, all of which #derive Serialize and Deserialize. This includes ~300 "top-level" types that I serialize/deserialize via Box<dyn Trait> with the help of typetag. So there is exactly one trait in the program annotated with #[typetag::serde] and ~300 types that implement this trait, also annotated with #[typetag::serde].

When I list the largest symbols in the compiled release program using this command from your old post, I see a lot of erased_serde stuff:

What I don't fully understand is why erased_visit_XXX functions ended up being instantiated 1348 times each. I expected one per #[typetag]-annotated type, but it looks like there is an instantiation per each monomorphized type in the program. Is this correct? Do you know if there is a way to reduce typetag-related code?

Thank you!

toadzky

toadzky

Comment Icon1

I know that there are other issues related to this, but they all seem to be about something different and I'm not sure if this is actually the same thing.

I'm trying to implement the visitor pattern. I have 2 traits:

because of the generic accept method in the trait, which has nothing to do with the contents of the implementation struct, the type tag doesn't work.

i'd rather not use an enum for this because the struct have a bunch of fields and implementing traits on enums is a PITA.

Rua

Rua

Comment Icon2

I find the crate very useful but it doesn't quite do what I want. I would like to implement a custom deserialisation that deserialises a map of TypeName:Value pairs into a Vec<Box<dyn Trait>>. In other words, a variant where externally::TaggedVisitor::visit_map reads multiple pairs instead of just one and returns a Vec<Box<T>>. However, there are next to no public interfaces in the crate, which makes it a "take it or leave it" deal.

Having delved into the source code to figure out what's going on, I found that there is a registry that automatically gets filled with one deserialiser for each implementing type, indexed by name. Being able to access that would allow me to implement the rest myself. Alternatively, if you can think of some other way to enable me to do what I want, that would be fine too, but this seems like the way that requires the least changes.

nlewycky

nlewycky

Comment Icon5

I tried the web_event example and it doesn't work for me at current head post 0.1.4 (5d8ff37da6958e1096c1ee08f842f048762bf314).

I've tried both +stable and +nightly and I get the same error.

This is affecting us on https://github.com/wasmerio/wasmer where we serialize WebAssembly virtual filesystem state. So far I'm the only one affected, it passes on my coworker's computer and on our CI, but the new test was only added yesterday.

jiangliu

jiangliu

Comment Icon0

A trait with associated types fails to compile. Suggest to check for those conditions in impl/src/tagged_trait.rs::expand().

#[typetag::serde(tag = "type")] trait WebEvent { type A; fn inspect(&self); }

/typetag.git $ cargo run --example web_event Compiling typetag v0.1.3 (/ws/src/rust/typetag.git) error[E0191]: the value of the associated type A (from the trait WebEvent) must be specified --> examples/web_event.rs:5:1 | 5 | #[typetag::serde(tag = "type")] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ associated type A must be specified 6 | trait WebEvent { 7 | type A; | ------- A defined here

error: aborting due to previous error

a-rodin

a-rodin

Comment Icon1

In addition to externally, internally, and adjacently tagged enum representations, serde also supports untagged representation.

It would be nice to have support for untagged representation in typetag too.

sphinxc0re

sphinxc0re

Comment Icon0

This seems like a feature serde itself would benefit from. Are there any future plans to integrate into the project?

ISibboI

ISibboI

Comment Icon2

Is it possible to set one of the trait impls as the default impl, that is chosen when no type is given?

Say I have a trait T, and multiple types A, B, C. I would like to be able to set A as the default. So I would for example write:

This would be a really nice feature for me, and I think in general this crate could benefit from that.

maplant

maplant

Comment Icon8

I have a pretty simple example of a generic impl that I am very excited to use typetag to deserialize eventually, so thus I would like to open this issue as a friendly reminder of the limitation :-) Here is my example:

Versions

Find the latest versions by id

0.1.8 - Nov 10, 2021

  • Update to inventory 0.2 (#42)

0.1.7 - Jan 27, 2021

  • Ship LICENSE files in typetag-impl crate (#32)

0.1.6 - Sep 21, 2020

  • Fix noncompilable code when one typetag trait is used as a supertrait of another (#28)

0.1.5 - May 29, 2020

  • Support impls where the Self type is a macro metavariable, as in impl Trait for $ty {...} (#22)

0.1.4 - Aug 15, 2019

  • Update to Syn 1.0

0.1.3 - May 30, 2019

  • Support building with #[deny(missing_docs)] (#9)

0.1.2 - Apr 07, 2019

  • Support serializing and deserializing trait objects with marker traits, like Box<MyTrait + Send>

0.1.1 - Jan 24, 2019

  • Fix parse error on pub traits
  • Insert marker trait bounds to make Serialize and Deserialize supertrait requirement visible in rustdoc
  • Documentation improvements

0.1.0 - Jan 23, 2019

Initial release!

Information - Updated Apr 24, 2022

Stars: 547
Forks: 16
Issues: 19

Repositories & Extras

The arkworks ecosystem consist of Rust libraries for designing and working with zero knowledge succinct...

This library is released under the MIT License and the Apache v2 License (see License)

The arkworks ecosystem consist of Rust libraries for designing and working with zero knowledge succinct...

Bespoke protocol and high-level implementation of Non-fungible token (NFT) technology 🚀

This project is duel-licensed under both the Apache licenses, so feel free to use either at your discretion

Bespoke protocol and high-level implementation of Non-fungible token (NFT) technology 🚀

Bespoke protocol and high-level implementation of Non-fungible token (NFT) technology 🚀

This project is duel-licensed under both the Apache licenses, so feel free to use either at your discretion

Bespoke protocol and high-level implementation of Non-fungible token (NFT) technology 🚀
CLI

132

The simple cli library

This project is duel-licensed under both MIT and Apache, so feel free to use either at your discretion

The simple cli library
CLI

132

The simple cli library

This project is duel-licensed under both MIT and Apache, so feel free to use either at your discretion

The simple cli library

A Rust wrapper library for smealum's ctrulib

A Rust wrapper library for smealum's LICENSE-APACHE, COPYRIGHT for details

A Rust wrapper library for smealum's ctrulib

OpenSearch Rust Client

a community-driven, open source fork of elasticsearch-rs licensed under the Apache v2

OpenSearch Rust Client

A 'Space Invader' clone made with rust and made for the terminal

Inspired by Apache License (Version 2

A 'Space Invader' clone made with rust and made for the terminal

The compact yet complete benchmarking suite for Rust

This project is licensed under the Apache-2

The compact yet complete benchmarking suite for Rust

Rust SDK for Structured Expression Project Toolkit (SEPT)

Copyright 2021 by Victor Dods, licensed under Apache 2

Rust SDK for Structured Expression Project Toolkit (SEPT)
Facebook Instagram Twitter GitHub Dribbble
Privacy