serde-rs/serde

Serde is a framework for serializing and deserializing Rust data structures efficiently and generically

Rust Greatest JSON weapon is Serde with over 4.4K stars on github and a massive developer community. This is considered a core Rust library for every developer to learn in BRC's opinion

Serde  

Serde is a framework for serializing and deserializing Rust data structures efficiently and generically.


You may be looking for:

  • An overview of Serde
  • Data formats supported by Serde
  • Setting up #[derive(Serialize, Deserialize)]
  • Examples
  • API documentation
  • Release notes

Serde in action

Click to show Cargo.toml. Run this code in the playground.

Getting help

Serde is one of the most widely used Rust libraries so any place that Rustaceans congregate will be able to help you out. For chat, consider trying the #rust-questions or #rust-beginners channels of the unofficial community Discord (invite: https://discord.gg/rust-lang-community), the #rust-usage or #beginners channels of the official Rust Project Discord (invite: https://discord.gg/rust-lang), or the #general stream in Zulip. For asynchronous, consider the [rust] tag on StackOverflow, the /r/rust subreddit which has a pinned weekly easy questions post, or the Rust Discourse forum. It's acceptable to file a support issue in this repo but they tend not to get as many eyes as any of the above and may get closed without a response after some time.


License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Serde by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Issues

Collection of the latest Issues

Lucretiel

Lucretiel

Comment Icon1

Sometimes, Deserializers don't check to ensure that a SeqAccess or MapAccess was fully consumed during visitation (for instance, https://github.com/3Hren/msgpack-rust/issues/287). While those Deserializers should implement the proper checks (because there's no way to guarantee that the sequence was consumed), it would be nice if serde could help them out by having fixed-size collections call next_element anyway, to ensure that the collection is fully drained.

For example, the tuple implementation could be improved as follows:

This proposal applies both to derived Deserializers (eg, for tuple structs) and to built-in standard library implementations (eg, for [T; N]).

Related to https://github.com/serde-rs/serde/issues/1557

g302ge

g302ge

Comment Icon2

I have an enum struct as this :

And I have implemented a self-defined serializer for this enum target to insert a new field into the final str. look like this

But in this test case I got an error

The runtime complain that could not match any varints in enum, So if there some approach to fix this problem. If not, should we introduce a new attributed like this

The attributed #[serde(ignore = "inserted")] semantic is that this contianer should ignore the fields defined in ignore attributed when deserializing from a serialized item.

nsunderland1

nsunderland1

Comment Icon0

Given this code (playground link):

I expect not to get deprecation warnings. Instead, I get the following:

Notice that the derives for Default don't trigger any warnings (this seems to have been fixed some time ago).

Not aware of any workaround (other than undeprecating), and this leads to a bunch of warnings for things that are deprecated but that we still want to be able to support serialization for.

m4b

m4b

Comment Icon4

In process of stumbling on skip not working with e.g., bincode (ex: https://docs.rs/bincode/2.0.0-alpha.1/bincode/serde/index.html#known-issues) (which makes sense, given that format), it occurred to me after a suggestion by kvark for a "skip if human readable", and ralith about manual implementations using the is_human_readable method, that my usecase for using skip_serializing_if is actually more precisely, "skip serializing when the format is human readable".

Hence I propose tentatively an additional derive annotation:

Which would skip this field when the format is human readable, e.g., ron, json, but for something like bincode, it would serialize the empty vector (and would then allow deserialization to work correctly).

TheEnbyperor

TheEnbyperor

Comment Icon1

Given an enum with a name and an alias, such as below.

Then put in a struct as such, and flattened.

And when an enum value with the alias name is presented to the deserializer, an error such as "no variant of enum PSCNatureOfControls found in flattened data" is given. When the enum variant is changed from alias to rename, it works as expected.

sfackler

sfackler

Comment Icon9

The Deserialize implementation of a type with a field annotated #[serde(flatten)] currently buffers up all of the unknown map entries in memory, and then deserializes the flattened value from them at the end. This approach has a couple of downsides:

  1. It causes the deserializer to allocate when that would normally not be the case.
  2. It confuses "smart" deserializers - things like serde_ignored_value or serde_json's error position tracking don't work properly.
  3. It can lose out on special casing in deserializers based on hinting.

Fortunately, there's alternate approach that avoids all of these issues! Instead of buffering, we can create a custom MapAccess which intercepts fields that we want to handle in the top-level type, and then returning the remainder to the nested deserialize implementation: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=96546d7ba882b70042268a0b48de6286

Lucretiel

Lucretiel

Comment Icon0

Currently, StrDeserializer doesn't forward deserialize_newtype_struct to visit_newtype_struct, which means that trivial newtype structs fail to deserialize from it: playground. This seems like an obvious case to support, especially since StrDeserializer does support deserializing into unit enums.

PoignardAzur

PoignardAzur

Comment Icon0

There's been some activity recently around the benchmark suite published by lqd. Those benchmarks are a trove of useful data on how rust crates are built and where the time is spent.

Looking at the benchmarks, I noticed that in multi-core builds, quite a lot of crates had a bottleneck around serde and serde_derive.

Because serde reexports serde_derive's macro, the compilation order is always syn -> serde_derive -> serde, sequentially. In the benchmarks, this usually accounts for around 9s that cargo fails to parallelize; rustc can do some parallelizing of its own during the codegen phase, but it's still a lot of CPU usage left on the table.

Instead I suggest the crate structure should be:

  • serde
    • serde_core
    • serde_derive

With the serde crate being a thin wrapper re-exporting types from the other two. That way the compilation pipeline would be syn -> serde_derive on one side, and serde_core on the other.

Another advantage is that crates could import serde_core instead of serde if they don't care about the macros; that way, they wouldn't be blocked on the compilation of syn + serde even if some other crate in the tree imports serde with the derive feature.

Looking at the benchmarks, it looks like doing so could shave 2 or 3 seconds off total build time for multi-core builds. The benchmarks may not be representative of real projects (they test lib crates, not bins), though. Overall, the smaller the project, the more likely serde is to be in its critical path, so the more likely it is to see improvement from a refactoring.

Possible issues

  • Backwards compatibility (though it should be fine?)
  • Documentation: items can either appear as serde::Serialize or serde_core::Serialize. Not sure how much of a problem it actually is.
avnerbarr

avnerbarr

Comment Icon1

I want to deserialize the same field twice into 2 separate named fields

I've tried using "alias" as well as "rename" attributes but not successful:

once that is possible, I would like to also have a custom deserialize function for the second occurrence since I need to apply a slight modification

brianc118

brianc118

Comment Icon0

Based on https://serde.rs/attr-flatten.html#capture-additional-fields and example usage in #1179 I assumed that having a flattened other: HashMap<String, Value> at the end of a struct to be deserialized into will always return us unknown fields. In the above example, deserializing into OuterWithExtra gives us all fields that are supplied in the JSON, while also setting a and b correctly. This doesn't seem like the desired behavior?

If we don't have nested #[serde(flatten)] it behaves as expected. See playground.

I was exploring this as a way to detect unknown fields after finding that serde_ignored doesn't work with #[serde(flatten)]: https://github.com/dtolnay/serde-ignored/issues/10

bdbai

bdbai

Comment Icon0

Some data types like std::net::IpAddr has two different representations for human-readable and non-human-readable Deserializers. However, the built-in deserializers that work with enums do not recognize human-readability of the data format, and expect a representation different from what serializers produce. The following example shows a round trip serialization with msgpack that fails during deserialization.

In this example, a ContentRefDeserializer is used behind the scene to pick a suitable variant of HostName. The implementation does not override is_human_readable method on Deserializer trait, thus uses the default value true. Therefore, the non-human-readable representation of IpAddr {"V4": [8, 8, 8, 8]} is rejected. Any suggestions?

PS: #1919 seems related.

hasezoey

hasezoey

Comment Icon0

This is a proposal to make the type Attr public (and by extension Symbol)

Why

I am currently writing a derive add-on and i would like to not re-invent the wheel when the implementations already exist and can be "generically" used

Implementation

To implement the proposal, the following things have to be done:

  • make symbol::Symbol public
  • make attr::Attr public (cant be done without the above)
GeeWee

GeeWee

Comment Icon1

Reopen of issue #773 as there still seems to be interest, and there is someone who's started doing some work on it #1544 . Of ocurse, feel free to close, but I hope you'll reconsider this as important.

Further info: We have a lot of untagged enums in our public-facing API, and it's extremely difficult to give proper error messages to the user as all we can get out of Serde is data did not match any variant of untagged enum - if there was an opt-in to getting better error messages here, it would make our life a lot easier.

gtsiam

gtsiam

Comment Icon1

I wanted to take a crack at fully solving #912, but there are a couple of open questions I run into that I'd like to see comments on before I fully implement. Given the wide use of serde, the stability it has to provide and the fact that I wrote most of this while trying to figure the problem out anyway, I think this is somewhat required to make a potentially controversial change (plus I don't want to implement it only to have it rejected). So with that out of the way:

Terminology

  • Variant: Rust enum variant
  • Variant value: Data associated with a variant (unit, tuple or struct)
  • "Other" variant: (note the quotation marks) The enum variant tagged with #[serde(other)]
  • Tag: Name of the variant to parse
  • Content: Data associated with a variant to parse

Summary

Acceptable "Other" variants:

  • Unit variant (This is unchanged)
  • Newtype: (tag)
  • 2-tuple: (tag, content)

The tag and content of the enum will be deserialized into the appropriate fields if they are present.

Details

Unit variant

The unit variant case should remain unchanged for compatibility reasons, which means that it will not store the tag and that it will expect a unit as its content.

Tag

When a tag field is specified, it will be used to store the tag on deserialization and to set the tag on serialization. This would ensure that no information is lost when propagating unknown enums. If no tag information is stored, the variant name will be used for serialization same as any other variant.

Content

When a content field is specified, it will be used for deserialization, otherwise a unit will be required. The content can be ignored explicitly with serde::de::IgnoredAny. This is not done by default for consistency with the unit variant case.

Representing the tag

Serde supports a few different forms of tags (Mainly integers, strings and bytes). The requirement here is that the tag type can be (de)serialized from/to the tag used in the stored data (eg. String and &str should work for json, since serde_json uses strings to signify tags)

Relation to field_identifier

A limited form of the aforementioned newtype #[serde(other)] variant is already supported by making the last variant of a #[serde(field_identifier)] a newtype over a tag. This approach should be consistent with that.

Motivation

History

First of all, #1382 has already implemented an (almost) complete version of #[serde(other)] which works by parsing the "other" variant same as any other variant. However that behavior is not currently usable on master due to the other attribute being limited to unit variants.

The reason for said limitations was due to unresolved questions around how to recover the original tag and some unintuitive behavior in a few common use cases. First, since the value of the "other" variant corresponds directly to the content of the un-parsed enum, there is nowhere to store the tag. @roblabla proposed to add a #[serde(other_tag)] option on one of the fields in the "other" variant's value, which is what I was going to implement initially, however a few issues arise:

Unrepresentable variants

In serde, variants with no values (unit variants), tuple, newtype and struct variants are handled differently. That means that:

and

are not the same. So that raises the question: After removing the #[serde(other_tag)] from the content, what form should the variant try to parse from? Assuming we do the obvious:

Then that means that you can't pase the simple string case

Unintuitive behavior

Derived from @dtolney's comment on the original pr:

Non-obvious fail case

Open questions

  • This approach would require the tag and content to be part of the public api. Though the exact details could be hidden through newtypes over the tag and content, adding or removing a tag or content field could be a breaking change. I think this is acceptable, but this may be a point to discuss.
  • In a past revision, I intended to implement one more "other" variant form: { tag, content }, but I removed it due to complications with how it would interact with struct fields. Maybe for struct variants, it'd be a good idea to support some form of #[serde(other_tag)]?

Final note

My reservations revolve mostly around the question of how intuitive this all is, so I'd appreciate any feedback.

Edit History

  • Removed the { tag, content } variant form due to complications with field attributes. Leaving it up to future a future pr.
ereslibre

ereslibre

Comment Icon0

With the following data structures:

I am able to unmarshal what I understand should be rejected:

Producing:

Please, let me know if I'm misunderstanding the combination of deny_unknown_fields and internally tagged enums. If this is a bug I would be interested in contributing a fix.

Thanks for serde!

Versions

Find the latest versions by id

v1.0.136 - Jan 25, 2022

  • Improve default error message when Visitor fails to deserialize a u128 or i128 (#2167)

v1.0.135 - Jan 25, 2022

  • Update discord channels listed in readme

v1.0.134 - Jan 21, 2022

  • Improve error messages on deserializing NonZero integers from a 0 value (#2158)

v1.0.133 - Jan 01, 2022

  • Optimize deserialization of data structures that contain recursive use of flatten fields or tag or untagged enums (#2148)

v1.0.132 - Dec 16, 2021

  • Enable Serialize and Deserialize impls for std::sync::atomic::{AtomicI64, AtomicU64} on riscv64 arch (#2141, thanks @Avimitin)

v1.0.131 - Dec 09, 2021

  • Avoid unused_results being triggered in generated code for adjacently tagged enum (#2116, thanks @tyranron)

v1.0.130 - Aug 28, 2021

  • Provide MapAccess and SeqAccess impl for reference to a dynamically sized existing impl (#2081)

v1.0.129 - Aug 28, 2021

  • Support deserialization of remote structs that used packed repr (#2078, #2079, #2080)

v1.0.128 - Aug 21, 2021

  • Enable 128-bit integers on emscripten in Rust 1.40+ (#2076, thanks @Manishearth)

v1.0.127 - Jul 31, 2021

  • Resolve warning in rustc nightly-2021-07-31+ compiling serde_test

v1.0.126 - May 12, 2021

  • Resolve conflict with forbid(future_incompatible) lint setting in generated code (#2026, thanks @hyd-dev)

v1.0.125 - Mar 22, 2021

  • Improve performance of serializing Ipv4Addr (#2001, thanks @saethlin)

v1.0.124 - Mar 06, 2021

  • Fix possible panic deserializing invalid data as SystemTime (#1997, thanks @cyang1)

v1.0.123 - Jan 25, 2021

  • Support Self keywords in fields of types that derive Deserialize (#1830, thanks @taiki-e)
  • Allow floats to be deserialized from ints in tagged unions (#1842, thanks @Timmmm)
  • Support Self inside fields that use serialize_with (#1970)

v1.0.122 - Jan 25, 2021

  • Add IntoDeserializer impl for &[u8] (#1898, thanks @Mingun)

  • Handle unrecognized numeric field keys during deserialization of a field_identifier, equivalently to string field keys (#1914, thanks @Mingun)

  • Add attribute to override default deserialization failure expectation message (#1916, thanks @Mingun)

  • Improve serde_test handling of map entries and error message construction (#1918, thanks @Mingun)

  • Produce more accurate location information on test failures from serde_test crate (#1920, thanks @Mingun)

  • Improve diagnostic on failure to parse a rename_all attribute (#1960, #1961)

  • Eliminate unnecessary trait bounds on some value Deserializer impls (#1963)

v1.0.121 - Jan 23, 2021

  • Support borrowed data during deserialization of a field identifier (#1917, thanks @Mingun)
  • Fix panic when deserializing Duration with nanoseconds that cause the seconds counter to overflow (#1958, thanks @jonasbb)

v1.0.120 - Jan 19, 2021

  • Fix deserialization of ignored fields containing 128-bit integer (#1955, thanks @TheJokr)

v1.0.118 - Dec 05, 2020

  • Support serialization of core::num::Wrapping in no_std mode (#1926, thanks @attente)

v1.0.117 - Oct 15, 2020

v1.0.116 - Sep 11, 2020

  • Fix deserialization of IpAddr, SocketAddr, Bound, Result, OsString in formats which process variant identifiers as u64 (#1888, thanks @joshtriplett)

v1.0.115 - Aug 10, 2020

  • Support #[serde(flatten)] on a field whose type is a type parameter and concrete type is () (#1873)

v1.0.114 - Jun 22, 2020

  • Improve packed repr matching to support deriving Serialize for structs having repr(C, packed) (#1813, thanks @TannerRogalsky)

v1.0.113 - Jun 19, 2020

  • Improve diagnostic when a serde trait is not implemented (#1827, thanks @taiki-e)

v1.0.112 - Jun 14, 2020

  • Support serde(flatten) on map types where the Serializer only works with serialize_entry (#1837)

v1.0.111 - May 30, 2020

  • Process borrowed lifetimes inside of interpolated macro_rules metavariables, such as in the case of #[derive(Deserialize)] struct S<'a> { field: $field } (#1821)

v1.0.110 - May 10, 2020

  • Support deriving Serialize impl for a repr(packed) struct (#1791, thanks @alvardes)

v1.0.109 - May 10, 2020

  • Allow adjacently tagged newtype variants containing Option to omit the content field when deserializing (#1553, #1706, thanks @zth0)
  • Avoid panicking when a SystemTime older than UNIX_EPOCH is serialized (#1702, thanks @hjiayz)

v1.0.108 - May 10, 2020

  • Provide a Serializer impl that can write primitives and unit variants to a &mut fmt::Formatter (#1705, thanks @jethrogb)

v1.0.107 - May 08, 2020

  • Fix panic during macro expansion when using serde(skip) and serde(other) in the same enum (#1804)

v1.0.106 - Apr 03, 2020

  • Hide dummy const implementation detail from rustdoc when documenting a binary crate (#1768, thanks @robo9k)

Information - Updated Apr 12, 2022

Stars: 5.3K
Forks: 512
Issues: 187

Repositories & Extras

A fast and flexible CSV reader and writer for Rust, with support for Serde

Dual-licensed under MIT or the If you're new to Rust, the

A fast and flexible CSV reader and writer for Rust, with support for Serde

SIMD JSON for Rust  

Rust port of extremely fast serde compatibility

SIMD JSON for Rust  

This is an experimental serializer for Rust's serde ecosystem, which can convert Rust objects to...

This is an experimental serializer for Rust's serde ecosystem, which can convert Rust objects to Python values and back

This is an experimental serializer for Rust's serde ecosystem, which can convert Rust objects to...
JSON

111

A Rust JSON5 serializer and deserializer which speaks Serde

Deserialize a JSON5 string with from_str

A Rust JSON5 serializer and deserializer which speaks Serde

This crate is a Rust library for using the Serde serialization framework with

This crate is a Rust library for using the Gura file format

This crate is a Rust library for using the Serde serialization framework with
JSON

272

Custom de/serialization functions for Rust's serde

Apply a prefix to each field name of a struct, without changing the de/serialize implementations of the struct using serde_with::rust::StringWithSeparator::&lt;CommaSeparator&gt;

Custom de/serialization functions for Rust's serde

serde-reflection: Format Description and Code Generation for Serde

This project aims to bring the features of a traditional IDL to Rust and Serde

serde-reflection: Format Description and Code Generation for Serde

A Rust implementation of the JSON-RPC 2

A Rust implementation of the serde) for JSON-RPC

A Rust implementation of the JSON-RPC 2

Conveniently serialize and deserialize rust types into / from their serde name

This crate originated as a fork of serde_variant which is maintained by

Conveniently serialize and deserialize rust types into / from their serde name
Facebook Instagram Twitter GitHub Dribbble
Privacy