bnclabs/jsondata

Why yet another JSON package in Rust ?

======================================

Why yet another JSON package in Rust ?

This crate makes several trade-offs that are tuned for big-data and document database.

  • Support for 128-bit signed integers.
  • Deferred conversion for JSON numbers.
  • Serialization from Rust native type to JSON text.
  • De-serialization from JSON text to Rust native type.
  • CRUD operation on JSON documents, using JSON Pointer.
  • Sorted keys in property object.
  • Streaming JSON parser.
  • Support JSON5 standard.
  • Common arithmetic and logic operations.
  • Sortable JSON.

Useful links

  • API Documentation
  • JSON Pointer.
  • JSON5.
  • Rust internal discussion on f64 -> integer.
  • Json sort order.
  • Json operations.
  • Release notes.

Deferred conversion for numbers

Converting JSON numbers to Rust native type is not always desired. Especially in the context of big-data where data is stored in JSON format and we need to lookup, only, specific fields within the document.

This implementation provides deferred conversion for JSON numbers that leads to a performance improvement of upto 30%.

CRUD operations on JSON document

Using Json Pointer it is possible to identify a specific field nested within a JSON document. For Example, with below document:

  • /age shall point to value 26.
  • /tags shall point to value [ "officia", "reprehenderit", "magna" ].
  • /tags/0 shall point to value "officia".
  • /friends shall point to value [{"id": 0, "name": "Glenda Chan"}].
  • /friends/name shall point to value "Glenda Chan".

List of operations

  • Get a field nested within a JSON document using JSON Pointer.
  • Set a field nested within a JSON document.
  • Delete a field nested within a JSON document.
  • Append string or array field withing a JSON document.

JSON5

  • Object keys may be an ECMAScript 5.1 IdentifierName.
  • Objects may have a single trailing comma.
  • Arrays may have a single trailing comma.
  • Strings may be single quoted.
  • Strings may span multiple lines by escaping new line characters.
  • Strings may include character escapes.
  • Numbers may be hexadecimal.
  • Numbers may have a leading or trailing decimal point.
  • Numbers may be IEEE 754 positive infinity, negative infinity, and NaN.
  • Numbers may begin with an explicit plus sign.
  • Single and multi-line comments are allowed.
  • Additional white space characters are allowed.

Track this feature.

Sortable JSON

  • Null type shall sort before all other types.
  • Boolean type shall sort after Null type.
  • Number type shall sort after Boolean type.
    • f64 values that are <= -2^127 will sort before all i128 integers.
    • f64 values that are >= 2^127-1 will sort after all i128 integers.
    • NaN, Not a Number, values shall sort after all i128 integers
    • -Infinity shall sort before all numbers.
    • +Infinity shall sort after all numbers.
    • NaN shall sort after +Infinity.
  • String type shall sort after Number type.
  • Array type shall sort after String type.
  • Object type shall sort after Array type.
    • All (key,value) pairs within the object shall be presorted based on the key.
    • When comparing two objects, comparison shall start from first key and proceed to the last key.
    • If two keys are equal at a given position within the objects, then its corresponding values shall be compared.
    • When one object is a subset of another object, as in, if one object contain all the (key,value) properties that the other object has then it shall sort before the other object.

Useful links

  • A detailed description of JSON sort order.
  • Rust-lang issue#46298 and issue#10184, discussing saturating cast of f64 -> integer.
  • Rust internal discussion on f64 -> integer.
  • Unicode collation TR10.
  • ICU collation.
  • Floating point Total ordering
  • Total ordering for floating point in stackoverflow.
  • Total ordering thread in http://users.rust-lang.org.
  • A good blog on floating point, to get started.

Operations on JSON documents

  • Arithmetic operations, ADD, SUB, MUL, DIV, REM, NEG.
  • Bitwise operations, SHL, SHR, BITAND, BITOR, BITXOR.
  • Logical operations, NOT, AND, OR.
  • Index operations.
  • Range operations.

Detailed description can be found here.

Contribution

  • Simple workflow. Fork - Modify - Pull request.
  • Before creating a PR,
    • Run make build to confirm all versions of build is passing with 0 warnings and 0 errors.
    • Run check.sh with 0 warnings, 0 errors and all testcases passing.
    • Run perf.sh with 0 warnings, 0 errors and all testcases passing.
    • Install and run cargo spellcheck to remove common spelling mistakes.
  • Developer certificate of origin is preferred.
Issues

Collection of the latest Issues

Roenbaeck

Roenbaeck

Comment Icon1

It looks like the following feature prevents the crate from compiling on stable Rust.

Is it possible to work around this feature in order to get the crate to compile on stable? If not, is there an expected time frame for when this feature might make it into stable? I would really like to use the crate, but the dependence on nightly could become an issue.

prataprc

prataprc

Comment Icon2

Values in jsondata::Json should not change in its hash value for any given hashing algorithm. Also the serialized form of jsondata::Json should not affect its hash value for any give hashing algorithm.

List of common hashing algorithm:

fasthash::city fasthash::murmur3 SHA1

prataprc

prataprc

Comment Icon0

In case you need jsondata to be part of a proprietary distribution or to be part of a different license agreement, please post your request here or contact the author directly.

prataprc

prataprc

Comment Icon0

Total ordering is complex for floating point numbers. Adding to that in JSON, due to mixed type of integer and floating-point, total ordering need to be worked out across both integers and floats.

A detailed description for total ordering is covered here Also there is a discussion thread in users.rust-lang.org.

prataprc

prataprc

Comment Icon0

Unicode specification for sorting string has this to say:

Because collation varies by language and not just by script, it is not possible to arrange the encoding for characters so that simple binary string comparison produces the desired collation order for all languages. Because multi-level sorting is a requirement, it is not even possible to arrange the encoding for characters so that simple binary string comparison produces the desired collation order for any particular language.

Note that the current implementation of Ord on JSON string simply does a byte-by-byte comparison.

prataprc

prataprc

good first issue
Comment Icon0

Benchmark JSON operations like:

  • Arithmetic operations, ADD, SUB, MUL, DIV, REM, NEG.
  • Bitwise operations, SHL, SHR, BITAND, BITOR, BITXOR.
  • Logical operations, NOT, AND, OR.
  • Compare operations EQ, NE, LT, LE, GT, GE.
  • Index operations.
  • Range operations.
prataprc

prataprc

Comment Icon1

JSON5 extends JSON specification.

  • Object keys may be an ECMAScript 5.1 IdentifierName.
  • Objects may have a single trailing comma.
  • Arrays may have a single trailing comma.
  • Strings may be single quoted.
  • Strings may span multiple lines by escaping new line characters.
  • Strings may include character escapes.
  • Numbers may be hexadecimal.
  • Numbers may have a leading or trailing decimal point.
  • Numbers may be IEEE 754 positive infinity, negative infinity, and NaN.
  • Numbers may begin with an explicit plus sign.
  • Single and multi-line comments are allowed.
  • Additional white space characters are allowed.

Refer: Full JSON5 specification

prataprc

prataprc

Comment Icon0

JSON Strings can be unicode encoded and parsing them can be costly: For example

By not supporting unicode encoded string, we can improve the performance of JSON deserialisation.

Right now parse_string() handles string parsing for JSON. Implement parse_ascii_string() as alternate feature.

prataprc

prataprc

good first issue
Comment Icon0

Would be nice to add the following badges.

  • Build passing, Travis continuous integration.
  • Code coverage, codecov and coveralls.
  • Crates badge
  • Downloads badge
  • License badge
  • Rust version badge.
  • Maintenance-related badges based on isitmaintained.com
  • Documentation
  • Gitpitch

Information - Updated Apr 09, 2022

Stars: 6
Forks: 1
Issues: 15

Repositories & Extras

Serde is a framework for serializing and deserializing Rust data structures efficiently and generically

Rust Greatest JSON weapon is Serde with over 4.4K stars on github and a massive developer community. This is considered a core Rust library for every developer to learn in BRC's opinion

Serde is a framework for serializing and deserializing Rust data structures efficiently and generically

Rust 버전 JsonPath 구현으로 Webassembly와 Javascript에서도 유사한 API 인터페이스를 제공 한다

JsonPath 구현으로 Webassembly와 Javascript에서도 유사한 API 인터페이스를 제공 한다

Rust 버전 JsonPath 구현으로 Webassembly와 Javascript에서도 유사한 API 인터페이스를 제공 한다

SIMD JSON for Rust  

Rust port of extremely fast serde compatibility

SIMD JSON for Rust  

JSON-E Rust data-struct paramter crate for lightweight embedded content with objects and much more

What makes JSON-e unique is that it extensive documentation and ease of use

JSON-E Rust data-struct paramter crate for lightweight embedded content with objects and much more
JSON

111

A Rust JSON5 serializer and deserializer which speaks Serde

Deserialize a JSON5 string with from_str

A Rust JSON5 serializer and deserializer which speaks Serde

Rust JSON Parser Benchmark

Download and Generate JSON Data

Rust JSON Parser Benchmark

Read JSON values quickly - Rust JSON Parser

AJSON get json value with specified path, such as project

Read JSON values quickly - Rust JSON Parser

Rust actix json request example

Send a json request to actix, and parse it

Rust actix json request example
JSON

140

json_typegen - Rust types from JSON samples

json_typegen is a collection of tools for generating types from

json_typegen - Rust types from JSON samples

Rust JSON parsing benchmarks

This project aims to provide benchmarks to show how various JSON-parsing libraries in the Rust programming language perform at various JSON-parsing tasks

Rust JSON parsing benchmarks

A tiny command line tool written in rust to print json data as a formatted...

A tiny command line tool written in rust to print json data as a formatted table

A tiny command line tool written in rust to print json data as a formatted...

A Rust implementation of the JSON-RPC 2

A Rust implementation of the serde) for JSON-RPC

A Rust implementation of the JSON-RPC 2
Facebook Instagram Twitter GitHub Dribbble
Privacy