A pure Rust implementation of the

reference C++ implementation

snap

A pure Rust implementation of the Snappy compression algorithm. Includes streaming compression and decompression using the Snappy frame format. This implementation is ported from both the reference C++ implementation and the Go implementation.

Licensed under the BSD 3-Clause.

Documentation

https://docs.rs/snap

Usage

Add this to your Cargo.toml:

Example: compress data on stdin

This program reads data from stdin, compresses it and emits it to stdout. This example can be found in examples/compress.rs:

Example: decompress data on stdin

This program reads data from stdin, decompresses it and emits it to stdout. This example can be found in examples/decompress.rs:

Example: the szip tool

szip is a tool with similar behavior as gzip, except it uses Snappy compression. It can be installed with Cargo:

To compress a file, run szip file. To decompress a file, run szip -d file.sz. See szip --help for more details.

Testing

This crate is tested against the reference C++ implementation of Snappy. Currently, compression is byte-for-byte equivalent with the C++ implementation. This seems like a reasonable starting point, although it is not necessarily a goal to always maintain byte-for-byte equivalence.

Tests against the reference C++ implementation can be run with cargo test --features cpp. Note that you will need to have the C++ Snappy library in your LD_LIBRARY_PATH (or equivalent).

To run tests, you'll need to explicitly run the test crate:

To test that this library matches the output of the reference C++ library, use:

Tests are in a separate crate because of the dependency on the C++ reference library. Namely, Cargo does not yet permit optional dev dependencies.

Minimum Rust version policy

This crate's minimum supported rustc version is 1.39.0.

The current policy is that the minimum Rust version required to use this crate can be increased in minor version updates. For example, if crate 1.0 requires Rust 1.20.0, then crate 1.0.z for all values of z will also require Rust 1.20.0 or newer. However, crate 1.y for y > 0 may require a newer minimum version of Rust.

In general, this crate will be conservative with respect to the minimum supported version of Rust.

Performance

The performance of this implementation should roughly match the performance of the C++ implementation on x86_64. Below are the results of the microbenchmarks (as defined in the C++ library):

Notes: These benchmarks were run with Snappy/C++ 1.1.8. Both the C++ and Rust benchmarks were run with the same benchmark harness. Benchmarks were run on an Intel i7-6900K.

Additionally, here are the benchmarks run on the same machine from the Go implementation of Snappy (which has a hand rolled implementation in Assembly). Note that these were run using Go's microbenchmark tool, so the numbers may not be directly comparable, but they should serve as a useful signpost:

To run benchmarks, including the reference C++ implementation, do the following:

To compare them, as shown above, install critcmp and run (assuming you saved the baseline above under the name snappy):

Finally, the Go benchmarks were run with the following command on commit ff6b7dc8:

Comparison with other Snappy crates

  • snappy - These are bindings to the C++ library. No support for the Snappy frame format.
  • snappy_framed - Implements the Snappy frame format on top of the snappy crate.
  • rsnappy - Written in pure Rust, but lacks documentation and the Snappy frame format. Performance is unclear and tests appear incomplete.
  • snzip - Was created and immediately yanked from crates.io.
Issues

Collection of the latest Issues

Icelk

Icelk

2

The encoder structs create a new raw::Encoder each time you instantiate them. As stated by the docs, it's recommended to save the raw::Encoder. To prevent creating new raw::Encoders for each new encoding, could this crate provide a way to reset the higher level Read and Write encoders? Thanks!

Icelk

Icelk

enhancement
4

Hi! What is the status of write::FrameDecoder? I would be very grateful if you implemented it, as it could reduce a large allocation in my code. Thanks for the great pure-rust compression algorithm!

BurntSushi

BurntSushi

1

CI build failure: https://github.com/BurntSushi/rust-snappy/runs/2751122501?check_suite_focus=true

The failing tests are tests that check that this library generates the same compressed output as the reference Snappy C++ implementation. So a failing test here doesn't necessarily indicate a correctness issue, but that perhaps the reference implementation changed or got some improvements that we should investigate.

My next steps would probably be:

  • Compare the versions on macOS and Linux CI. Are they the same? If so, try to understand why the failures might be happening on macOS and not Linux. If they aren't the same, try reproducing the problem on Linux with the same Snappy version used on macOS.
  • Review the commit log for the reference Snappy C++ implementation and see if there are any relevant recentish changes.

For now, I've disabled running the cpp roundtrip tests on macOS in CI.

Versions

Find the latest versions by id

Information - Updated Jun 22, 2022

Stars: 345
Forks: 33
Issues: 5

Rust library for Self Organising Maps (SOM)

Add rusticsom as a dependency in Cargo

Rust library for Self Organising Maps (SOM)

Rust library for parsing configuration files

The 'option' can be any string with no whitespace

Rust library for parsing configuration files

Rust library for the Pimoroni Four Letter pHAT

This library aims to port ht16k33 (or rather a fork, as of right now) so credit goes to ht16k33-diet

Rust library for the Pimoroni Four Letter pHAT

Rust library for emulating 32-bit RISC-V

This library can execute instructions against any memory and register file that implements

Rust library for emulating 32-bit RISC-V

Rust library for connecting to the IPFS HTTP API using Hyper/Actix

You can use actix-web as a backend instead of hyper

Rust library for connecting to the IPFS HTTP API using Hyper/Actix

Rust library to manipulate file system access control lists (ACL) on macOS, Linux, and FreeBSD

This module provides two high level functions, getfacl and setfacl

Rust library to manipulate file system access control lists (ACL) on macOS, Linux, and FreeBSD

Rust library translation (rust-src/rust-std/stdlib/rustlib translation)

This is the place to translate Having a documentation in your native language is essential if you don't speak English, and still enjoyable even if...

Rust library translation (rust-src/rust-std/stdlib/rustlib translation)

Rust library for using Infrared hardware decoders (For example a Vishay TSOP* decoder),

enabling remote control support for embedded project

Rust library for using Infrared hardware decoders (For example a Vishay TSOP* decoder),

Rust library for interaction with the OriginTrail Decentralized Knowledge Graph

open up an issue on this repository and let us know

Rust library for interaction with the OriginTrail Decentralized Knowledge Graph

Rust library for parsing COLLADA files

Notice: This library is built around files exported from Blender 2

Rust library for parsing COLLADA files

Rust library for low-level abstraction of MIPS32 processors

This project is licensed under the terms of the MIT license

Rust library for low-level abstraction of MIPS32 processors
Facebook Instagram Twitter GitHub Dribbble
Privacy