elastic is an efficient, modular API client for Elasticsearch written in Rust

Elasticsearch written in simple examples

elastic

elastic is an efficient, modular API client for Elasticsearch written in Rust. The API is targeting the Elastic Stack 7.x.

elastic provides strongly-typed documents and weakly-typed queries.

Quick reference:

  • simple examples
  • example apps

Also check out the official elasticsearch crate!

Stability

This crate is still quite unstable and is likely to continue to churn breaking releases over the near future with not-so-detailed changelogs.

If you run into any problems upgrading in your own open source projects feel free to open up an issue and we'll give you a hand. The goal is definitely to offer a stable API eventually.

Build Status

Platform Channel Status (master)
Linux / macOS Stable/Nightly
Windows Nightly

Documentation

Version Docs
current (master)

Example

Add elastic to your Cargo.toml:

Create a SyncClient and start making requests:

elastic also offers an AsyncClient for use with the tokio asynchronous io stack. See the examples folder for complete samples.

Building documents

Document mapping is derived at compile-time from your Plain Old Rust Structures. Just add a #[derive(ElasticType)] attribute:

And you can start using MyDocument in Client request methods.

See the docs for more details.

Alternatives

elastic.co released an official client, elasticsearch. Although it is still in an alpha stage (as of 2020-02-10), it is very comprehensive and generates most of its code from the official REST API specifications.

Additionally, if you'd like to use a strongly-typed Query DSL builder see rs-es. This client does the hard work of providing an idiomatic Rust API for interacting with Elasticsearch. It has the advantage of letting you know your queries will parse at compile-time instead of runtime.

Goals

To provide a full-featured and efficient Elasticsearch client for Rust over asynchronous io. Rust gives us a lot of tools for building super-performant but highly accessible libraries, which we aim to continue. elastic is aimed at people who need to work with Elasticsearch and are considering using Rust, as well as users that are already using Rust. We want to offer a solution to interacting with Elasticsearch that's compelling from both within and outside the Rust ecosystem.

The REST API is covered by a simple inline JSON macro like serde_json's json! so it's always possible to build any query. This means you don't need to learn another API for interacting with Elasticsearch; queries mocked in Dev Tools could just be copy+pasted into your Rust source.

The core focus of this project is on strong typing over your document types and query responses in Elasticsearch, rather than trying to map the entire Query DSL.

Support for Elastic's plugin products, like watcher and graph could be added as feature-gated modules as necessary.

License

Licensed under either of these:

Issues

Collection of the latest Issues

lfdominguez

lfdominguez

2

The default mapping type for serde_json::Value is "nested" and that not allow make normal queries on ElasticSearch for dynamics data. How i can change the type to "object". Already i changed the code on file "src/elastic/src/types/document/mapping.rs" on trait ObjectMapping from NESTED_DATATYPE to OBJECT_DATATYPE and now my elastic queries are responding with results.

ryankurte

ryankurte

1

Hey, thanks for making this!

In the docs it's noted that you can specify index expressions, however, turning this on results in a huge pile of errors as afaik many of the library methods are implemented over StaticIndex, and it's not super clear how to fix this?

An example is put_mapping, which requires StaticIndex to create a PutMappingRequestBuilder, unfortunately the inner type is private so unless the type fulfills StaticIndex it is,, impossible (?), to set a mapping?

jawline

jawline

1

Whenever I try to make a bulk query with the following code my program crashes with a stack overflow exception:

Could you let me know what I am doing wrong?

TomPridham

TomPridham

8

is there a way to enable verbose errors? i am having trouble querying a remote elasticsearch cluster, but the errors i am getting don't reveal anything that i could look into as a potential problem.

mwilliammyers

mwilliammyers

0

I think we should expose the queries sub crate; many thanks to @stephanbuys for the awesome crate!

TODO:

  • Integrate it into the main elastic crate or keep it as a sub-crate?
  • Relation to rs-es?
  • Keep derive_builder dependency?
    • Update derive_builder
  • Remove error-chain dependency?
  • Improve docs
  • Make all enum fields UpperCamelCase
  • Add more fields to the Query struct
  • Add more aggregations
  • Flesh out Bool struct (improve bool filter support)
    • Add minimum_should_match, boosting etc.
    • Make it so bool filters can be nested
DevQps

DevQps

1

Description

Currently types such as u32 or u16 cannot be automatically mapped. It would be nice if these numbers could be converted to a larger signed value on the fly.

KodrAus

KodrAus

change:breaking
0

Using captured fluent builders means we need to allocate for each call to _fluent, and means we need to impose Send + Sync constraints on the builder methods themselves. There are other footguns around APIs that issue multiple requests, like a bulk stream that may go on indefinitely. Since we don't rebuild the parameters we'll always issue those requests to the same node.

So we should just remove the fluent builder infrastructure and apply fluent methods to values inline.

DevQps

DevQps

help wanted
0

Description

Elasticsearch also supports IPv6 addresses in its IP fields, but currently std::net::IPv6Addr is not yet mapped.

NavyaZaveri

NavyaZaveri

0

I'm trying to use the ElasticType macro against a struct parameterized on a lifetime, like so:

This throws an error:

As far as I'm aware, Rust doesn't have any semantics for parameterizing macros. If a fix isn't possible, It would be more useful to see an error message stating it's not possible to use ElasticType on a lifetimed struct.

wfxr

wfxr

3

Here is a very simple scroll request, but I can't make it work. Did I get something wrong?

The error message is :

If I keep request body empty:

Another error showed:

KodrAus

KodrAus

0
  • #316
  • #317
  • #277 (needs more integration testing)
  • #362
  • #381
  • #382
  • #402
  • Tidy up docs (they're probably a bit inaccurate by now)
  • Note change of supported Elasticsearch version in docs and readme
  • Check doc links
KodrAus

KodrAus

change:non-breaking
1

Related to #266

reqwest has a specific error variant for timeouts. We could surface methods on ClientError that can be used to determine whether or not the error is due to a timeout:

It's introducing a bit of a different paradigm for dealing with variants on the client error, but I think this is ok. It's convenient to be able to pull properties out of API erorrs, like the index name and use that property to create the index.

softprops

softprops

change:breaking
9

Very nice looking crate!

I'm evaluating this as an option to use aws's hosted elastic search. aws requires all requests to be signed which seems most doable with this crate if I implement a Sender that signs requests before sending them. However it seems impl Sender requires implementing a trait that's not visible outside of this crate, elastic::client::private::Sealed.

Do you have another recommendation on how I could accomplish request signing?

KodrAus

KodrAus

change:breaking
20

Since we don't map a lot of the Query DSL yet this should just be a matter of tweaking some bits in elastic_types and updating CI to target 6.0.

The biggest change to think about is the removing support for multiple types per index. An implication is that we could remove indices as required parameters and rely on the type name as a default. It raises a question of whether all operations, like index_create, should require a generic document type.

KodrAus

KodrAus

question
1

For supporting a strongly-typed Query DSL, we should generate some runtime-queryable metadata for each field when deriving ElasticType.

That way we can capture properties of each field that might affect generated queries as types. Things like date formats for example.

I don't have a clear picture of what this should look like yet, I think we'll end up with some strongly-typed structure implementing traits for each field, one per type, but it needs some proper design.

KodrAus

KodrAus

design
1

Motivation

Right now, calls to client.search, or client.get_document etc will return a response builder with a few generic parameters like:

This isn't very nice to include in a function signature.

Single traits

Design

Offer two traits SyncRequest and AsyncRequest that are implemented for many request builders and can be used to erase 'implementation detail' generics from request builders when they're returned from methods.

Multiple traits

Design

One idea is to use traits and impl Trait (or Box) to erase the type:

Using traits has a benefit over type definitions in that we can effectively erase generic parameters through a blanket implementation (TBody doesn't appear in the SearchRequest trait).

Open questions

  • Is this intuitive enough? We'd need to add an example for returning request builders from methods
  • What about raw requests?
  • What if the trait name conflicts with an endpoint type?
  • Could we leverage traits better for these things?
  • Is this too much extra boilerplate for request methods?
  • Should these go in the prelude or not?
KodrAus

KodrAus

change:breaking
1

Looks like there's a new typed_keys parameter we could utilise to parse aggregations exactly.

Since it depends on the parameter being present in the request we would probably have to have an alternative Aggregations parser that expects it to be present.

API Gist

See here

KodrAus

KodrAus

change:breaking
0

Even though none of the types in elastic_responses borrow from the data (because you might want to call from_reader) we shouldn't limit IsOk to just owned responses because callers could write their own.

KodrAus

KodrAus

help wanted
2

We need to figure out how to make some per-request state available throughout the various repos for logging (like a correlation id) that:

  • Is available and correct when in the context of an async Task, where many Tasks are executed on a given thread
  • Is available and correct when outside an async Task for operations sent across threads (like start a request on one thread that's actually sent on a different one)

Ideally this shouldn't be leaky in the sense of needing to pass some context to most methods explicitly and it should be supported everywhere, or else it's basically useless.

I think this will be a bit tricky, and will need some thought. It might also be infeasible for both sync and async.

KodrAus

KodrAus

0

Look into integrating something like bors-ng into this project so we've got a less ad-hoc approach to getting code built, tested and merged.

I'm growing weary of automatically triggering builds on PRs, which can probably be solved just through Travis/AppVeyor, but I've been meaning to look into this anyways, and with all the code under the one umbrella it's a good opportunity to get it sorted. Anything that can make builds faster, more reliable and more secure is worth investigating.

KodrAus

KodrAus

change:non-breaking
4

Right now we support Elasticsearch document mapping through traits that are implemented for:

  • Some std/common type, like bool, DateTime<Utc>
  • A wrapper type that accepts a mapping, like Boolean<M>, Date<M>

In order to use different mapping for a type you need to use one of these wrapper fields, which depends on some dubious use of Deref to minimise impact throughout your code. The justification is that you should probably have separate types to represent documents than other parts of your code (rather than, for example, trying to re-use the same type for diesel, elastic, and other parts of your code).

This does have a few problems:

  • We've been lucky that the Elasticsearch name for types hasn't clashed with anything else yet. But for something like Uuid this doesn't work anymore
  • You do need to change the shape of your structures in order to map differently. There's a big gap between the default mapping, and any custom mapping

So I think it's worth looking in to use attributes on the derive to pick up custom mapping. So instead of this:

You could have:

I'll need to work out how this would expand though, how it would play with alternative field implementations, and how it should work for fields that aren't mappable types.

Versions

Find the latest versions by id

v0.20.10 - May 07, 2018

  • Fixes up a bug attempting to use HashMap as a field in a document
  • Fixes up some dead links in the readme

v0.20.8 - Dec 17, 2017

This is another house-keeping release without any code changes.

  • Fixes some documentation errors around the index_exists method
  • Updates the licence field in Cargo.toml files to reflect the dual Apache/MIT license

v0.20.7 - Dec 17, 2017

This release adds the index_exists method to the client.

elastic

  • Adds an index_exists method to the sync and async clients

elastic_responses

  • Add an IndexExistsResponse type. Elasticsearch doesn't actually return any data with an index exists response (it just uses the HTTP status code), so this type doesn't actually represent what Elasticsearch's API returns
  • Adds support for returning arbitrary json data from the IsOk trait. This makes it possible to return data to deserialise as an IndexExistsResponse depending on the status code

v0.20.6 - Nov 30, 2017

  • Change the version bound for quick-error from ~1.1 to ~1
  • Change the version bound for bytes from =0.4.5 to ~0.4.5

v0.20.5 - Nov 25, 2017

  • Fixes a compile error in the queries crate. There aren't any other code changes in this release.

v0.20.3 - Oct 27, 2017

This release adds some documentation to the generated Elasticsearch endpoint types.

elastic_requests

Generate documentation along with the endpoint types. The docs include:

  • The HTTP verb and default path for the endpoint
  • The path for each variant for the endpoint

This should make it easier to work out what type corresponds to what request endpoint in Elasticsearch.

v0.20.2 - Oct 20, 2017

This release includes integration tests, doc fixes and some more endpoints on the client.

elastic

Added the following new endpoints to the Client:

  • document_update
  • document_delete
  • index_close
  • index_open
  • index_delete
  • ping

Fixed some errors in documentation.

elastic_responses

Added the following new types:

  • UpdateResponse
  • DeleteResponse

v0.20.1 - Oct 16, 2017

There aren't any observable source changes in this release, just some house-keeping for builds and metadata.

  • Add elastic_queries into the tree. There isn't a release on crates.io for this yet
  • Add readme keys to Cargo.toml files

v0.20.0 - Oct 05, 2017

A note on this release

There have been a lot of changes across many repositories. This makes it difficult to provide a comprehensive list of breaking changes (there are a number of them, but upgrading should be fairly mechanical). For future releases we'll start maintaining a CHANGELOG to collect changes as they occur.

The following is a general list of significant changes.

Async

This release adds async support to elastic using the tokio and futures ecosystem.

The implementation uses generics to parameterise request builders. This means we share a lot of code between the sync and async APIs, but can make the docs noisier and harder to follow.

Combining repositories

Each of the crates in the elastic organisation have been pulled in to this single repository. This will make it easier to track changes in the future, as well as share CI infrastructure. Overall it should lead to a more solid and contributor-friendly project.

elastic now contains the following crates with synchronized versions:

  • elastic_requests
  • elastic_responses
  • elastic_types_derive_internals
  • elastic_types_derive
  • elastic_types
  • elastic_derive
  • elastic

Some specifics

elastic

  • Client has been renamed to SyncClient and ClientBuilder has been renamed to SyncClientBuilder
  • Client::new has been removed in favour of SyncClientBuilder

elastic_types

This release refactors Date to start reducing dependence on chrono::DateTime, and to make sure date formats aren't implicitly changed. Some of these changes might turn out to be too restrictive and may be reverted in the future (allowing formats to change without explicit conversions).

  • Change remap methods to be static methods instead of instance methods
  • Add DateExpr for supporting date math expressions

Changes to Date

  • Change Date<F, M = DefaultDateMapping<F>> to Date<M>. So you can't just write Date<EpochMillis> anymore, it needs to be Date<DefaultDateMapping<EpochMillis>>. This simplifies the generics, and makes Date easier to work with. To get the ergonomics of Date<EpochMillis> back, you can use type aliases:

  • Adds a DateValue and FormattableDateValue type

  • Use DateValue in the DateFormat::parse and DateFormat::format methods instead of chrono::DateTime

  • Remove the conversion from chrono::DateTime into Date<M> unless M::Format = ChronoFormat. This is because chrono::DateTime is already mappable with the ChronoFormat, so to make sure that formats aren't implicitly changed, you need to convert a chrono::DateTime into a DateValue first, which doesn't have any format:

Before:

After:

Changes to GeoPoint

  • Like Date, GeoPoint<F, M = DefaultGeoPointMapping<F>> has been changed to GeoPoint<M>. Use the same type aliases approach for ergonomics

elastic_responses

  • Make all fields on response types private
  • Add iterators to response types. For BulkResponse and BulkErrorsResponse, call iter or into_iter. For SearchResponse call hits, into_hits, documents or into_documents.
  • Remove the SearchResponseOf and GetResponseOf types. Now SearchResponse and GetResponse require a generic type for the kind of document they contain. serde_json::Value is re-exported for convenience.

v0.13.3 - Aug 01, 2017

Fixes a build failure on nightly due to unused doc comments in error_chain.

v0.13.2 - Jul 30, 2017

Adds basic logging support for requests and responses. This will log:

  • Request path
  • Response status if request succeeded
  • Response error if request failed (io error, rather than Elasticsearch error)

v0.13.0 - Jun 22, 2017

  • Updates elastic_types to 0.18.0. See elastic_types release notes
  • Refactoring the RequestBuilder generics so they're more consistent for raw requests and high-level builders. This shouldn't really affect you unless you're taking RequestBuilders as arguments or returning them from methods

v0.12.3 - Jun 11, 2017

  • Re-export ParseResponseError from client::responses::parse

v0.12.2 - May 31, 2017

  • Deprecates ResponseBuilder.raw and adds a new ResponseBuilder.into_raw method that does the same thing. The raw method will be removed in the next breaking release

v0.12.1 - May 31, 2017

  • Adds build badges to Cargo.toml
  • Fixes some typos in the documentation

v0.12.0 - May 31, 2017

Initial release to track cargo releases.

Information - Updated Jun 23, 2022

Stars: 252
Forks: 39
Issues: 35

Rust bindings for libinjection

Add libinjection to dependencies of Cargo

Rust bindings for libinjection

Tantivy is a full text search engine library written in Rust

It is closer to Elasticsearch or benchmark break downs

Tantivy is a full text search engine library written in Rust

Rust bindings for the C++ api of PyTorch

LIghtweight wrapper for pytorch eg libtorch in rust

Rust bindings for the C++ api of PyTorch

Rust leveldb bindings

Almost-complete bindings for leveldb for Rust

Rust leveldb bindings

rust-analyzer is a modular compiler frontend for the Rust language

It also contains some tips &amp; tricks to help you be more productive when using rust-analyzer

rust-analyzer is a modular compiler frontend for the Rust language

Rust-Lightning is a Bitcoin Lightning library written in Rust

lightning, does not handle networking, persistence, or any other I/O

Rust-Lightning is a Bitcoin Lightning library written in Rust

Rust FUSE - Filesystem in Userspace

Rust library crate for easy implementation of Crate documentation

Rust FUSE - Filesystem in Userspace

Rust crate to implement a counterpart to the PBRT book's (3rd edition) C++ code:

Some images of the test scenes are shown below, but you can find more

Rust crate to implement a counterpart to the PBRT book's (3rd edition) C++ code:

Rust Persian Calendar

1** provides functionality for conversion among Persian (Solar Hijri) and Gregorian calendars

Rust Persian Calendar

Rust DjangoHashers

A Rust port of the password primitives used in alternatice implementation: the package library that requires OpenSSL

Rust DjangoHashers

Tantivy is a full text search engine library written in Rust

It is closer to Elasticsearch or benchmark break downs

Tantivy is a full text search engine library written in Rust
Facebook Instagram Twitter GitHub Dribbble
Privacy