ryogrid/funnelkvs

FunnelKVS: Rust implementation of autonomous distributed key-value store which has REST interfaces

TODO list (Japanese, includes finished tasks)

    • https://gist.github.com/ryogrid/9b1f340babfbb6cdc9d03e57b7834e71

What is Funnel?

  • It is name of wepons which appears in MOBILE SUIT GUNDUM (animation) series
  • Several type of funnel attacks enemy cooporate with multi units
  • https://www.google.com/search?q=fin+funnel&tbm=isch

Distributed KVS

  • Architecture (Japanese)

    • https://github.com/ryogrid/rust_dkvs/blob/master/ARCHITECTURE.md
  • Referenced site (about REST API implementation. first one is Japanese)

    • https://qiita.com/yukinarit/items/c5128e67d168b4f39983
    • https://rocket.rs/v0.4/guide/getting-started/
  • Fault torelance

    • data replication is implemented
    • functionality for keeping network healthy at occuring node down is also implemented
  • Data consistency

    • care for data consistency is not enough yet...
    • First, running test on occring node downs and joins condition is planned
  • How to execute node daemon

    • $ curl https://sh.rustup.rs -sSf | /bin/bash -s -- -y --default-toolchain nightly
    • $ rustup override set nightly
    • $ rustup update && cargo update
    • $ cargo build --release
    • $ target/release/rust_dkvs [born_id: 1...N] [IP addr to bind] [Port number to bind] [IP addr of medietor] [Port number of medietor] [log output path: currently not referenced]
  • Setup KVS system

    • launch example of node daemons which compose KVS system
      • you can build KVS system which is composed of multiple machines if you place rust_dkvs program binary to the machines and kick these with some tool like SSH
        • procedures wrote below are single machine example for easy trying
      • born_id of first node must be 1 but born_id of other node has no restriction except thst 1 can't be used
        • 1 can be used by first node only
      • IP address and port number has no restriction but all nodes should be able to communicate directly with each other
    • $ target/release/rust_dkvs 1 127.0.0.1 11000 127.0.0.1 10999 ./
    • $ target/release/rust_dkvs 2 127.0.0.1 11001 127.0.0.1 11000 ./
    • $ target/release/rust_dkvs 3 127.0.0.1 11002 127.0.0.1 11000 ./
    • ....
  • REST interfaces which are offered by KVS system

    • for easy testing with Web browser or something (use http GET request)
      • http://[node addr]:[node_port]/global_put_simple?key=[string only includes ascii character]&val=[string only includes ascii character]
      • http://[node addr]:[node_port]/global_get_simple?key=[string only includes ascii character]
      • http://[node addr]:[node_port]/global_delete_simple?key=[string only includes ascii character]
    • for using from code or HTTP client tool (use http POST request to send JSON text)
      • charactor code of body part must be UTF-8
      • http://[node addr]:[node_port]/global_put
          - body at POST -> { "key_str" : "[charactors]", "val_str" : "[charactors]" }
      • http://[node addr]:[node_port]/global_get
          - body at POST -> "[key charactors]"
      • http://[node addr]:[node_port]/global_delete
        • body at POST -> "[key charactors]"
  • Utility CLI tool (tools/dkvs_client.go)

    • setup chord network (on local machine)
      • $ go run -op=setup-nodes -arg1=[launch nodes num]
    • KVS system network health check
      • $ go run -op=check-chain
      • if the network is helthy, launched or alive nodes (process) are listed without duplication
    • testing put values to KVS via specified node
      • $ go run -op=put-test-values -arg1="127.0.0.1:11000"
      • put 100 values
      • node you specify with arg1 has no restiction like that local addresses can be only specified. above is example
    • testing get already put values from KVS via specified node
      • $ go run -op=get-test-values -arg1="127.0.0.1:11005"
      • get 100 values

Simulator of distributed KVS (chord_sim dir)

  • design verification with simulator wrote by Python (verification is finished)
    • you can simulate distributed kvs working behavior. on the simulation put, get, stabilize, join operations are issued continuously on node downs and node joins occuring condition.

Simulator of distributed KVS (chord_sim_rust dir)

  • design verification with simulator wrote by Rust (verification is finished)
  • Rust implemantation of chord simulator can be executed like below
    • $ curl https://sh.rustup.rs -sSf | /bin/bash -s -- -y --default-toolchain nightly
    • $ rustup override set nightly
    • $ rustup update && cargo update
    • $ cd chord_sim_rust
    • $ cargo run

Runnable platforms for KVS system (= you can build the daemon binary for the platform)

  • Windows native
    • dev env is also OK
  • MacOS
    • dev env is also OK
  • Linux (and Windows Subsystem for Linux environment)
    • dev env is also OK
    • probably ...
  • other UNIX like OS environments
    • please try!

Information - Updated Jul 30, 2022

Stars: 14
Forks: 1
Issues: 1

Repositories & Extras

This is an experimental serializer for Rust's serde ecosystem, which can convert Rust objects to...

This is an experimental serializer for Rust's serde ecosystem, which can convert Rust objects to Python values and back

This is an experimental serializer for Rust's serde ecosystem, which can convert Rust objects to...

Robust and Fast tokenizations alignment library for Rust and Python

Blog post: maturin to build the wheel

Robust and Fast tokenizations alignment library for Rust and Python

Rust Python Integration Made Easy

PIME is a Rust crate, which allows PyO3, neotasker Python module

Rust Python Integration Made Easy

A Rust clone of Python's click, built on the structopt crate

A Rust clone of Python's structopt crate

A Rust clone of Python's click, built on the structopt crate

A simple safe Rust library to cache functions (or sections of code) in memory, mainly...

A simple safe Rust library to cache functions (or sections of code) in memory, mainly ported from my Python module Rust

A simple safe Rust library to cache functions (or sections of code) in memory, mainly...

Orbit System is a Command & Control system for Windows written both in the Rust...

Orbit System is a Command & Control system for Windows written both in the Rust and Python programming languages

Orbit System is a Command & Control system for Windows written both in the Rust...

Quick demo of getting python and rust to communicate with graphql

The main point of this is to show how we can validate a graphql query

Quick demo of getting python and rust to communicate with graphql

A simple programming language inspired by Rust and Python

The most important type in Kuru is a Set

A simple programming language inspired by Rust and Python

Python/Rust module for mol2 format (de)serialization

Default level of compression here is 3

Python/Rust module for mol2 format (de)serialization

PYO3 Python Classes in Rust

Calling Rust from Python using PyO3

PYO3 Python Classes in Rust
Facebook Instagram Twitter GitHub Dribbble
Privacy