Speculare PGCDC

Capture Data Change for Speculare

allows you to listen to changes in your PostgreSQL database via logical replication and then broadcast those changes over websockets.

Realtime server works by:

  1. listening to PostgreSQL's logical replication (here using Wal2Json).
  2. filtering the incoming message
  3. broadcasting the message over websocket

Explaination

You probably know that Postgresql is not a realtime databse. So if we want to stream the change of it to a websocket or any UI it's not possible by default.

Hopefully Postgresql have that sweet feature named Logical Replication, which stream the change made into the database over a replication slot. We can use a multitude of plugins to format the output of this stream, but for Speculare-PGCDC we've chosen to use wal2json.

This project create a replication slot on the targeted postgres instance and then stream the change from this slot to all the websockets connected.

Project Status

This project is still under development and the documentation is constantly changing. You are welcome to try it, but expect some changes with future updates. TAGs are used to mark a certain version in all Speculare repositories as compatible with each others.

Server setup / Dev setup

  • Install all deps
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ sudo apt-get install libssl-dev libpq-dev pkg-config build-essential

There pretty much only one step in this setup (other than Docker (see below) and is optional)

  • Create a Pgcdc.toml file based on Example.toml

⚠ WARNING: Check the docs !

Usage

$ wss://server/ws?query=change_type:table:col.eq.val

will get change_type event from table where col is equals to val.

The change_type and table parameters are mandatory, if you're missing them you'll get a 400 error. change_type can be any of those: *, insert, update, delete. table must be a valid table of your database.

I decided to restrict the API in such way that a single websocket can only listen to one table. This might change in the future if needed, but as of now and in the current shape of Speculare, it's not needed.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Issues

Collection of the latest Issues

Versions

Find the latest versions by id

v0.0.5 - Jul 28, 2021

Switched to a custom impl of the CDC stream & use warp instead of Actix.

  • Overall = performance gain and stability.

v0.0.4 - Jul 24, 2021

v0.0.3 - Jun 22, 2021

  • Switch to TimescaleDB

v0.0.2 - Jun 21, 2021

  • Fix tables not found after some days due to PARTMAN

v0.0.1 - Jun 14, 2021

First point in history where I consider the source code as alpha-ish.

Information - Updated May 13, 2022

Stars: 7
Forks: 0
Issues: 2
Actix

14.1K

Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust

Rust's premier framework for working with HTTP/1.x HTTP/2 requests. Includes core features like websockets, keep-alives and slow requests handling

Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust
CLI

3.9K

Netcat, curl and socat for WebSockets

Netcat, curl and socat for pre-built executables

Netcat, curl and socat for WebSockets

The WebSocket Transfer Agent

wsta is a cli tool written in rust for interfacing with WebSockets

The WebSocket Transfer Agent

Rust-WebSocket is a WebSocket (RFC6455) library written in Rust

Rust-WebSocket is a WebSocket (crates

Rust-WebSocket is a WebSocket (RFC6455) library written in Rust

Rust Websocket Physics Playground

A multiplayer physics playground, written in rust, using tokio-tungstenite for websockets and rapier-rs for physics

Rust Websocket Physics Playground

Memory based database with integrated Web Socket Server

Takes around 30µs (microseconds) to locate a specific account from amongst 50 million other accounts

Memory based database with integrated Web Socket Server
Cargo

1.0K

Rust-WebSocket

Note: Maintainership of this project is slugglish

Rust-WebSocket
Facebook Instagram Twitter GitHub Dribbble
Privacy