bwt-dev/bwt

Bitcoin Wallet Tracker

⛓️ on-chain or ⚡ lightning via BTCPay

Bitcoin Wallet Tracker

bwt is a lightweight wallet descriptor/xpub tracker and query engine for Bitcoin, implemented in Rust.

🔸 Personal wallet indexer (EPS-like)
🔸 Descriptor-based tracking
🔸 Electrum RPC server (also available as a plugin!)
🔸 Developer-friendly, modern HTTP REST API
🔸 Real-time updates with Server-Sent-Events or Web Hooks

Support development: ⛓️ on-chain or ⚡ lightning via BTCPay

  • Intro
  • Setting up bwt
    • Installation
    • Electrum-only server
    • Pruning
    • Real-time indexing
    • Advanced options
  • Electrum plugin 💥
  • Manual Electrum setup
  • HTTP API
    • Wallets
    • Transactions
    • Addresses
    • Outputs
    • Blocks
    • Mempool & Fees
    • Server-Sent Events 🌟
    • Miscellaneous
  • Web Hooks
  • Developing 👩‍💻
  • Reproducible builds
  • Thanks

Intro

bwt is a lightweight and performant descriptor-based wallet indexer backed by a bitcoin full node, using a model similar to that of Electrum Personal Server. It can serve as a personal alternative to public Electrum servers or power bitcoin apps such as wallet backends, payment processors and more.

It uses bitcoind to keep track of your wallet addresses (derived from your xpub(s)) and builds an index of their history that can be queried using the Electrum RPC protocol or using bwt's custom designed HTTP API.

Real-time updates are available through Server-Sent events (a streaming long-lived HTTP connection), or using Web Hooks push updates (an HTTP request sent to your URL with the event).

The index is currently managed in-memory and does not get persisted (this is expected to change), but building it is pretty fast: bwt can index thousands of transactions in under a second.

TL;DR: EPS + Rust + Modern HTTP API

Setting up bwt

Get yourself a synced Bitcoin Core node (v0.19 is recommended, v0.17 is sufficient. txindex is not required, pruning is supported) and install bwt using one of the methods below.

Installation

💥 Electrum desktop users can also install bwt as an Electrum plugin, which provides a GUI and doesn't require the standalone server installation described below.

Signed pre-built binaries

Available for download on the releases page (Linux, Mac, Windows and ARMv7/v8).

The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on the PGP WoT, github, twitter, keybase, hacker news and this video presentation (bottom of slide).

The signature verification should show Good signature from "Nadav Ivgi <[email protected]>" ... Primary key fingerprint: FCF1 9B67 ... and bwt-0.2.4-x86_64-linux.tar.gz: OK.

The builds are reproducible and can be verified against Travis CI. See more details here.

From source

Install Rust and:

Or using the crates.io package:

(Make sure ~/.cargo/bin is in your PATH)

With Docker

Assuming your bitcoin datadir is at ~/.bitcoin,

(Mounting the bitcoin datadir is not necessary if you're not using the cookie file.)

Running bwt

bwt --xpub <xpub> should be sufficient to get you rolling.

You can configure the --network (defaults to bitcoin), your --bitcoind-url (defaults to http://127.0.0.1:<default-rpc-port>), --bitcoind-dir (defaults to ~/.bitcoin) and --bitcoind-auth <user:pass> (defaults to using the cookie file from bitcoind-dir).

You can set multiple --xpubs to track. This also supports ypubs and zpubs. To set an explicit script type, use --xpub <xpub>:<type>, where type is one of wpkh, shwpkh or pkh.

You can also track output script descriptors using --descriptor. For example, --descriptor 'wpkh(<xpub>/0/*)'.

Standalone addresses can be tracked with --address <address> or --addresses-file <path>.

To speed up rescanning for historical transactions, you can provide the wallet creation date with --rescan-since <timestmap>. The timestamp can be a YYYY-MM-DD formatted string or 'now' to disable rescanning and watch for new transactions only (for newly created wallets). Setting this is highly recommended.

By default, the Electrum server will be bound on port 50001/60001/60401 (according to the network) and the HTTP server will be bound on port 3060. This can be controlled with --electrum-addr and --http-addr.

⚠️ Both the HTTP API server and the Electrum server are unauthenticated and unencrypted by default. If you're exposing them over the internet, they should be put behind a secure transport like an SSH tunnel, a VPN, or a Tor hidden service.

Authentication (but not encryption) can be enabled by following the instructions here.

You may set -v to increase verbosity or -vv to increase it more.

See --help for the full list of options.

Configuration file

Configuration options can be set under ~/bwt.env as environment variables in the dotenv format.

Options that accept multiple values (DESCRIPTORS, XPUBS, ADDRESSES and WEBHOOKS_URLs) can be provided as a ;-separated list, or using wildcard XPUB_*/DESC_*/ADDRESS_* variables.

For example:

Setting the environment variables directly is also supported.

Electrum-only server

If you're only interested in a standalone Electrum server, you may disable the HTTP API server by downloading the electrum_only pre-built binary, using the shesek/bwt:electrum docker image or building bwt with --no-default-features --features cli,electrum.

This removes several large dependencies and disables the track-spends database index (which is not needed for the electrum server).

(Also see the Electrum plugin.)

Pruning

You can use bwt with pruning, but a pruned node is only able to scan the recent blocks it still has available for transactions related to your wallet. This means that the --rescan-since date has to be within the range of non-pruned blocks, or set to 'now'. This makes pruned nodes primarily suitable for tracking newly created wallets.

There is, however, an opportunity to scan for your wallet's full history during the initial sync of your node, as the blocks will get scanned before getting pruned. You'll need to set --no-wait-sync to import the addresses without waiting for bitcoind to finish syncing first and make sure bwt runs before the earliest block containing a wallet transaction gets processed.

You can use --prune-until <target> to automatically prune the chain up to the given target (height, unix timestamp or YYYY-MM-DD formatted date). This requires configuring bitcoind with prune=1 to allow manual pruning via the RPC.

To connect Electrum, you will need to configure it with --skipmerklecheck to tolerate missing SPV proofs (they will be unavailable for transactions in pruned blocks).

If you're running Electrum with --skipmerklecheck, you may also configure bwt with --electrum-skip-merkle to save some resources by not generating SPV proofs even when it's possible. Both of these settings are automatically enabled when using the Electrum plugin.

Bitcoin Core multi-wallet

If you're using multi-wallet, you can specify which wallet to use with --bitcoind-wallet <name> (or -w <name>).

Using a separate wallet for bwt is recommended. You can set -w bwt --create-wallet-if-missing to have bwt create one for you.

Note that EPS and bwt should not be run on the same bitcoind wallet with the same xpub, they will conflict.

Advanced options

Real-time indexing

By default, bwt will query bitcoind for new blocks/transactions every 5 seconds. This can be adjusted with --poll-interval <seconds>.

To get real real-time updates, you may configure your bitcoind node to send a POST /sync request to the bwt http server whenever a new block or wallet transaction is found, using the walletnotify and blocknotify options.

Example bitcoind configuration:

After verifying this works, you may increase your --interval-poll to avoid unnecessary indexing and reduce cpu usage.

If you're using the electrum-only mode without the http server, you may instead configure bwt to bind on a unix socket using --unix-listener-path <path> and open a connection to it initiate an indexer sync.

For example, start with --unix-listener-path /home/satoshi/bwt-sync-socket and configure your bitcoind with:

If nc is not available, you can also use socat - UNIX-CONNECT:/home/satoshi/bwt-sync-socket.

If you're using docker, you can bind the socket on a directory mounted from the host to make it available outside the container. For example, --unix-listener-path /bitcoin/bwt-socket.

Gap limit

You may configure the gap limit with --gap--limit <N> (defaults to 20). The gap limit sets the maximum number of consecutive unused addresses to be imported before assuming there are no more used addresses to be discovered.

You can import larger batches with a higher gap during the initial sync using --initial-import-size <N> (defaults to 350). Higher value means less rescans. Should be increased for large wallets.

Scriptable transaction broadcast

You may set a custom command for broadcasting transactions via --tx-broadcast-cmd <cmd>. The string {tx_hex} will be replaced with the hex-encoded transaction.

The command will be used in place of broadcasting transactions using the full node, which may provide better privacy in some circumstances.

For example, to broadcast transactions over Tor using the blockstream.info onion service, you can use:

(Replace port 9050 with 9150 if you're using the Tor browser bundle.)

Electrum plugin

You can setup bwt as an Electrum plugin, which allows connecting Electrum to a Bitcoin Core full node backend by running an embedded Electrum server within the Electrum wallet itself.

See https://github.com/bwt-dev/bwt-electrum-plugin for more details and installation instructions.

Manual Electrum setup (without the plugin)

Setup the bwt server, then start Electrum with:

Alternatively, you can also set 127.0.0.1:50001:t as your server using the server selection GUI (note the :t, which disables TLS). Note, however, that it is not possible to configure oneserver and skipmerklecheck using the GUI.

Note that setting --skipmerklecheck is only necessary if your node is pruned, but it can also be used to save some resource when combined with --electrum-skip-merkle. See more details here.

HTTP API

All the endpoints return JSON. All bitcoin amounts are in satoshis.

Wallets

Each wallet represents an output script descriptor.

Note that xpubs specified via --xpub will be represented as two descriptor wallet entries, one for the external chain (used for receive addresses) and one for the internal chain (used for change addresses). You can associate the wallets to their parent xpub using the bip32_origins field.

Wallet format

  • desc - the output script descriptor tracked by this wallet
  • network - the network this wallet belongs to (bitcoin, testnet, signet or regtest)
  • is_wildcard - a boolean indicating whether the descriptor includes wildcard keys (xpub../*)
  • bip32_origins - bip32 origin information for the keys contained in the descriptor
  • gap_limit - the gap limited configured for this wallet
  • initial_import_size - the gap limit used during the initial import
  • rescan_policy - how far back rescanning should take place
  • max_funded_index - the maximum derivation index that is known to have history
  • max_imported_index - the maximum derivation index imported into bitcoind
  • satisfaction_weight - an upper bound on the weight of a satisfying witness to the transaction (also see here)

See GET /wallet/:checksum for an example.

GET /wallets

Get a map of all tracked descriptor wallets, as a json object indexed by the descriptor checksum.

Expand...

Example:

These wallets are the result of starting bwt with:

  • --descriptor 'wpkh(xpub...xToJdcy/0/*)' (the xjm8w0el wallet)
  • --descriptor 'wsh(multi(2,xpub...3CwvtjV/0/*,[16eabcf7/2]xpub...eHRwJzH/1/*))' (the k38panl4 wallet)
  • --xpub 'xpub...2E18PjR' (the cletf5fc and ftu25peq wallets)

GET /wallet/:checksum

Get information about the descriptor wallet identified by its checksum.

Expand...

Example:

GET /wallet/:checksum/:index

Get basic information for the wallet child address at derivation index index.

Expand...

Returned fields:

  • address
  • scripthash
  • origin - descriptor wallet origin information in <checksum>/<index> format
  • desc - the descriptor for this address
  • bip32_origins - an array of bip32 origins for the derived keys at this index

Examples:

GET /wallet/:checksum/next

Get the next unused address in the specified wallet.

Expand...

Issues a 307 redirection to the url of the next derivation index (/wallet/:checksum/:index) and responds with the derivation index in the responses body.

Note that the returned address is not marked as used until receiving funds; If you wish to skip it and generate a different address without receiving funds to it, you can specify an explicit derivation index instead.

Non-wildcard descriptors always return 0 as their next index.

Examples:

GET /wallet/:checksum/gap

Get the current maximum number of consecutive unused addresses in the specified wallet.

Expand...

Example:

Transactions

Wallet transaction format

This format is only available for wallet transactions and includes contextual wallet information about funded outputs and spent inputs. It does not include inputs/outputs that are unrelated the wallet.

Transaction fields:

  • txid
  • block_height - the confirming block height or null for unconfirmed transactions
  • funding - contains an entry for every output created by this transaction that is owned by the wallet
    • vout - the output index
    • amount - the output amount in satoshis
    • scriptPubKey fields
      • address - the address funded by this output
      • scripthash - the scripthash funded by this output
      • origin - descriptor wallet origin information in <checksum>/<index> format
      • desc - the descriptor for this script
      • bip32_origins - an array of bip32 origins for this script keys
    • spent_by - the transaction input spending this output in txid:vin format, or null for unspent outputs (only available with track-spends)
  • spending - contains an entry for every input spending a wallet output
    • vin - the input index
    • amount - the amount in satoshis of the previous output spent by this input
    • prevout - the <txid>:<vout> being spent
    • scriptPubKey fields
      • address, scripthash, origin, desc, bip32_origins
        (same format as above for funding)
  • balance_change - the net change to the wallet balance inflicted by this transaction

Additional fields for unconfirmed mempool transactions:

  • own_feerate - the fee rate paid directly by the transaction, in sat/vB
  • effective_feerate - the effective transaction fee rate, taking unconfirmed ancestor transactions into account
  • bip125_replacelable - whether this transaction can be replaced due to BIP 125 replace-by-fee (because either it or one of its unconfirmed ancestors are signaling to opt-into rbf)
  • has_unconfirmed_parents - whether this transaction has unconfirmed parents used as its inputs

The effective fee rate is calculated as MIN(own_fee/own_vsize, (own_fee+ancestor_fee)/(own_vsize+ancestor_vsize)).

The mempool fields may be temporarily unavailable if an error is encountered while syncing the mempool entries. This should be a rare event, for example if Bitcoin Core crashes, but API clients should prepare for the possibility.

GET /tx/:txid

Get the transaction in the wallet transaction format.

Expand...

Available for wallet transactions only.

Example:

GET /tx/:txid/verbose

Get the transaction in JSON as formatted by bitcoind's getrawtransaction with verbose=true.

Expand...

Available for all transactions that bitcoind is aware of (i.e. not pruned). Requires txindex to work for non-wallet transactions.

Example:

GET /tx/:txid/hex

Get the raw transaction formatted as a hex string.

Expand...

Example:

GET /tx/:txid/proof

Get the merkle inclusion proof for the transaction.

Expand...

Returned in bitcoind's merkleblock format.

Example:

GET /txs

GET /txs/since/:block-height

Get all wallet transactions confirmed at or after block-height, plus all unconfirmed transactions, for all tracked addresses.

GET /txs is an alias for GET /txs/since/0.

Expand...

Returned in the wallet transaction format. Sorted with oldest first.

Example:

GET /txs/since/:block-height/compact

Get a compact minimal representation of all wallet transactions since block-height.

Expand...

Returns a simple JSON array of [txid, block_height] tuples, where block_height is null for unconfirmed transactions. Sorted with oldest first.

Example:

POST /tx

Broadcast a raw transaction to the Bitcoin network.

Expand...

Returns the txid on success.

Body parameters:

  • tx_hex - the raw transaction encoded as a hex string

Example:

Addresses, Scripthashes & Keys

GET /address/:address

GET /scripthash/:scripthash

GET /wallet/:checksum/:index

Get basic information for the provided address, scripthash or descriptor index.

Expand...

Returned fields:

  • address
  • scripthash
  • origin - descriptor wallet origin information in <checksum>/<index> format
  • desc - the descriptor for this address
  • bip32_origins - an array of bip32 origins for the derived keys at this index

Example:

GET /address/:address/stats

GET /scripthash/:scripthash/stats

GET /wallet/:checksum/:index/stats

Get basic information and stats for the provided address, scripthash or descriptor index.

Expand...

Returned fields:

  • scriptPubKey fields
    • address
    • scripthash
    • origin
    • desc
    • bip32_origins
  • tx_count
  • confirmed_balanace
  • unconfirmed_balanace

Example:

GET /address/:address/utxos

GET /scripthash/:scripthash/utxos

GET /wallet/:checksum/:index/utxos

Get the list of unspent transaction outputs owned by the provided address, scripthash or descriptor index.

Expand...

Query string parameters:

  • min_conf - minimum number of confirmations, defaults to 0
  • include_unsafe - whether to include outputs that are not safe to spend (unconfirmed from outside keys or with RBF), defaults to true

Examples:

GET /address/:address/txs

GET /scripthash/:scripthash/txs

GET /wallet/:checksum/:index/txs

Get the list of all transactions in the history of the provided address, scripthash or descriptor index.

Expand...

Returned in the wallet transaction format.

Example:

GET /address/:address/txs/compact

GET /scripthash/:scripthash/txs/compact

GET /wallet/:checksum/:index/txs/compact

Get a compact minimal representation of the history of the provided address, scripthash or descriptor index.

Expand...

Returns a simple JSON array of [txid, block_height] tuples, where block_height is null for unconfirmed transactions.

Example:

Outputs

Output format

  • txid - the transaction funding this output
  • vout - the output index
  • amount - the output amount in satoshis
  • scriptPubKey fields
    • address - the address funded by this output
    • scripthash - the scripthash funded by this output
    • origin - descriptor wallet origin information in <checksum>/<index> format
    • desc - the descriptor for the funded script
    • bip32_origins - an array of bip32 origins for this script keys
  • block_height - the confirming block height or null for unconfirmed transactions
  • spent_by - the transaction input spending this output in txid:vin format, or null for unspent outputs (only available with track-spends)

GET /txo/:txid/:vout

Get information about the specified transaction output.

Expand...

Available for wallet outputs only.

Example:

GET /utxos

Get all unspent wallet outputs.

Expand...

Query string parameters:

  • min_conf - minimum number of confirmations, defaults to 0
  • include_unsafe - whether to include outputs that are not safe to spend (unconfirmed from outside keys or with RBF), defaults to true

Example:

Also see: GET /address/:address/utxos

Blocks

GET /block/tip

Get the current tip of the block chain.

Expand...

Returned fields:

  • height
  • hash

Example:

GET /block/:hash

Get the block header of the specified block hash as formatted by bitcoind's getblockheader with verbose=true.

Expand...

Example:

GET /block/:hash/hex

Get the block header of the specified block hash as a hex string.

Expand...

Example:

GET /block/:height

Get the block hash at the specified block height.

Expand...

Issues a 307 redirection to the block url (/block/:hash) and responds with the block hash in the responses body.

Example:

Mempool & Fees

GET /mempool/histogram

Get the mempool feerate distribution histogram.

Expand...

Returns an array of (feerate, vsize) tuples, where each entry's vsize is the total vsize of transactions paying more than feerate but less than the previous entry's feerate (except for the first entry, which has no upper bound). This matches the format used by the Electrum RPC protocol for mempool.get_fee_histogram.

Cached for 2 minutes.

Example:

In this example, there are transactions weighting a total of 102,131 vbytes that are paying more than 53 sat/vB, 110,990 vbytes of transactions paying between 38 and 53 sat/vB, 138,976 vbytes paying between 34 and 38, etc.

GET /fee-estimate/:target

Get the feerate estimate for confirming within target blocks. Uses bitcoind's smartestimatefee.

Expand...

Returned in sat/vB, or null if no estimate is available.

Cached for 2 minutes.

Example:

Server-Sent Events

Event categories

  • ChainTip(block_height, block_hash) - emitted whenever a new block extends the best chain.
  • Reorg(block_height, prev_block_hash, curr_block_hash) - indicates that a re-org was detected on block_height, with the previous block hash at this height and the current one.
  • Transaction(txid, block_height) - emitted for new transactions as well as transactions changing their confirmation status (typically from unconfirmed to confirmed, possibly the other way around in case of reorgs).
  • TransactionReplaced(txid) - indicates that the transaction conflicts with another transaction and can no longer be confirmed (aka double-spent).
  • TxoFunded(funding_txid:vout, scripthash, amount, block_height) - emitted when an unspent wallet output is created (for new transactions as well as confirmation status changes).
  • TxoSpent(spending_txid:vin, scripthash, prevout, block_height) - emitted when a wallet output is spent (for new transactions as well as confirmation status changes).

For unconfirmed transactions, block_height will be null.

GET /stream

Subscribe to a real-time Server-Sent Events stream of indexer update notifications.

Expand...

Query string parameters for filtering the event stream:

  • category
  • scripthash
  • outpoint

Examples:

GET /address/:address/stream

GET /scripthash/:scripthash/stream

GET /wallet/:checksum/:index/stream

Subscribe to a real-time notification stream of TxoFunded/TxoSpent events for the provided address, scripthash or descriptor index.

Expand...

This is equivalent to GET /stream?scripthash=<scripthash>.

Example:

Catching up with missed events & re-org detection

To catch-up with historical events that your app missed while being down, you can specify the synced-tip query string parameter with the <block-height>:<block-hash> of the latest block known to be processed.

If the synced-tip is still part of the best chain, this will return all historical Transaction, TxoFunded and TxoSpent events that occurred after block-height (exclusive, ordered with oldest first, unconfirmed included at the end), followed by a single ChainTip event with the currently synced tip, followed by a stream of real-time events.

If the synced-tip is no longer part of the best chain, a 410 Gone error will be returned indicating that a reorg took place. One way to recover from reorgs is to re-sync since N blocks before the orphaned synced-tip and consider any entries that no longer show up as double-spent (where N is large enough such that reorgs deeper than it are unlikely).

You can specify synced-tip with just the height to skip reorg detection (for example, 0 to get all events since the genesis block).

Expand... Example:

The synced-tip functionality also supports the SSE Last-Event-ID header. This makes it work transparently with the built-in automatic reconnection mechanism. You'll still need to manually persist and specify the synced-tip in case your app restarts.

Miscellaneous

POST /sync

Trigger an indexer sync. See Real-time updates.

GET /dump

Dumps the contents of the index store as JSON.

GET /debug

Dumps the contents of the index store as a debug string.

GET /banner.txt

Get the welcome banner text. (see live example here, or a static one here)

Web Hooks

You can set --webhook-url <url> to have bwt send push notifications as a POST request to the provided <url>. Requests will be sent with a JSON-serialized array of one or more index updates as the body.

It is recommended to include a secret key within the URL to verify the authenticity of the request.

You can specify multiple --webhook-url to notify all of them.

Note that bwt currently attempts to send the webhook request once and does not retry in case of failures. It is recommended to occasionally catch up using the GET /txs/since/:block-height or GET /stream endpoints (see "Catching up with missed events").

Tip: services like webhook.site or requestbin can come in handy for debugging webhooks. (needless to say, for non-privacy-sensitive regtest/testnet use only)

If you're building bwt from source, you'll need to set --features webhooks to enable web hooks support. This will also require to apt install libssl-dev pkg-config.

The pre-built binaries (except for ARM) and the shesek/bwt docker image come with webhooks support enabled by default.

Developing

Developer Resources

To integrate bwt into non-Rust software, check out libbwt and libbwt-jni.

Documentation for the public Rust API is available on docs.rs.

A yuml diagram showing how the big pieces interact together is available here.

An example of initializing bwt and issuing queries against its db from Rust is available at examples/use-from-rust.rs. (Note that the Rust API provides weaker backwards compatibility guarantees compared to the HTTP API.)

An example JavaScript client utilizing the HTTP API for wallet tracking is available at examples/wallet-tracker.js.

Development environment

To quickly setup a development environment, you can use scripts/dev-env.sh to create a bitcoind regtest network and two Electrum wallets, fund the wallets, start bwt with tracking for both wallets' xpubs, and start the Electrum GUI.

To use it, simply run $ ./scripts/dev-env.sh from the root directory with bitcoind, bitcoin-cli and electrum installed in your PATH.

You can set FEATURES to specify which features to enable (see below) or set NO_GUI=1 to leave the Electrum wallet running in daemon mode without starting the GUI.

If you have cargo watch installed, it'll be used to watch for changes and automatically restart bwt.

Features

bwt has 7 optional features: cli, http, electrum, webhooks, track-spends, proxy, ffi and extra.

All are enabled by default except for webhooks and ffi.

If you're working on code that is unrelated to the HTTP API, it is faster to build without the http feature.

You can use scripts/check.sh to run cargo check for some feature combos. This is important to ensure no errors were introduced for feature combos that you didn't use.

Tests

End-to-end integration tests can be run with ./test/tests.sh. The tests deploy a regtest network, a bwt instance and an Electrum wallet connected to it (in headless mode), then run some basic tests using the Electrum client and against the HTTP REST API.

Run with bash -x test/tests.sh -v to get more verbose output.

Contributions

Are welcome!

The only guideline is to use cargo fmt.

You can check out the list of enhancement issues.

Reproducible builds

The builds can be reproduced in a Docker container environment as follows:

The builds are reproduced on Travis CI using the code from GitHub. The SHA256 checksums are available under the "Reproducible builds" stage.

Thanks

  • @romanz's electrs for the fantastic electrum server implementation that bwt is based on.

  • @chris-belcher's electrum-personal-server for inspiring this project and the personal tracker model.

  • rust-bitcoin, rust-bitcoincore-rpc and the other incredible modules from the rust-bitcoin family.

License

MIT

Issues

Collection of the latest Issues

GordianLN

GordianLN

Comment Icon0

As per subject, this is what you get when you verify the file:

craigraw

craigraw

Comment Icon0

The configuration parameter --bitcoind_timeout was added in 5f92c47, which configures the connect, read and write timeouts on socket connections to Core RPC.

However, one does not typically want the same values for the socket connect timeout as the read and write timeouts. Importing a wallet using importmulti can take many minutes or even hours, much longer than a user would want to wait for a timeout when testing a connection. Therefore, currently it is not possible to both set a short connect timeout and a long read timeout to handle both situations.

It would be helpful to have an additional --bitcoind_connect_timeout parameter which specifically sets only the value for RPC socket connections at https://github.com/bwt-dev/bwt/blob/6be05b82bc156a94201b21c24529b1529d5347f6/src/util/jsonrpc_proxy.rs#L64

Alternatively, the current --bitcoind_timeout parameter could be altered to only control the socket connect timeout, since it is likely that read and write timeouts should be indefinite anyway.

craigraw

craigraw

Comment Icon9

Although I expect this will require some of the dependencies to be updated first, I would like to start by adding this issue to request support for Taproot (P2TR) wallets.

I am currently getting the following two errors when connecting Sparrow v1.4.3 to Bitcoin Core 0.21.1 over BWT:

jsarenik

jsarenik

enhancement
Comment Icon3

When bwt is run, following is printed:

This output is full of (probably fancy, just that I do not see them) UTF-8 characters. Even the half-sized capitals (for example Fᴇᴇ ʀᴀᴛᴇ) are printed as question-marks in my terminal (rxvt-unicode using Terminus font).

I think it would be a big help if there was an --ascii option, even better if ASCII output was the default and there was a --fancy option… What do you think @shesek ? Thank you for this amazing tool! It works very welly. This is just a way to say someone else uses it :)

tiero

tiero

Comment Icon4

It looks like it's based on electrs, how much of an effort would be make it work for elements chains?

Not that confident yet with Rust, but I could try to make contribution if you think is feasible to use the same codebase. If not, would be better to do it as a fork?

shesek

shesek

Comment Icon1
  • JNI bindings
  • BwtDaemon wrapper
  • Background worker
  • Settings screen
    • Custom date picker
    • Expandable advanced section
    • Help texts
    • Validation
    • Automatically restart the daemon
    • Defaults
  • Main screen
    • Controls
    • Progress bar
    • Status
    • Error reporting
    • Server info
    • Log viewer
      • Fix lag
      • Clear on startup
      • Fix INFO logging
  • QR
    • Display Electrum server details
    • Scan Bitcoin Core RPC server details
  • Separate repo
  • Build process
    • Reproducilbity
  • Documentation
Stadicus

Stadicus

Comment Icon5

Adding XPUBs and rescanning the Bitcoin blockchain is the one big drawback of lightweight Electrum middleware like BWT and EPS. This should be as easy as possible for the user, or even 3rd-party software to add XPUBs automatically.

The bwt.env config file seems like a good place to manage the keys that are monitored. I was happy to see that the following works in BWT 0.1.5:

  • adding multiple XPUBS= lines, not needing to concatenate the very long xpub strings into one, which would be a maintenance nightmare
  • adding the rescan arguemnt to the xpub, like on the command line: XPUBS='zpub12345....XYZ:2020-10-01'

While the first point still seems to work with BWT 0.2-RC1, the second usage gives now an error:

I think the option to easily add a new XPUB and specify the "rescan since" option just for that XPUB is very important. Adding it directly to the xpub string seems the easiest way to manage the keys.

Is this an unintended change? Or, if now, would it be feasible to make that work again?

shesek

shesek

Comment Icon0
  • Implement persistent storage (#4)

Bandwidth optimizations:

Tor support:

shesek

shesek

Comment Icon0
  • structopt (cli only, used for parsing arguments)
  • dotenv (cli only, used for config file)
  • dirs (cli only or enabled with extra, provides auto-detection of the bitcoin datadir)
  • pretty_env_logger (cli only or enabled with extra, used for logging)
  • signal-hook (cli only or enabled with extra, used to handle SIGINT/SIGTERM signals)
  • hex (removed)
  • chrono (also used for the welcome banner apart from the cli, but this could be avoided)
  • lazy_static
shesek

shesek

enhancement
Comment Icon0
  • Add --tx-broadcast-p2p <proxy-addr> option to broadcast directly to the P2P network using a socks proxy
  • Add --tx-broadcast-tor option as a shortcut for setting --tx-broadcast-p2p with tor
  • Enable --tx-broadcast-tor by default if tor is found on the machine?
shesek

shesek

Comment Icon2

Collisions are highly unlikely for typical bwt usage, but might become an issue in setups where there are many dynamically-managed xpubs, for example if each customer is given its own xpub for deposits.

This could be resolved by using the full 160 bit identifier, or possibly by using the descriptor checksums as the primary identifier instead of the bip32 fingerprint (once descriptor-based tracking is implemented, #1 / #37).

Versions

Find the latest versions by id

v0.2.4 - Mar 24, 2021

Changelog

  • Allow specifying an explicit script type for xpubs using -x <xpub>:wpkh, -x <xpub>:shwpkh or -x <xpub>:pkh.

  • Fix blockchain.block.headers sending wrong number of headers (#87, thanks @stevenroose!)

  • Check testmempoolaccept prior to custom broadcast (#80)

  • Dependency updates and various small enhancements


Also see the v0.2.4 releases for bwt-electrum-plugin, libbwt, libbwt-nodejs and libbwt-jni.

Installation

Installation instructions are available on the README.

Verifying signatures

The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on the PGP WoT, github, twitter, keybase, hacker news and this video presentation.

The signature verification should show Good signature from "Nadav Ivgi <[email protected]>" ... Primary key fingerprint: FCF1 9B67 ... and bwt-0.2.4-x86_64-linux.tar.gz: OK.

Reproducible builds

The builds are fully reproducible.

You can verify the checksums against the v0.2.4 builds on Travis CI: https://travis-ci.org/github/bwt-dev/bwt/builds/764306060

See more details here.

v0.2.3 - Mar 17, 2021

Changelog

  • New --prune-until <target> option to automatically prune the chain up to the given target (height, unix timestamp or YYYY-MM-DD formatted date)

    Require configuring bitcoind with prune=1 to allow manual pruning via the RPC.

  • New --no-wait-sync option to allow importing addresses without waiting for bitcoind to finish syncing first

    Useful for tracking wallets during the IBD of a pruned node, so transactions could be indexed before the blocks get pruned.

  • Fix compatibility with the upcoming v0.22 Bitcoin Core release (https://github.com/sparrowwallet/sparrow/issues/67#issuecomment-791522237)

  • HTTP: Set the WWW-Authenticate header to make logging-in possible via web browsers

  • Implement an fd-based daemon readiness notification mechanism, enabled with FD_NOTIFY=<fd>

  • Fix whitepaper extractor

  • UX touchups for progress indicator, welcome banner and logs

  • Allow setting UNIX_LISTENER_MODE to control permissions for the unix socket notification listener

  • Allow setting NO_REQUIRE_ADDRESSES as an env variable

Breaking CLI changes:

  • The Electrum SOCKS5-based authentication needs to be explicitly enabled with --electrum-socks-auth, in addition to enabling the --auth-* options. By default, authentication will only be enabled for the HTTP API server.

Also see the v0.2.3 releases for bwt-electrum-plugin, libbwt, libbwt-nodejs and libbwt-jni.

Installation

Installation instructions are available on the README.

Verifying signatures

The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on the PGP WoT, github, twitter, keybase, hacker news and this video presentation.

The signature verification should show Good signature from "Nadav Ivgi <[email protected]>" ... Primary key fingerprint: FCF1 9B67 ... and bwt-0.2.3-x86_64-linux.tar.gz: OK.

Reproducible builds

The builds are fully reproducible.

You can verify the checksums against the v0.2.3 builds on Travis CI: https://travis-ci.org/github/bwt-dev/bwt/builds/763199070

See more details here.

v0.2.2 - Jan 29, 2021

Changelog

  • Authentication support for the Electrum and HTTP API servers (#70)

  • Compatibility with Bitcoin Core v0.21 (#78)

  • Support for Signet

  • New GET /bitcoin.pdf HTTP API endpoint for extracting the Bitcoin whitepaper from the block chain (https://twitter.com/shesek/status/1352368296553836544)

  • New --create-wallet-if-missing option to ease the creation of a designated bitcoind wallet (#76)

  • Docker: Multi-arch images for amd64, arm32v7 and arm64v8 (#79)

  • Indexer: Fix detection of conflicted mempool transactions

  • Support setting boolean options using environment variables (FORCE_RESCAN, CREATE_WALLET_IF_MISSING, ELECTRUM_SKIP_MERKLE, NO_STARTUP_BANNER and VERBOSE)

  • Accept wildcard envirnoment variables for options that accept multiple values (XPUB_*, DESC_*/DESCRIPTOR_* and ADDRESS_*)

  • Upgrade to rust-bitcoin v0.26.0, rust-miniscript v5.0.1 and bitcoincore-rpc v0.13.0


Also see the v0.2.2 releases for bwt-electrum-plugin, libbwt, libbwt-nodejs and libbwt-jni.

Installation

Installation instructions are available on the README.

Verifying signatures

The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on the PGP WoT, github, twitter, keybase, hacker news and this video presentation.

The signature verification should show Good signature from "Nadav Ivgi <[email protected]>" ... Primary key fingerprint: FCF1 9B67 ... and bwt-0.2.2-x86_64-linux.tar.gz: OK.

Reproducible builds

The builds are fully reproducible.

You can verify the checksums against the v0.2.2 builds on Travis CI: https://travis-ci.org/github/bwt-dev/bwt/builds/756663238

See more details here.

v0.2.1 - Jan 14, 2021

Changelog

  • Migrated to the @bwt-dev github org and split up into:

    bwt, bwt-electrum-plugin, libbwt, libbwt-jni and libbwt-nodejs.

  • Java Native Bindings for libbwt (libbwt-jni, #73)

  • Support for tracking standalone addresses (#14)

    Using --address <address> or --address-file <path>.

  • New config options: force_rescan (9e7ccbe), setup_logger (35fc49f) and require_addresses (162790d)

  • Gracefully wait for bitcoind to warm-up (dec6d46)

  • Support for android_logger (74b2b2f)

  • Scrub bitcoin authentication from logs (c31def7)

  • Improved syncing/scanning progress updates (faba3f, 6e282fd, fdd46f3, 5ba2a0b)

  • Indexer: Fix excessive importing/rescanning (a20ae79)

  • Indexer: Fix cache invalidation for spends lookups (360eaee)

  • Indexer: Fix handling of missing mempool entries (e9b7511)

  • Electrum: Fix TCP listener not shutting down on shutdown signal (5bd639a)

  • Docker/CI: Update to Rust v1.49 (5bd639a)

Breaking CLI changes:

  • The --bare-xpub option was removed. Use a descriptor instead.

Also see the v0.2.1 releases for bwt-electrum-plugin, libbwt, libbwt-nodejs and libbwt-jni.

Installation

Installation instructions are available on the README.

Verifying signatures

The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on the PGP WoT, github, twitter, keybase, hacker news and this video presentation.

The signature verification should show Good signature from "Nadav Ivgi <[email protected]>" ... Primary key fingerprint: FCF1 9B67 ... and bwt-0.2.1-x86_64-linux.tar.gz: OK.

Reproducible builds

The builds are fully reproducible.

You can verify the checksums against the v0.2.1 builds on Travis CI: https://travis-ci.org/github/bwt-dev/bwt/builds/754451937

See more details here.

v0.2.0 - Nov 24, 2020

Changelog

  • Descriptor based tracking! ✨🎉 (#1)

    You can now specify output script descriptors to track via --descriptor <descriptor>. Descriptors are also used internally to represent user-provided --xpubs.

    The HTTP API was updated to be fully descriptor-based. Wallets and wallet origins are now identified by the descriptor checksum, addresses have descriptors associated with them, and a new bip32_origins field is available based on the descriptor origin information.

  • Support for Electrum multi-signature wallets (#60)

    For a manual server setup, this requires using the sortedmulti() descriptor. For example, for a 2-of-3 wallet: sortedmulti(2,xpub1...,xpub2...,xpub3...).

    With the Electrum plugin, this should Just Work™.

  • Alpha release of libbwt (#64), a C FFI interface for managing the bwt servers, and of nodejs-bwt-daemon (#65), a nodejs package that wraps it.

  • Support non-wallet transactions in blockchain.transaction.get / GET /tx/:txid/hex (requires txindex and no pruning)

  • Emit wallet rescan and blockchain sync progress updates (via mpsc, ffi and the console)

  • Support binding on ephemeral port (e.g. --http-addr 127.0.0.1:0) (#63)

  • Reduce the number of dependencies (#61)

  • Shutdown cleanly, via SIGINT/SIGTERM for CLI or a custom signal for library users (#62, #66)

  • HTTP: Alias GET /txs/since/0 as GET /txs

  • Fix blockchain.scripthash.listunspent / Query::list_unspent to return an empty set instead of erroring when there's no history.

  • Electrum: Fix mempool.get_fee_histogram (5af7bfc62d7d98)

  • Upgrade to rust-bitcoin v0.25, rust-miniscript v4.0.0 and rust-bitcoincore-rpc v0.12

Breaking CLI changes:

  • The -d CLI option was changed to mean --descriptor instead of --bitcoind-dir (which is now available as -r).

  • Renamed --http-server-addr to --http-addr and --electrum-rpc-addr to --electrum-addr

  • The CLI now accepts a single --rescan-since timestamp instead of a separate one for each descriptor/xpub.

  • The separator for environment variables with multiple values is now ; instead of ,. For example: DESCRIPTORS="wpkh(xpub../0/*);wpkh(xpub../1/*)"


Downloads

Full Server 1 Electrum Server 2 Electrum Plugin 3
Linux 📥 Download 📥 Download 📥 Download
macOS 📥 Download 📥 Download 📥 Download
Windows 📥 Download 📥 Download 📥 Download
ARMv7 📥 Download 📥 Download 📥 Download
ARMv8 📥 Download 📥 Download 📥 Download
1 Includes Electrum RPC, HTTP API and WebHooks support (learn more)
2 Includes Electrum RPC support only (learn more)
3 Plugin with an embedded RPC server - hot wallets are unsupported (learn more)

Installation

Installation instructions are available on the README.

Verifying signatures

The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on keybase, github, twitter and HN. The signature can be verified as follows (replace x86_64-linux with your download):

You should see Good signature from "Nadav Ivgi <[email protected]>" ... Primary key fingerprint: FCF1 9B67 ... and bwt-0.2.0-x86_64-linux.tar.gz: OK.

Reproducible builds

The builds are fully reproducible.

You can verify the checksums against the v0.2.0 builds on Travis CI: https://travis-ci.org/github/shesek/bwt/builds/745668826

See more details here.

v0.2.0-rc.1 - Nov 21, 2020

v0.1.5 - Oct 05, 2020

Changelog

  • Reproducible builds using Docker (#51)

  • Pre-built binary releases for macOS (#24) and ARMv7/v8 (#19)

  • Electrum plugin: Compatibility with Electrum v4 — except for lightning which is tricky with personal servers (#53)

  • Electrum: New welcome banner (#44)

  • Scriptable transaction broadcast command via --tx-broadcast-cmd <cmd> (#7)

    The command will be used in place of broadcasting transactions using the full node, which may provide better privacy in some circumstances. The string {tx_hex} will be replaced with the hex-encoded transaction.

    For example, to broadcast transactions over Tor using the blockstream.info onion service, you can use:

    (Replace port 9050 with 9150 if you're using the Tor browser bundle.)

    h/t @chris-belcher's EPS for inspiring this feature! 🎩

  • Load bitcoind wallet automatically (#54)

  • Electrum plugin: Fix hot wallet test (#47)

  • Electrum: Fix docker image libssl dependency with the http feature (#48)

  • Improve block download check on regtest (#45, #35)

  • HTTP API: Fix GET /block/tip (#46)

  • HTTP API: Add GET /banner.txt (#44)

  • Tests: Upgrade to Electrum v4


Downloads

Full Server 1 Electrum Server 2 Electrum Plugin 3
Linux 📥 Download 📥 Download 📥 Download
macOS 📥 Download 📥 Download 📥 Download
Windows 📥 Download 📥 Download 📥 Download
ARMv7 📥 Download 📥 Download 📥 Download
ARMv8 📥 Download 📥 Download 📥 Download
1 Includes Electrum RPC, HTTP API and WebHooks support (learn more)
2 Includes Electrum RPC support only (learn more)
3 Plugin with an embedded RPC server - hot wallets are unsupported (learn more)

Installation

Installation instructions are available on the README.

Verifying signatures

The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on keybase, github, twitter and HN. The signature can be verified as follows (replace x86_64-linux with your download):

You should see Good signature from "Nadav Ivgi <[email protected]>" ... Primary key fingerprint: FCF1 9B67 ... and bwt-0.1.5-x86_64-linux.tar.gz: OK.

Reproducible builds

The builds are fully reproducible.

You can verify the checksums against the v0.1.5 builds on Travis CI: https://travis-ci.org/github/shesek/bwt/builds/732966695

See more details here.

v0.1.4 - Jun 22, 2020

Changelog

  • Implement improved mempool tracking, including support for an "effective feerate" metric that takes unconfirmed ancestors into account, calculated as MIN(own_fee/own_vsize, (own_fee+ancestor_fee)/(own_vsize+ancestor_vsize)).

    HTTP API: the wallet transaction format now includes new own_feerate, effective_feerate, bip125_replaceable and unconfirmed_parents fields available for unconfirmed transactions.

    Electrum server: provide fee information for unconfirmed transactions using the effective feerate metric. This is unlike other Electrum server implementations, that report the direct own fee without regard to ancestors. (#10)

  • Electrum server: Implement --electrum-skip-merkle to save some resources by not generating SPV proofs entirely, even when it's possible. (#34)

  • Electrum plugin: Automatically enable --skipmerklecheck and --electrum-skip-merkle, for better out-of-the-box pruning support and to save some resources. (#34)

  • Indexer: Use listsinceblock instead of listtransactions. This makes syncing more bandwidth-efficient and simplifies the implementation. (#33)

  • Electrum server: Optimize dispatching notifications to subscribers.

  • Electrum server: Use height of -1 to indicate that a transaction has unconfirmed parents as its inputs. (#40)

  • Electrum plugin: Disable support for hot wallets.

  • Make rust builds 65% faster (#15, 8e785cbb2dc63ce599a2f65d3d5a162da9a52b8c)


Installation

Installation instructions are available on the README.

Verifying signatures

The releases are signed by Nadav Ivgi (@shesek). The public key can be verified on keybase, github, twitter and HN. The signature can be verified as follows (replace x86_64-linux with your download):

You should see Good signature from "Nadav Ivgi <[email protected]>" ... Primary key fingerprint: FCF1 9B67 ... and bwt-0.1.4-x86_64-linux.tar.gz: OK.

Electrum plugin

The Electrum plugin is available for download for Linux and Windows, as the electrum_plugin package.

⚠️ NOTE: The plugin supports watch-only wallets only and cannot be used with hot wallets. This is done as a security measure, which is expected to eventually be relaxed. You can use the plugin with hardware wallets or with an offline Electrum setup. For hot wallets, you will need to setup a standalone server instead of using the plugin, ideally away from your keys.

v0.1.3 - Jun 02, 2020

  • Electrum: Use dummy SPV proofs to support pruning with the --skipmerklecheck option.

v0.1.2 - May 30, 2020

  • Electrum plugin: restore the previous oneserver setting when the plugin is disabled, to prevent users from inadvertently connecting to public Electrum servers with this setting still on.

  • Electrum plugin: allow specifying additional custom CLI arguments using the GUI

  • Electrum plugin: check for permissions before attempting the bind the real-time sync unix socket.

  • Make builds over 40% smaller by stripping symbols, which rust apparently doesn't do for release builds. Thanks @elichai for brining this to my attention.

v0.1.1 - May 27, 2020

  • Make bwt available as an Electrum plugin! 💥

  • HTTP: Implement the synced-tip option to catch up with missed events (#6)

  • Unite the History event into Txo{Funded,Spent}

  • Fix: Update the confirmation status of send-only (no change) transactions

v0.1.0 - May 20, 2020

First release!

Information - Updated Sep 20, 2022

Stars: 233
Forks: 25
Issues: 44

Repositories & Extras

Bitcoin Wallet Tracker - C FFI

Support development: bwt, libbwt-nodejs

Bitcoin Wallet Tracker - C FFI

Bitcoin Wallet Tracker - JNI bindings

Java Native Interface bindings for HTTP API

Bitcoin Wallet Tracker - JNI bindings

Bitcoin Hashes Library

This is a simple, no-dependency library which implements the hash functions

Bitcoin Hashes Library

Bitcoin vault architecture for multi-party situations

as well as the #revault on Libera

Bitcoin vault architecture for multi-party situations

bitcoind-observer

An experimental Prometheus metric exporter for Bitcoin Core based on Userspace,

bitcoind-observer

Bitcoin vault architecture for multi-party situations

Released under the BSD 3-Clause Licence

Bitcoin vault architecture for multi-party situations

Bitcoin vault architecture for multi-party situations

#revault on Libera for discussing Revault development

Bitcoin vault architecture for multi-party situations

Bitcoin Mining Pool Identification

This Rust crate implements a new PoolIdentification trait on rust-bitcoin's

Bitcoin Mining Pool Identification

bitcoinquery is a python package developed in rust

for accessing bitcoin-core binary files as a database

bitcoinquery is a python package developed in rust

bitcoinquery is a python package developed in rust

for accessing bitcoin-core binary files as a database

bitcoinquery is a python package developed in rust

bitcoin-exporter

Serve bitcoind core metrics under /metrics path

bitcoin-exporter
Facebook Instagram Twitter GitHub Dribbble
Privacy