Testing BTC-RPC-Proxy to simulate an unpruned node
openoms opened this issue · 29 comments
From: #298 (comment)
This code https://github.com/Start9Labs/btc-rpc-proxy-wrapper (though in another language) is able to fetch pruned blocks from other peers.
Based on: https://github.com/Kixunil/btc-rpc-proxy (written in Rust)
This could help Specter to run a with pruned node and maybe would make the use of external block explorers unnecessary (see #718)
Quick installation steps as I did on Debian Linux: https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6#file-btc-rpc-proxy-md
Need to allow all RPC commands used. Got these when searched for the self.rpc.commands in specter-desktop: https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6#configure , but likely missing something still.
When running Specter (python3 -m cryptoadvance.specter server) the detection of the bitcoinRPC on localhost (BTC-RPC-proxy) works and connection is tested successfully but immediately runs into these errors:
https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6#file-specter-desktop-log
Same time errors in BTC-RPC-Proxy are showing the bad requests:
https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6#file-btc_rpc_proxy-log
Either using BTC-RPC-Proxy is really close and I am just missing something or the next step could be to request @Kixunil for the feature if we figure it out what is exactly needed.
Hi, happy you're interested!
Quick installation steps as I did on Debian Linux
You don't need to install Rust by piping commands to sh on Debian 10. Just apt install cargo. You need to use --features=compat during cargo run/build/install but that should be fine.
I'm now looking into re-packaging it in my Debian repository, so you will have even easier time installing. :)
Also your instructions describe cloning master, which does not yet contain block fetching. Not sure if mistake in the instructions or you actually attempted that. The new version is here: https://github.com/Start9Labs/btc-rpc-proxy/tree/feature/refactor
Regarding configuration, daemon cookie authentication is supported and recommended. I plan to implement it for users as well eventually. Also you seem to be missing fetch_blocks = true in [user.specter] section. (Should be unrelated to error messages you posted.)
but immediately runs into these errors:
Would you be willing to try again and use Wireshark to dump the exact request? (right click on one of the request packets -> follow TCP stream) (feel free to censor private information)
There's a possibility it got fixed during refactor but I'd like to understand what I was doing wrong so that I can prevent it in the future.
Then if you actually tried the old version, try again with the new version.
and I am just missing something
It's still an alpha feature. :) But it's intended to be entirely transparent so that the application doesn't even know it's not a real bitcoind. Therefore if there's any problem, definitely ping us.
Thank you will test and update the instructions.
$ cargo --version
cargo 1.42.1
did now:
sudo apt install cargo
$ cargo --version
cargo 1.42.1
then
git clone https://github.com/Start9Labs/btc-rpc-proxy.git
cd /home/brp/btc-rpc-proxy
git checkout feature/refactorYou don't need to install Rust by piping commands to sh on Debian 10. Just apt install cargo. You need to use --features=compat during cargo run/build/install but that should be fine.
$ cargo build --features=compat --release
error: Package `btc-rpc-proxy v0.2.3 (/home/brp/btc-rpc-proxy)` does not have these features: `compat`
and I get this with:
~/btc-rpc-proxy$ cargo build --release
Compiling btc-rpc-proxy v0.2.3 (/home/brp/btc-rpc-proxy)
error[E0658]: use of unstable library feature 'str_strip': newly added
--> src/rpc_methods.rs:207:22
|
207 | host.strip_suffix(".onion")
| ^^^^^^^^^^^^
|
= note: for more information, see https://github.com/rust-lang/rust/issues/67302
error[E0658]: use of unstable library feature 'str_strip': newly added
--> src/users.rs:27:31
|
27 | let auth = header_str.strip_prefix("Basic ")?;
| ^^^^^^^^^^^^
|
= note: for more information, see https://github.com/rust-lang/rust/issues/67302
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0658`.
error: could not compile `btc-rpc-proxy`.
To learn more, run the command again with --verbose.
git clone -b feature/refactor or git checkout feature/refactor ;)
Sorry, misread compat was renamed to old_rust a few hours ago and I didn't notice it.
compatwas renamed toold_rusta few hours ago and I didn't notice it.
ok so instead of cargo build --release using now:
cargo build --features=old_rust
but only made it one step further:
Compiling btc-rpc-proxy v0.2.3 (/home/brp/btc-rpc-proxy)
Compiling bitcoin v0.25.2
error: reached the type-length limit while instantiating `std::pin::Pin::<&mut std::future...}[1]::poll[0]::{{closure}}[0])]>`
|
= note: consider adding a `#![type_length_limit="1717928"]` attribute to your crate
error: aborting due to previous error
error: could not compile `btc-rpc-proxy`.
To learn more, run the command again with --verbose.
--verbose adds:
Caused by:
process didn't exit successfully: `rustc --crate-name btc_rpc_proxy --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 --cfg 'feature="old_rust"' -C metadata=8a6d28061db7a096 -C extra-filename=-8a6d28061db7a096 --out-dir /home/brp/btc-rpc-proxy/target/debug/deps -C incremental=/home/brp/btc-rpc-proxy/target/debug/incremental -L dependency=/home/brp/btc-rpc-proxy/target/debug/deps --extern anyhow=/home/brp/btc-rpc-proxy/target/debug/deps/libanyhow-a60a363e5d463479.rlib --extern async_channel=/home/brp/btc-rpc-proxy/target/debug/deps/libasync_channel-ca2c185d99c63ccc.rlib --extern base32=/home/brp/btc-rpc-proxy/target/debug/deps/libbase32-dd9c6c99f7aa7297.rlib --extern base64=/home/brp/btc-rpc-proxy/target/debug/deps/libbase64-cc0a44c658e37c07.rlib --extern bitcoin=/home/brp/btc-rpc-proxy/target/debug/deps/libbitcoin-31748b201c53a99e.rlib --extern btc_rpc_proxy=/home/brp/btc-rpc-proxy/target/debug/deps/libbtc_rpc_proxy-81550c3dd6e26901.rlib --extern configure_me=/home/brp/btc-rpc-proxy/target/debug/deps/libconfigure_me-f3f495b33f6eb7cc.rlib --extern derive_more=/home/brp/btc-rpc-proxy/target/debug/deps/libderive_more-dc99989a833fc446.so --extern enum_future=/home/brp/btc-rpc-proxy/target/debug/deps/libenum_future-449d7b56773ce8da.rlib --extern futures=/home/brp/btc-rpc-proxy/target/debug/deps/libfutures-130c99e4757d5bad.rlib --extern hex=/home/brp/btc-rpc-proxy/target/debug/deps/libhex-3cbd71142fd11adf.rlib --extern http=/home/brp/btc-rpc-proxy/target/debug/deps/libhttp-bb230329d5ab06a6.rlib --extern hyper=/home/brp/btc-rpc-proxy/target/debug/deps/libhyper-18ff0ac97f68344d.rlib --extern itertools=/home/brp/btc-rpc-proxy/target/debug/deps/libitertools-13d0639a11625612.rlib --extern lazy_static=/home/brp/btc-rpc-proxy/target/debug/deps/liblazy_static-c29a2df2db37e551.rlib --extern linear_map=/home/brp/btc-rpc-proxy/target/debug/deps/liblinear_map-1ae9d653bdff1c99.rlib --extern rand=/home/brp/btc-rpc-proxy/target/debug/deps/librand-f810091db6f4f624.rlib --extern serde=/home/brp/btc-rpc-proxy/target/debug/deps/libserde-96ed189b4695cdfb.rlib --extern serde_json=/home/brp/btc-rpc-proxy/target/debug/deps/libserde_json-985bac7fce40e2c5.rlib --extern slog=/home/brp/btc-rpc-proxy/target/debug/deps/libslog-509e6cc692f62a51.rlib --extern slog_async=/home/brp/btc-rpc-proxy/target/debug/deps/libslog_async-de5fbff42b9096bc.rlib --extern slog_term=/home/brp/btc-rpc-proxy/target/debug/deps/libslog_term-8f1dca20b7419dfa.rlib --extern socks=/home/brp/btc-rpc-proxy/target/debug/deps/libsocks-6eadc86464380ab5.rlib --extern tokio=/home/brp/btc-rpc-proxy/target/debug/deps/libtokio-85e6852c5e59b92d.rlib -L native=/home/brp/btc-rpc-proxy/target/debug/build/secp256k1-sys-af69f7285ae39d23/out` (exit code: 1)
Ah, yeah, I reported it and it seems they forgot to add it. The suggestion from the compiler works, so for the time being you can manually add #![type_length_limit="1717928"] line at the top of src/main.rs
It works now:
Can see a wall of RPC calls passing through the Proxy:
no errors in the Specter server and no popup about the pruned node either!
Updating instructions and will explore more.
Next problem is that Specter still perceives as it is a pruned node:
Specter server:
ERROR in controller: Request error: Can't rescan beyond pruned data. Use RPC call getblockchaininfo to determine your pruned height. while rescanblockchain
No errors in the Proxy.
Updated the instructions to show how I got this far: https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6
Nice!!! I still want to test it a bit more before merging and signing as the official version. Perhaps doing a second pass of review as well. If you know of other projects that need non-pruned node, then you can help push this forward by trying out the proxy with them as well and reporting your findings.
I also want to add a new configuration option for log level later so that production users are not spammed with those messages. The logs are quite useful for the time being.
Weird, GitHub showed me your latest message only after I commented...
Looks like specter attempted wallet rescan and that seems to be unimplemented in btc-rpc-proxy.
@dr-bonez are you aware of this? I think we should support it somehow, although it'll likely be heavy - will need to re-download whole chain. Perhaps caching block filters could be a reasonable trade-off.
Thinking about it again, I think adding electrs support as I suggested in the PR could help with this too. Yeah, electrs has 68G on my machine but that's still less than 365G of chain. And rescan would become much quicker. What do you think?
now tried to import my backed up wallets while connected to BTC-RPC-Proxy backed by a real unpruned bitcoind node.
It went on to rescan indefinitely repeating this in the Specter logs:
INFO in wallet: Didn't get transactions on address xxxxxx. Refilling keypool.
WARNING in wallet_manager: Couldn't find wallet xxxxx in core's wallets.Silently ignored!
WARNING in wallet_manager: Couldn't find wallet xxxxx in core's wallets.Silently ignored!
...
WARNING in wallet_manager: Couldn't find wallet xxxxx in core's wallets.Silently ignored!
WARNING in wallet_manager: Couldn't find wallet xxxxxr_3 in core's wallets.Silently ignored!
INFO in wallet: Didn't get transactions on address xxxxxxxx. Refilling keypool.
The scanning was triggered in bitcoind, but as it seems the wallets couldn't be saved while going through the Proxy.
There were no errors in the Proxy logs.
Specter loads and unloads wallets to bitcoind and that must be not playing well together with BTC-RPC-Proxy.
Proxy should have no effects on this behavior. It looks like you forgot to restore the wallet files or something.
If the content of the bitcoind directory looks OK (contains the wallet xxxx), then please try same setup, just without proxy - set your node directly in Specter.
You are right, it was a problem with restoring with a used bitcoind wallet. I've recreated the bitcoind wallet.dat and emptied the wallets directory from .specter and the errors are gone.
Works through the BTC-RPC-proxy too.
I am inclined to close this issue since it is not (yet) possible to do rescan on a pruned node without redownloading the full blockchain or plugging an electrs instance to either Specter itself or to BTC-RPC-Proxy.
Looking forward to the further developments on BTC-RPC-Proxy and it was well proven that it is capable of setting the fine-grained bitcoin RPC permissions as it was intended to.
I'm not sure if this is even solvable problem at all. If you don't have any kind of index or wallet metadata, then downloading the whole chain (in theory does not have to be stored) is the only trustless way to recover a wallet.
But if you're still interested in some of the other tradeoffs available, I'd love to hear your opinion which of them sounds most reasonable to you so that maybe I can look into it one day.
Options that seem to be realistic:
- Download whole chain (without storing - consumes bandwidth, not storage) and do trivial rescan
- Try to fetch block filters from peers and download relevant blocks
- Try to cache block filters during the lifetime of the node and use them. The question is what should happen if there are missing filters.
- Use
electrsto get the relevant transactions/blocks
I think
Try to fetch block filters from peers and download relevant blocks
would be the most desirable option given there would be more nodes serving them.
Also this option was I heard @stepansnigirev talk about on this recent podcast: https://stephanlivera.com/episode/231/.
Yeah, that sounds pretty good to me too. We might also check if bitcoind can keep them despite pruning and set it to do so. But AFAIK only Knots can send filters over RPC.
Try to fetch block filters from peers and download relevant blocks
just a note, this is a privacy leak. Ideally it should be configurable as to whether you download the whole chain, or only blocks that you need.
Not extremely big privacy leak, I believe. It should be hard to correlate transactions from whole blocks (which even have false-positives) and each block (candidate) could be downloaded over separate Tor connection.
Having it configurable for paranoid users would be nice of course.
Simulating blockfilterindex (providing the getblockfilter rpc) for a pruned node would be a huge win, since I bet it will be a long time before such a thing ends up in core, however it would change the role of proxy from being just a light proxy, to something that needs to sync. If you want to go that route though I'd be happy to help.
I'm definitely not opposed to the idea of having the proxy a bit heavier if it is configurable (opt-in) and the code is properly organized. Especially if it means Bitcoin users no longer have to store 300G of chain data to access very useful Bitcoin features. I don't think I can find the time to work on it myself in the following month or two but I could review/merge a PR if you're motivated to do it sooner.
FYI, @chrisguida is working on this. It requires giving proxy control over pruning though, to do it efficiently, so that it can ensure undo files are not deleted before they are used.
That's great news! I'm not very happy with controlling pruning over RPC as it'll either break my setup or I'll have to do some serious redesign. Would be nice if fallback was possible.
But I do realize the benefits so I do intend to think about redesigning. It'll just take some time to do it.
@chrisguida yes I know, I wrote that part of the docs :) It should be compatible with RPC proxy faking non-pruning though as long as jsonrpc_import is used. Didn't test yet but would be highly surprised if it didn't work.
@Kixunil the alternative is to maintain a second UTXO set, separate from core in order to know the scripts associated with spends. It's much slower and takes a lot more storage.
@dr-bonez ah, didn't realize those scripts are needed to build block filters. Would appreciate if it was possible to turn this feature off which would also stop control of pruning. At least for some time.
What I'm more concerned about is that if a user has already installed pruned version and wants to start using btc-rpc-proxy later, it'd fail. So some kind of fallback even if slow and temporarily using more storage would be useful. I can understand if this is not priority, just something to keep in mind and perhaps design the code to not be hard to add.
oh for sure. All of these kinds of features should be opt in. But they won't get block filters if they don't turn it on.
the overhead of an additional fallback system probably isn't worth it right now. But they are adding support for block filters to pruned nodes in 0.22, so this is a temporary holdover anyway.
Ah, my apologies, I hadn't noticed that 😅
Waiting for 0.22 is not too bad.
@chrisguida no problem, no offense taken. Just had some laugh. :)