[WIP] API design considerations
Opened this issue ยท 22 comments
API design considerations
Background
There multiple different types of API that can be used by the software stack
related to LNP/BP. Here we analyze criteria to choose the proper API
technologies and serialization standards for different cases.
In general, software might require API for:
- Interprocess communications (IPC), including those between daemons, their
instances, or used in microservice architectures on the same machine or in
network-connected docker containers behind DMZ. - Non-web-based client-server interactions crossing DMZ, following either
request/reply or subscribe/publish patterns. - REST Web-based APIs for requesting resource-based data (i.e. with clear
data hierarchical structure) - Real-time or transactional web-based APIs, including requests for remote
procedures (RPC) over web or bidirectional/realtime (RT) client-server
communications using Websockets.
API type | Sample use cases | Typical scenarios |
---|---|---|
IPC | c-lightning IPC | Microservice IPC for servers and daemons |
Non-web client-server | electrum protocol; bitcoind RPC | High-throughput or non-custodial solutions |
Web-based REST | esplora | Blockchain explorers |
Web-based RPC/RT | many web apps | Wallets |
Today, many different API description languages, serialization formats and
transport layers exist that may be used in the mentioned scenarios. However, in
most of the cases the choice of the particular formats are nearly arbitrary or
related to historical reasons. Here I'd like to systematize criteria for API
technic selection in LNP/BP for future apps that may allow to avoid many bad
practices of the past.
Overview
API components
The classical API consists of three main components:
- Data serialization format, allowing all parties involved in communication
read/write data with the same deterministic result. Usually classified to be
binary or textual basing on human readibility/ASCII character set. The most
common formats are:- ASCII-based/text/human readable:
- XML (XML Schema)
- JSON (JSON Schema)
- YAML (YAML Schema)
- Binary serialisation
- BSON
- Protocol buffers
- ASN.1
- RPC framework-specific (like used in ZMQ, Apache Thrift)
- Custom/vendor-specific (Bitcoin core and others)
Many serialization formats have a schema- (mostly for human-readable) or
DSL-based definition of possible values used by a particular application/API,
which may be used for an automatic code generation and/or data packet
validation.
- ASCII-based/text/human readable:
- API per se, specifying available resources or procedures which may be invoked
via IPC/network communication. Thus, API may fall into two classes:- RPC (remote procedure call), where each API call consists of the invoked
procedure name and a list of it's arguments - very much alike procedural
programming languages. Server-side components with RPC paradigm usually
have their state. - REST (representational state transfer), used to call ACID-based methods
on a well-defined hierarchical graph of resources - Custom/non-standard approaches, like GraphQL
- RPC (remote procedure call), where each API call consists of the invoked
- Transport-layer protocol, defining the means of transporting information
about API calls and associated data over the underlying network topology:- POSIX sockets
- POSIX IPC
- TCP/IP
- UDP/IP
- HTTP (pure or over TLS/SSL)
- Websockets (pure or over TLS/SSL)
- Tor/SOCKS
Many existing API automation frameworks (see below) cover more than a single
API component.
API Protocols and Frameworks
Here we provide information only about modern and most recently used frameworks:
Framework name / protocol family | Layers | Transport protocol requirements | Best suited/designed for |
---|---|---|---|
Apache Thrift | 1 (many), 2 (RPC), 3 (custom) | HTTP(s), TCP | Microservice architectures (only Req/Rep however) |
GraphQL | 1 (JSON), 2 (custom) | HTTP(s) | Complex data-centric web applications with non-hierarchical data graphs |
gRPC/Protobuf | 1 (binary/custom), 2 (RPC) | HTTP(s), TCP, ? | Microservice architectures (only Req/Rep however) |
JSON-RPC | 1 (JSON), 2 (RPC) | HTTP | Legacy/insecure |
OpenAPI | 1 (JSON), 2 (REST) | HTTP(s) | REST web applications |
SOAP/WSDL | 1 (XML), 2 (RPC) | HTTP | Enterprise system bus-centered enterprise architectures |
WAMP | 1 (JSON or other), 2 (RPC) | Websockets, TCP, POSIX | Real-time web apps, socket-based apps |
XML-RPC | 1 (XML), 2 (RPC) | HTTP | Legacy/insecure |
ZeroMQ | 1 (binary/custom), 2 (RPC) | POSIX sockets, POSIX IPC, TCP, USD | High throughput, Pub/Subs, IPCs, Microservice architectures |
IPC for Microservices
The requirements for this are:
- Compact binary data serialization format
- Support for custom serialization (i.e. consensus-based for Bitcoin-related
data structures) - No third-party code generation tools (safety for consensus-critical data)
- High throughput transport
- Support for all types of IPC sockets including Tor
- Ability to use encryption at transport layer
- Support of Request-Reply (RPC) and Publish-Subscribe patterns
- Well suited for serialization of hashes, public keys etc.
Much less important for the protocols:
- Web compatibility
- Human readability
ZeroMQ seems to be a tool of choice for the transport layer, which have to be
combined with custom RPC API DSL and serialization protocol.
Client-server (non-web)
ZeroMQ seems to be the tool of the choice here as well
Web-based REST
OpenAPI seems to be the tool of the choice.
Web-based RPC
WAMP seems to be the tool of the choice for apps that require live updates
(Websockets).
Another alternative to consider is GraphQL, however it should be noted that id
usually has a poor performance and is not suited for Websocket apps.
End notes
Protocol buffers or Apache Thrift serialization can't be used in all of the cases due to:
- A lot of code generation
- No support for hashes or public keys
Original: https://github.com/dr-orlovsky/notes/blob/master/api_design.md
LOL, just wanted to open a new issue since I missed this one, but got a nice heads up from GitHub. Here's what I wanted to write:
I was thinking for some time that it might be useful to standardize at least a minimal RPC protocol for the most basic operations:
- authentication
- permissions metadata (so that an app knows what it's allowed to do, it may gray out some fields for instance)
- creating invoices (on-lightning and off-lightning)
- managing channels
- spending money (on-lightning and off-lightning)
- this is probably not minimal, but being able to work with specific UTXOs would be great for sharing seed between very different, smart-contract-enabled applications (think Bisq, Wasabi, Lighthouse, OpenBazaar) Even better allow all of them to use the same HW signer (wallet)
The current situation of each LN implementation having their own RPC is quite terrible as it causes a lot of code duplication or things like only Eclair supports Turbo channels, but RTL doesn't support Eclair, so I can't have both.
I'm not sure whether it's better to take one of the existing protocols and specify minimum features or create a new set of highly-formalized protocols. (Was thinking modeling on Rust strong type system with serde.)
End of what I intended to write.
I like your summary! Will think about it. One thing that comes to my mind is: you seem to want to avoid code generation for a good reason. Code generation has some nice benefits. Can we get the benefits of codegen AND security of less codegen? First idea that comes to my mind:
- Make generated code as simple as possible, so it can be reviewed.
- Commit the generated code to the repository and only re-generate it when necessary
- Have a tool that warns if the source file for codegen changed in order to avoid "why the hell it doesn't work?!" hair pulling (I experienced too many scenarios like this one)
Let's not forget that codegen also provides security - the long history of manually-written parses with various vulnerabilities or annoying compatibility/logic bugs should be a sufficient argument. :)
Orthogonal issue: how to connect the various services together easily? On same machine, I did the interface files proposal. I'm thinking about how to enable remote communication as well. Ideally not requiring people to configure each service separately. (That means connect to my remote electrs
and ecclair
from laptop in a single step.)
One thing that comes to my mind is: you seem to want to avoid code generation for a good reason. Code generation has some nice benefits. Can we get the benefits of codegen AND security of less codegen?
It's not only me, it's the most of the dev community in the sphere of consensus-important protocols. I've risen this question some time ago with bitcoin core; and after that with other parts of the community. Sometimes it goes up to Satoshi quote: https://bitcointalk.org/index.php?topic=632.msg7090#msg7090. But all agree to avoid codegen in all parts that are related to consensus-important parts (including P2P protocols/APIs). One of discussions you may find here: rgb-archive/spec#84. Another one is here: https://t.me/rgbtelegram/1470
all the way up to here: https://t.me/rgbtelegram/1522
However it still can be used in any client-facing APIs without any problems! But that would be vendor-specific (which does not include the fact that it can be standartized over the industry, like another LNP/BP standard). And I am up for a work to do it!
In this regard, your points are good working one. Let's try to experiment with that.
Orthogonal issue: how to connect the various services together easily? On same machine,
I am already thinking and a bit experimenting about ZMQ DLS for IPC in rust. May be can be done with simple derives, without codegen. Would be pleased to join the forces in that effort. Here is my current take on it: https://github.com/LNP-BP/lnp-node/tree/master/src/msgbus
Here is a sample of how data structure definition based on ZMQ can look like:
https://github.com/LNP-BP/lnp-node/blob/8a95459a898f595fd1087e6a338d33e64090bd0b/src/msgbus/proc/connect.rs#L22-L53
And here is RPC part:
It is without derive's yet; but they can be quite simply added; @elichai did a great crate derive-wrapper
https://github.com/elichai/derive-wrapper which he gladly can extend to cover those many From
s that are required.
Good points about consensus, I agree completely. I guess maybe if there was a special tool for that it might be feasible, but quite likely not worth the effort.
My interest of "doing something with API" is primarily about client RPC, not P2P. I think codegen should be preferred there at least for one additional reason: automated codegen across different languages.
I'm not really sure it's worth reinventing the wheel here, unless all solutions are very bad. gRPC seems to be the leader here, but there are other interesting projects. I quite like Capnproto, but didn't try it out myself yet. I also like the simplicity of just deriving serde, but that could be sub-optimal as that can't be translated to other languages.
Another interesting option is Swagger, but I'm not sure if forcing HTTP is a good idea. TBH, I'm not a huge fan of HTTP for RPC, since it adds overhead without significant value. Yes, browsers can use it, but browsers can easily use websockets translated to whatever other transport is used. I even made a tool for unix sockets: https://github.com/Kixunil/ws-unix-framed-bridge
Regarding the orthogonal issue, I didn't mean the communication protocol, but "configuration protocol". I'm currently thinking about using interface files with environment variables. Something like this: INTERFACE_LNP_BP=/etc/interfaces/lnp_bp
, but that also needs a meta file so that we know what interfaces the application can work with.
Agree on API thing.
Few comments (in general alignment with what you've said):
- gRPC (with Protobufs) are really bad with hashes/public keys. As well as all other alternative. So for LNP/BP-related stuff (meaning bitcoin, lightning and related apps) we clearly have something to develop
- Swagger is good (it's OpenAPI now, which I mentioned in the table), but: (a) no Websocket support, which is really bad (and in general REST is not working for Websockets) and (b) needed only in Web apps (not even for mobile).
In general, after a decade of REST popularity, RPC strikes back b/c of ZMQ/microservice architectures in Enterprise and Websocket/push/publication models for Web. Bitcoin and Lightning models are also not that much resourceful, rather procedural, so let's stick for RPC-like solutions leaving REST for edge cases like blockchain explorers etc.
What's exactly the problem with hashes and public keys? gRPC has bytes data type, so that should work, or am I missing something?
Yeah, let's avoid REST if possible.
It's preferred to have a fixed-length serialization for them to avoid in-flight data modifications/attacks. Also endianness play a trick sometimes with them.
In general, very good talk on data serialization design in APIs can be found here: https://developer.apple.com/videos/play/wwdc2018/222/
@Kixunil In implementing bp-node and lnp-node (bitcoin and lightning nodes, implementing things required for RGB & many other L2/L3 stuff, including DLCs, PTLC etc) I found that the priority number 1 for me (in terms of APIs) is a serialization for common data structures (primitives + bitcoin/lightning-specific) for ZMQ. Will be working on it from next week. Right now I am contemplating of writing custom Serde binary serializer as a simplest and fastest option. Do you have any considerations/other suggestions?
My original take was just to implement From
s for zmq::Message
<-> data types (I gave some samples in the comments above); but this does not work smoothly with primitives and bitcoin/lightning data types, since they are external and in rust you can't impl
external trait (like From
or TryFrom
) for an external type (which are defined by rust-bitcoin and rust-lightning libs). You can do with a wrapper, but it creates so much boilerplate code, that I assume usage of Serde (which already can do all that boilerplate code with derive
s) is the best way forward.
Actually, got a better idea than Serde. Will post solution soon.
In general, very good talk on data serialization design in APIs can be found here: https://developer.apple.com/videos/play/wwdc2018/222/
Yeah I do pretty much exactly what he's talking about, just with Results for error handling (thanks God for it, it's much more elegant than set serro and return nil)
I think it shouldn't be hard to persuade other friendly libs to accept (feature gated) serialization PRs. Worst case, serde has deserialize_with
.
Anyway, looking forward to your idea!
Sometimes rust compiler gives so much pain with generics, unlike C++...
Here is my current state of experimentation (not working yet): LNP-BP/rust-lnpbp@cf2317b
Spent the whole day... The bottom line is: any two auto trait A
implementations with generic on two distinct traits B
and C
always fail; even if all three traits are local: compiler either says "upstream may implement this trait" (if some of them is not local) or "downstream may implement this trait" if they all local. So it just not works. I'm talking about this:
LNP-BP/rust-lnpbp@cf2317b#diff-1f71c41ac92514987c842bb92e7a92cfR52-R80
I.e. this will always fail:
trait A { }
trait B { }
trait C { }
impl<T> A for T where T: B { }
impl<T> A for T where T: C { }
Even negative traits are not working as promiced: impl<T> !B for T where T: C { }
gives compiler error; however (suprisingly!) impl !B for dyn C { }
compiles without any problem, but does nothing!
And this is the case when you have marker traits that allow you to separate distinct types; otherwise it's a simple orphan trait impl problem...
Serde dealt with that using plenties of macros, including derives, basically generating wrappers for any type.
BTW, #[repr(transparent)]
are not working โ neither for generic wrappers, nor for a simple ones...
It is without derive's yet; but they can be quite simply added; @elichai did a great crate
derive-wrapper
https://github.com/elichai/derive-wrapper which he gladly can extend to cover those manyFrom
s that are required.
Actually, they are already covered: elichai/derive-wrapper#2
I think I understand what you're trying to do, but not 100% sure. There's a technique I invented using marker trait. It's used in embedded-hal
and I use a modified version of it in parse_arg
as well.
I wrote a trivial demostration of the marker trait idea for this case.
An annoying thing about it is as you can only define implementation of your trait in terms of one trait (e.g. I can't provide ParseArgUsingTryFrom
marker that'd defer to TryFrom
), but I think I have a workaround for that. Going to try it out.
Good news, I've figured it out!
It needs a few marker types, but nothing terrible. The advantages of that approach:
- Trivial impl of
Encode
for any type you'd like if that type impls some other interesting trait already. - For each new interesting trait, we only need one marker and one
impl
- If I'm not mistaken foreign crates can define their own markers and their own impls for arbitrary traits!
- Not a single macro
- Works on stable
- Not surprising, but
no_std
-compatible - You can have two different markers implementing
Encode
using the same trait slightly differently. For instance, you could somehow transform error types in one impl. (But at last, the type still has to have a single implementation ofEncode
)
In case my last point (in parentheses) bothers you, another alternative is to just parametrize Encode
by Strategy
with a default. Then it's possible to have a wrapper for forwarding. I did that in fast_fmt
But it's probably not nice in situations in which wrappers are not nice too (no experience with that). Maybe it's possible to somehow combine the two ideas, but it sounds very wild. ๐คฃ
Very good concept indeed! Thank you! Will try to apply it.
What I am trying to do is to get ability to pass data structures between processes - or through client-server APIs with binary ZMQ (both REQ/REP and PUB/SUB) at the lowest coding cost. In case of bitcoin/lightning related protocols (and RGB) it means that I can just "inherit" already existing serialization methods for bitcoin data structures (blocks, transactions and related stuff) in bitcoin wire protocol - and BOLT-related serializations; also RGB-related client-validated data serializations. So you are right, I am trying to gather under the same hood several binary serializators (already implemented elsewhere), and I am sure that each of the data structures I am using have one and only one serializator available. With this serializators, I am just construct a single ZMQ message per data structure, and for complex requests I am just assembling them into a multipart packets (feature of ZMQ).
At the end of the day I hope it will end up with simplified concept for RPC API definition: jus a rust struct with few derives
to generate required From
s. This rust code can be used as a DSL to generate a code for other languages as well in an automated fashion.
I'm very happy to hear that! According to your description, the solution looks really good.
I will need to look at ZMQ better (I have some small prior experience) to see if there's more that can be done about it.
Using Rust struct as DSL is something that I was thinking about too, but so far it feels like it'd be quite hacky. I'm still willing to give it a second look.
Good news, I've figured it out!
I started with generics in 1995, when Borland was just doin their first versions of Turbo C++ supporting generics ("templates"), and followed generic concept development through these decades... But your code scares the shit out of me :) I am trying to comprehend it
I will need to look at ZMQ better (I have some small prior experience) to see if there's more that can be done about it.
Actually ZMQ is so damn simple that there is nothing to look at. What is does is almost under the hood and does not affect data structures anyhow: ZMQ lib manages to make network communications reliable with message queues. This has no implications for the code: you are just doing usual binary sockets which simply do not fail if the remote is not there, and perform 100% async. They also support all the flavors of many-to-many communications without you noticing that.
So you may thing of ZMQ as a real async TCP or UDP (working over IPC sockets/file streams as well) where your messages can be multipart (consisting of number of packages) โ and you always know that you get the whole (multipart)message, not a part of it.
When thinking about Rust generics, you just need to think in terms of logic. The math keywords are already there for
, where
. impl
basically means "there exists exactly one". Another way to look at it is they are type-level functions (but I personally don't have this view very naturally in my head).
The trick with Helper
is a workaround for Rust not being able to recognize these impls:
impl<T> Trait1 for T where T: Trait2<Assoc=A> {}
impl<T> Trait1 for T where T: Trait2<Assoc=B> {}
do not overlap. We know that because T
can have only one impl of Trait2
, then different Assoc
implies different T
, but rustc currently doesn't understand that. Relevant issue: rust-lang/rust#20400
If rustc could understand that, there'd be no helper and we would just write:
impl<T> Encode for T where T: Into<Message> + T: EncodeUsingOtherTrait<Strategy=IntoStrategy> {
// ...
}
I hope the thing above is significantly clearer. :)
Fortunately, rustc knows that Helper<T, A>
is different than Helper<T, B>
, so we implement Encode
for the Helper
and then we implement Encode
for all types that can use Helper
with a strategy defined by the EncodeUsingOtherTrait
associated type.
Hope this helps, let me know if you need more help understanding something.
One more thing worth noting: coherence requires us to have Encode
, EncodeUsingOtherTrait
and the blanket impl in the same crate.
Another thing that I consider important: it'd be possible to just use a tuple instead of Helper
, but I think it'd be confusing for people and may be problematic if you wanted to implement Encode
for actual tuples.
Yes, this gives a very good intuition into the matter, let me meditate overnight on it
@Kixunil It has worked! It's a kind of magic ๐ฏ Thank you very much for finding a way of implementing such features!