Improve the proto-compiler to generate canonical rust files
Closed this issue · 3 comments
Crate
ibc-proto & ibc-proto-compiler
Summary
Improve the proto-compiler to serve as a means for generating canonical Rust files by compiling SDK & IBC proto files.
Problem Definition
- The rust files generated using SDK and IBC protos are not usable (i.e. they don't compile) out-of-the box and require a fix, see here.
- Other Rust projects in the ecosystem might not be able to use these pre-built/patched files because the ibc-proto-compiler crate explicitly disables gRPC server code generation ->
https://github.com/informalsystems/ibc-rs/blob/26087d575c620d1ec57b3343d1aaf5afd1db72d5/proto-compiler/src/cmd/compile.rs#L104-L106 - This also means that such projects are unable to benefit from ibc-rs trait implementations involving the generated raw types (e.g.
TryFrom<RawType> for DomainType,From<DomainType> for RawType, etc.).
Proposal
Here's one possible solution to address these issues:
- ibc-proto-compiler could use a rudimentary patch management system that automatically patches the generated files based on predefined rules. This should be relatively straightforward to implement using git and patch files, see quilt for a similar tool that is used in embedded linux build systems. (#78)
- ibc-proto must contain generated server code.
Acceptance Criteria
ibc-proto files are usable by other ecosystem projects.
For Admin Use
- Not duplicate issue
- Appropriate labels applied
- Appropriate milestone (priority) applied
- Appropriate contributors tagged
- Contributor assigned/self-assigned
- The rust files generated using SDK and IBC protos are not usable (i.e. they don't compile) out-of-the box and require a fix, see here.
This problem is now tracked in the SDK cosmos/cosmos-sdk#10323
One more serious problem we're seeing is that the output of proto compiler (i.e., the Rust-generated files) is not deterministic. Soares first spotted this problem while working on no_std. Mentioned in informalsystems/hermes#1439
Add ci/sync-protobuf.sh script to reliably re-generate the protobuf Rust definitions. Note however that the current protobuf-compiler is non-reproducible, so multiple calls to ci/sync-protobuf.sh may yield different results.
This is a problem because it makes reviewing difficult. The diff contains many changes, but most of the diff changes are irrelevant, spawning from re-ordering of types in the generated Rust files. For an example see this.
We don't have a solution to this yet. One idea suggested was that the problem is caused by tonic or prost using HashSet internally, which has non-deterministic ordering of elements, so the order of enums/structs/fns in the proto-compiler output is non-deterministic; the solution would be to replace the HashSet with something else. (@soareschen)