- IPv6 support
- Instrumentation
- Tokio
- Serde
- Recovery
- Producer
- Consumer
- Timeout
- Client API
- Producer
- Consumer
- Compression
- Headers
- Admin
- Transactions
- SSL
- Integration testing
- Test with different executors (std-async, tokio)
- Open Tracing
- Use crate::tracing
- Wrappers
- C#
- Java
- Learn Netty buffers and async
- F#
- Python
- Rust https://willcrichton.net/rust-api-type-patterns/
- Enforce protocol response errors checks
- Decide on error strategy. Use dynamic error?
- Migrate off
failure
crate- enum + impl Error manually (in library)
- anyhow (recommended by withoutboats)
- thiserror
- eyre
- snafu
- tokio-console
- Validate that BytesMut has always enough reserve bytes. Or switch to Vec?
- Remove ByteOrder because
bytes
already have it - Make sure that command send anr response are corresponding even in parallel scenario.
- Producer sent message size limit
- Evaluate
select!
usage for cancellation safety: https://tomaka.medium.com/a-look-back-at-asynchronous-rust-d54d63934a1c- Joshua's
Stream::merge
: https://blog.yoshuawuyts.com/futures-concurrency-3/#issues-with-futures-select
- Joshua's
- Implement BufferPool to minimize large vectors allocations
- Use crossbeam::epoch::Pointable to store list of known brokers and metadata. Allows for lock-free walks over brokers.
-
Try actors approach: https://ryhl.io/blog/actors-with-tokio/ - Test when topic partition count changes (topic re-create)
- Are topics case-sensitive?
- Tracing conventions https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md
-
async-backtrace
crate
- Try parallel-streams
- Consider
Cow<>
when deserializing. - Consider
flume
for channels - Considr
dashmap
for mutithreading access. - Consider
slotmap
https://docs.rs/slotmap/1.0.1/slotmap/ - Consider
parking_lot
for non-poisoning locks. - Consider
tinyvec
andsmolstr
for stack-optimized strings/arrays - Consider using arena: https://manishearth.github.io/blog/2021/03/15/arenas-in-rust/
- Audit that
copy
is used whenever possible, instead ofclone
- Tcp: nodelay, experiment with tx,rx buffer size
- Adaptive buffer size
- Trained dictionary for lz4
- Stats about producer/consumer buffers waiting, server throttling
- CI: do
cargo audit
- Failure model: fail up to connection reset instead of panic.
Use hashmap's raw_entry- Is unstable
- Try Async Local Executors
futures-concurrency
- Glommio and Monoio are not Send and use a "thread-per-core" design
- CLI tools
- Read CLI guideline: https://github.com/cli-guidelines/cli-guidelines/blob/main/content/_index.md
- Consider cafkacat compatible command line
- Support avro, protobuf, thrift
- Kafka mock: replay from file
- Wireshark decoder
- Compaction and record batch analyzer
- Enterprise UI
- FS adapter
- Server-side filtering
https://matklad.github.io/2020/10/15/study-of-std-io-error.html https://willcrichton.net/rust-api-type-patterns/ https://aturon.github.io/blog/2015/08/27/epoch/ https://ryhl.io/blog/actors-with-tokio/