Floating point variable-length encoding is optimized for values like 0.000000000000000000000000000000000000000000014

[cormac-ainc]

I'm coming at this from the perspective of writing a musli-wire format for a very float-heavy data structure. As a primary observation, floats in musli-wire are encoded by bitcasting them to u32, and then encoding that integer using the chosen integer encoding (fixed or variable, default variable).

Variable-length-int-encoding for floats is pretty useless. If you bitcast 1.0f32 to a u32, you get 1065353216. This takes 4 bytes to encode in variable encoding, resulting in [68, 63, 128, 0, 0]. The circumstances under which you get an int representation small enough to make use of variable length encoding are not obvious without studying IEEE754, and furthermore it is not a useful set of values that gets optimised: f32::from_bits(10) prints as 0.000000000000000000000000000000000000000000014. It doesn't help with "small integral floats" or even something like "values in a small range with 8-bit quantization". It only helps with the really really small non-negative floats, and +0.0. If you have floats that close to zero at ANY time, you are probably using floats wrong and about to get an enormous floating point error from your calculations using it. The wire format should not be optimized for those danger-zone floats. (Check out Herbie to see this charted visually: example where values of x close to 0 produce extremely inaccurate results.)

I suggest:

1. Add a type param to musli-wire's Encoding for floating point encoding, defaulting it to the chosen (or default) integer encoding, and therefore allowing independent selection of fixed/variable for ints, lengths, and floats.
2. Optionally extend that with a new VariableFloat encoding that for lossless conversion to f32/f24/f16/f8. This is probably a lot of work. There's a C++ library called vf128 that does something in this vein: https://github.com/michaeljclark/vf128. I also think I've seen one of the serde crates doing this, can't remember which.

I don't personally need the latter.

Finally, because creating a musli-wire Encoding isn't very ergonomic, and in this issue + #23 I am suggesting a lot more type params on Encoding, I suggest adding an impl of Default for Encoding such that you can write:

pub const WIRE_ENCODING: musli_wire::Encoding<
    musli::mode::DefaultMode,
    musli_wire::int::Fixed,
    musli_wire::int::FixedUsize,
    ...
> = musli_wire::Encoding::default();

instead of duplicating all your choices between the type params and the builder methods.

[udoprog]

Float encoding isn't something I focused on at all so far, so I'm glad you opened as series of issues about it. We'll definitely want to add another type parameter for dealing with it, maybe even making it externally configurable since there might be a lot of complexity involved.

Finally, because creating a musli-wire Encoding isn't very ergonomic

I'm experimenting with building a macro to make it better, because I want to offer more flexibility as well without exposing the large number of type parameters. So don't worry too much about the parameters for now. It'll allow for both building an encoding implementation and a type which encapsulates it.

[AaronKutch]

The problem with float encoding is that most intermediate calculations and even most literals (e.x. when something as simple as 0.1 is converted from base 10 to base 2, it ends up with repeating digits and irregular changes at the beginning and end due to rounding) will fill the entire mantissa with practically incompressible bits. Detecting when a floating point input is equivalent to the round-to-even version of a short base 10 string would be a performance nightmare that probably is never going to be practical for any real time formats, only for very optimized storage. For lossy floats (do not do this for the default, only as a special option that people have to configure for their use case), you would need to decompose into exponent, sign, mantissa, handle NaN and infinity types, etc. and decide on some kind of storage scheme that probably will not get more than a 2x average improvement for most use cases.

I do have https://docs.rs/awint/0.10.0/awint/struct.FP.html which can convert floats to any fixed width format and back, but there are a lot of caveats

[AaronKutch]

Except for one thing I just thought of, values of exactly 0 and some class of fixed point numbers will be very common. What we could do is have a scheme where we multiplex if the unbiased exponent is some value (e.x. the value right before NaNs/infinity which will rarely be encountered). If the exponent is not that value, we just interpret the bits as-is as the floating point number for the fast and most general case, otherwise we reinterpret as a small fixed point number and convert that to its floating point number, and also leave another bit for the rare case in which the special exponent value is what is actually needed, in which we need another byte to complete the encoding

[cormac-ainc]

Yes, floats are wonderful but awfully resistant to things like this. Your library looks pretty cool, but I don't think I would use a smart float encoding in musli even if there were one. Anyone who's worried about floating point size on the wire/disk and wants to make tradeoffs for it should consider using f16 (ie https://docs.rs/half), just as those big LLM weights files do. I mainly care about the waste of time doing variable integer encoding on the float bits for no material gains in the size department, not the waste of bytes on the wire. The size wins are clearly not worth it with the integer encoding, and as you say, given that my floats are not human-generated (like perhaps numbers in an excel spreadsheet would be), the likelihood there's any compression scheme that would help them is pretty low.

I detailed in #25 how I want to encode floats -- big flat arrays of 'em. The kind you can decode very quickly with vectorised instructions, not itty bitty one-byte-at-a-time stuff. Maybe a variable float would score you a couple of % on an object like { "document": "e3bc73e690", "score": 0.00773 } but the benefits will not be particularly real.

[AaronKutch]

oh, ok I misunderstood the angle you were coming from