bfloat16 Support
tarlanahad opened this issue · 4 comments
Anyway to use the model with bfloat16 data type?
Most RTNeural layers are agnostic to the input data type. Although they're mostly designed to work with floating-point data types, it's also 100% possible to use most RTNeural layers with types like int or std::complex<T> (although an RTNeural::TanhActivationT<int> might not be all that useful).
I haven't tried using RTNeural yet with a bfloat16 data type, but I imagine it should work, provided that the backend being used with RTNeural also supports bfloat16. I don't believe XSIMD has bfloat16 support. Eigen supports bfloat16 as of version 3.4, so that should work in RTNeural (RTNeural uses Eigen version 3.4.90). If you're using the STL backend, C++23 is supposed to have support for bfloat16, but I'm not sure what compilers support that feature at the moment.
We should probably add some tests to the RTNeural test suite, to explicitly make sure that bfloat16 is working with the available layers, and see if there are any potential issues there. I should have some time to work on that next week. In the meantime, if you already have a use-case for bfloat16, I'd be curious to hear if you run into issues (or if it "just works")!
Tried it doesn't seem to work with std:: bfloat16_t. It needs some of the parts to be statically casted, which solved only some parts of the issue. That would be really nice if you could have a quick look at it.
So the answer is, it doesn't work.
Thanks for the update!
Would it be possible to share a compiler log with any errors messages that may have resulted.
It would also be nice to know the compiler and RTNeural backend that you were using in your test.
If you already have a public repo or something similar where you've been doing your testing, that would be very useful as well!