Optimise the Dart code
Opened this issue · 4 comments
The Dart code should be analysed by dev tools to optimize it's performance.
There is still some work to do on the memory allocation from the fixes applied on issue 2, this should also be done on this issue.
Memory allocation issue now fixed, memory use now stable at 11.5GB across multiple queries.
How fast is dart with this? Since it is one-threaded type of language ?
Its slower than its C++ counterpart in the VM but on par if you compile the code to an executable. This is because there isn't really a lot of Dart code, all the heavy lifting is done in the ggml library which is accessed via FFI, this library starts and manages its own threads as needed(configurable) so although Dart is single threaded this app isn't as such as far a Linux is concerned.
I think the Dart code is as good as it gets without restructuring it to be me more Dart like and less C++ like although I haven't yet done an exhaustive performance analysis so this is really just guesswork.