Custom Tokenizer
Opened this issue ยท 0 comments
tomassosorio commented
๐ Feature
Allow the user to add a custom Tokenizer with minor implementation.
Motivation
It is impossible to integrate every tool, but that does not mean that it shouldn't exist a CustomTokenizer that allows the user to use its own implementation.
Needs: callable function or object
Warnings: If Compose.create_from_steps
should throw an error since it cannot load its implementation