/llm_surprisal

Simple tool for generating tokens with open source transformers and/or calculate per-token surprisal.

Primary LanguagePython