A better practice on the same dataset produced by @SakuraUmi
Run server.py
in SakuraLLM/Sakura-13B-Galgame
python server.py \
--listen "127.0.0.1:5000" \
--model_name_or_path "SakuraLLM/Sakura-13B-LNovel-v0.8" \
--trust_remote_code \
--model_version 0.8 \
--use_gptq_model \
--no-auth