ReAIterator: A Python repository from Maykeye

ReAIterator: reiterate text file through AI

Simple script to call the text editor to edit prompt for the model. Editor is configured by $EDITOR env var(vim is default).

Options: --backend, -b Backend to use. Supported: exllama(default), exllama2, autogptq, transformers, vllm, llama_cpp --model, -m (mandatory) Path to the model. E.g. /home/user/models/llama2.gptq/gptq/model-4bit-128g.safetensors --prompt, -p (default: /tmp/prompt.ptxt) Path to write the story to --n_gens, -g (default: 4) Number of responses to generate --n_tokens, -t (default: 128) Number of tokens to generate --max_len, -x Maximum number of tokens in prompt --g_temperature, --g_repetition_penalty, --g_top_p, --g_top_k Generator parameters

Example:

$ python reAIterator.py --model ~/models/MythoMix-L2-13B-GPTQ/gptq_model-4bit-128g.safetensors --prompt /tmp/a.ptxt -g 4 -t 120 --g_temperature=0.69 --g_repetition_penalty=1.13 --g_top_p=0.95 -x 4000

$ python reAIterator.py --model ~/models/mistral-7b-v0.1.Q4_K_M.gguf --prompt /tmp/a.ptxt

Features are

It allows to mark blocks of text to be excluded from generation.
It generates 4 responses one by one to preserve precious VRAM
Responses are generated with randomized setting (each setting is +/- 5%)
It has token count threshold. After threshold is reached, script will commplain and ask to edit the prompt.
Goes with even simpler script roll.py to roll dice in another session
(Currently exLlama backend only): randomizes parameters every N tokens during single generation to shake text generation

Caveats:

vim adds EOL. Use :set noeol / add ;;;- at the end to remove it
vllm doesn't support models with headsizes not divisible by 8. openllama3b will not work

Blocks of text are separated by marker ;;; ;;;--- means exclude all further blocks ;;;- means exclude block until the next ;;;

For example you had

You are an AI. Write cool story.
### Instruction:
Write touhou story.

### Response:
(Cool adventures)
Remilia: Sakuya, lets start danmaku
[The fight starts]

And now you want to know what characters are in this setting. So you change the story to

You are an expert TTRPG game-master. Describe NPC characters.
### Instruction:
Write character description for Koakuma
(Copy paste background information for Koakuma from wiki)

### Response:
{"Koakuma":
  {"STR":;;;---
### Instruction:
Write touhou story.

### Response:
(Cool adventures)
Remilia: Sakuya, lets start danmaku
[The fight starts]

Old instruction after ;;;--- will be discarded for the next generation. After that you can use the result to describe the fight scene.

Possible TODO: [ ] parametrize finetune step and explore it so it works rather than hinder experience [ ] figure out is it possible/feasable to finetune on the go for gptq/exllama [ ] Add --remove-last-eol-because-i-am-too-lazy-to-configure-vim option
[ ] use input() instead of system, so VS code in different window can be used
[ ] nested structure.
[ ] better markers for "remove N whitespaces from the last section"

Maykeye/ReAIterator