Increase block size

[AjibolaPy]

Can the model block size be increased from 1024 to 3000 atleast?

[AdityaNG]

Yes, the block size is variable as shown below:

from kan_gpt.model import GPT
from transformers import GPT2Tokenizer

model_config = GPT.get_default_config()
model_config.model_type = "gpt2"
model_config.vocab_size = 50257
model_config.block_size = 1024
model = GPT(model_config)

Check out the get_default_config function for details

[AjibolaPy]

Thanks for your reply.
Can I increase the block size, without retraining the model?.
I think block size is same as context length(maximum sequence length) if I'm not wrong

[AdityaNG]

Interesting question @AjibolaPy , from my understanding it is possible but challenging to increase the block size without retraining your LLM (reference). And yes, block size is the same as context length