lucidrains/x-transformers

Issues

• Correct interaction between CLS token and RoPE

  #249 opened 10 days ago by oasidorshin
  5

• Question: problem with xval implementation

  #248 opened 21 days ago by HarshaSatyavardhan
  5

• [Bug] XL-recurrence with AlibiPositionalBias and mems not working correctly

  #242 opened 19 days ago by pfeatherstone
  17

• Confusion about image->caption example

  #239 opened 3 months ago by mtran14
  1

• [Bug] Error when `rotary_pos_emb` set to True in cross attention

  #247 opened a month ago by BakerBunker
  3

• Was it a clerical error ? ScaleNorm.g init form dim ** -0.5. I think it should be dim ** 0.5

  #246 opened a month ago by junphine
  1

• [Question] very small attention scores

  #245 opened 3 months ago by pfeatherstone
  7

• High memory usage compared to Huggingface and Autocast has no effect?

  #197 opened 7 months ago by LarsHill
  3

• Question: rotary embeddings and bad length extrapolation

  #241 opened 3 months ago by pfeatherstone
  1

• How can I add custom attention masks to a Decoder?

  #240 opened 3 months ago by DerEchteFeuerpfeil
  3

• Generation for PaLI?

  #238 opened 3 months ago by BurgerAndreas
  0

• `layer_mem` is unbound (when called from `ContinuousTransformerWrapper`)

  #237 opened 3 months ago by amitkparekh
  6

• how to set inputs to the right shape

  #218 opened 5 months ago by emadkavousi
  1

• RotaryEmbedding XPOS doesn't work with mems

  #233 opened 4 months ago by pfeatherstone
  5

• Question: How to implement rel_pos_bias in cross_attention?

  #229 opened 4 months ago by alexdemartos
  11

• [Minor; noob question] Uniform distribution instead of normal

  #232 opened 4 months ago by p0p4k
  0

• How to build optimizer

  #230 opened 4 months ago by pfeatherstone
  9

• Simplifying Transformer Blocks (https://arxiv.org/abs/2311.01906)

  #214 opened 5 months ago by Froskekongen
  9

• attn_num_mem_kv > 0 and attn_one_kv_head = True error

  #228 opened 4 months ago by pfeatherstone
  8

• Adding memmask to ContinuousTransformerWrapper

  #227 opened 4 months ago by pfeatherstone
  3

• XL-recurrence with RotaryEmbedding and mems not working correctly.

  #223 opened 4 months ago by pfeatherstone
  34

• Removed biases breaks pre-trained models

  #225 opened 4 months ago by zqevans
  5

• Seq len missing in rotary embedding

  #226 opened 4 months ago by raganato
  3

• "Stabilizing Transformer Training by Preventing Attention Entropy Collapse" improvement to ViT

  #217 opened 4 months ago by catid
  1

• Enhancement: Multi Input/Output transformers

  #222 opened 4 months ago by RyanKim17920
  1

• Question: How to load model trained on earlier version of x-transformers

  #221 opened 4 months ago by tmphex
  3

• kv cache breaks generation

  #219 opened 5 months ago by ad8e
  5

• Question: num_memory_tokens > 0 and return_mems = True

  #216 opened 5 months ago by pfeatherstone
  3

• Support for NormSoftmax

  #215 opened 5 months ago by catid
  16

• Bert token type embedding

  #213 opened 6 months ago by eyalmazuz
  2

• ONNX export failed

  #212 opened 6 months ago by pfeatherstone
  14

• rotary embedding issues when training in mixed precision

  #210 opened 6 months ago by zqevans
  2

• Masking for prepend_embeds

  #211 opened 6 months ago by zqevans
  7

• Question: masking in token shifting

  #208 opened 6 months ago by pfeatherstone
  1

• [Bug] ContinuousTransformerWrapper - return_mems doens't work

  #209 opened 6 months ago by pfeatherstone
  1

• Do you consider adding rwkv

  #207 opened 6 months ago by chaodreaming
  1

• Question: return_mems and rotary_pos_emb

  #205 opened 6 months ago by pfeatherstone
  5

• Using Rotary Positional Encoding with Continuous Wrapper

  #203 opened 6 months ago by anthonyzhou-1
  1

• pre_norm_has_final_norm kwarg not used

  #202 opened 6 months ago by sashakunitsyn
  1

• Transformer Goat

  #201 opened 7 months ago by darkman111a
  1

• Question: attn_head_scale with use_scalenorm

  #200 opened 7 months ago by pfeatherstone
  1

• Feature request: different normalization layers at different depths

  #199 opened 7 months ago by pfeatherstone
  0

• Lack of the deep_norm variants of transformer

  #198 opened 7 months ago by ZegangC
  1

• [Question] difference between num_mem_kv and num_memory_tokens

  #193 opened 7 months ago by pfeatherstone
  7

• ContinuousTransformer num_memory_tokens bug

  #194 opened 7 months ago by pfeatherstone
  1

• [Feature request] support num_memory_tokens in ContinuousTransformerWrapper

  #192 opened 7 months ago by pfeatherstone
  7

• Issue with torch.compile

  #189 opened 8 months ago by scopello
  4

• Issue with qk scale between flash attention and normal attention when qk_norm is True

  #190 opened 8 months ago by mistycube
  2

• Feature request: add local and reformer

  #188 opened 8 months ago by samvanstroud
  1

• Bugs in generation with cache and seq_start_pos

  #187 opened 8 months ago by LouChao98
  8