ndif-team/nnsight

Is it possible to apply layer modules out-of-order?

arjunguha opened this issue · 2 comments

arjunguha commented

This is a reduced version of what I'm trying: https://gist.github.com/arjunguha/46c0557ee0323f748eddbbce1981c5a5

Using (Code) Llama, I'm applying modules out of order (http://nnsight.net/notebooks/features/modules/). However, I'm trying to apply a decoder layer out of order.

Is this supported?

JadenFiotto-Kaufman commented

@arjunguha Okay I see what the issue is here. In the meantime I believe if you were to set validate=False it would be a workaround for now.

  • 🚀1
JadenFiotto-Kaufman commented

@arjunguha Could you give this a try on the dev branch? I think I've fixed it. However I now get:

  File "/share/u/jadenfk/miniconda3/envs/ndif/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 134, in forward
    inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
AttributeError: 'NoneType' object has no attribute 'shape'

I think you need to explicitly pass in a position_ids keyword argument when calling the module.