openai/automated-interpretability

Problem about activation calculation

Daftstone opened this issue · 3 comments

Daftstone commented

I would like to know how neuron activation is calculated and how to map neuron activation to each input token. Or can you provide me with related work on calculating neuron activation, I would be very grateful.

JacksonWuxs commented

Yes, I have the same question regarding to the calculation of token-level activations.
It is not clear in both the paper and code.
If anyone could give some hints, I would also be very grateful.

JacksonWuxs commented

Dear authors,

I found that this section provides the definition of neuron-token-level connection weights. First, I want to confirm if the word-neuron activation is extracted based on this section. I am confused because it seems that this activation does not take into account the context information. Specifically, according to the equation h{l}.mlp.c_proj.w[:, n, :] @ diag(ln_f.g) @ wte[t, :], the output weight of a neuron (l, n) to the token t appears to be independent of other tokens.

I would greatly appreciate it if someone could address my confusion and provide clarification on this matter.

Best,
Xuansheng

WuTheFWasThat commented

yes, that's right - it doesn't take context information into account. it would probably be better to use something activation instead of weight based

  • 👍2