Will the whole mixture activation require OP/Fl（linear, convolution, or attention）?
LHQUer opened this issue · 2 comments
As the given Figure 3 in the paper shown, while through the Scatter operation, we will get the whole mixture activation including old and new activation of patch. And the Figure 3 show that the mixture activation will perform OP/Fl in the next step. If that's the case, then isn't it true that each device in each layer performs OP/Fl on the entire activation? Isn't this enough to reduce the computational load on each device? If the old activation of the patch does not participate in the OP/Fl computation, how can new and old activation be fused and learned?
Perhaps I don't quite understand this Figure 3 and how to achieve activation interaction while reducing the computational load of each device. I hope to receive guidance. Thank you very much.
This is an illustration for better understanding. Only a minor portion of stale activation gets involved in the computation. For example, if Fl is a convolutional layer, then only the boundaries of the stale activations get involved in the computation. The majority part does not even instantiate. You can refer to our code for more details.