Per layer activation
scriptandcompile opened this issue · 5 comments
How do you feel about pushing the activation function to the neural layer instead of the network?
This would allow us to add different activation functions per layer instead of using the same activation function for the entire network.
It might even be possible to eventually fit per layer parameters through the activation function this way. Not sure, but possible.
Hmm, then instead of passing one single Activation function to the entire network, we are going to assign Activation functions to each layer, right? And in this case, a NN with two layers can have two different activation function?
Looks good to me 👍 but I'm not sure if this change can improve the overall performance of training or not
Oh, I doubt it will improve the performance. It's likely it will have no effect to performance or maybe a tiny reduction in speed if that, but it's a lot more flexibility.
It should allow us the ability to make 'modules' blocks of networks; a common feature of most research papers.
#7 implements this.