tanh and its derivative are both wrong

[felipetavares]

This is what you did (red should be tanh, blue should be its derivative):

This is what it should be (green is tanh, purple is its derivative):

The wrong part is this:

(BoundNumbers.Exp(input * 2.0) - 1) / (BoundNumbers.Exp(input * 2.0 + 1))

It should be

(BoundNumbers.Exp(input) - BoundNumbers.Exp(-input)) / (BoundNumbers.Exp(input) + BoundNumbers.Exp(-input))

The derivative calculation is right, but since it uses a wrong definition of tanh it produces a wrong result too (which isn't even the derivative of the described function, this will cause errors in any gradient-based method since they rely on correct derivatives to learn).

[carlosribeiro1987]

Thanks man. I'll fix these issues.

[carlosribeiro1987]

The suggested changes were made on the code. I'm closing this issue.