denizyuret/AutoGrad.jl

grad doesn't work on trivial identity

tscode opened this issue · 6 comments

tscode commented

I am on the official julia 1.0.1 release with all packages up-to-date.
The following returns nothing,

using AutoGrad
y = Param(1.)
f(x) = x
d = @diff f(y)
grad(d, y) # returns nothing

while this works:

f(x) = 1x
d = @diff f(y)
grad(d, y) # returns 1.
tscode commented

Okay I found another problem with broadcasting of user-defined functions that may be related:

y = Param([1.,2.])
f(x) = 1x
d = @diff sum(f.(y))
grad(d, y) # returns nothing

In contrast, it works with predefined functions:

d = @diff sum(sin.(y))
grad(d, y) # returns something reasonable
denizyuret commented
tscode commented

Okay, but I expect the gradient of the identity x -> x to be 1, and not 0. So it should not return nothing, if I understood your comment correctly?

denizyuret commented
denizyuret commented

The first example failed because I did not handle the case where the differentiated function just returns one of the parameter inputs. It should be fixed in the latest master.

The second example failed because AutoGrad does not yet support broadcasting arbitrary functions, only some primitives. I added code that throws an error reflecting this instead of silently failing.

denizyuret commented

The issue of supporting custom broadcasting also came up in Knet: denizyuret/Knet.jl#342

I am leaving this issue open: custom broadcasting used to work in Julia 0.6, maybe we can find a simple fix for 1.0 as well.