LeNet and MNIST data
fatihtalu opened this issue · 11 comments
Dear Deniz,
The "mnist.jl" file under the data folder has been removed. Therefore, I obtained the data using MLDatasets. But when I run the LeNet application it doesn't update the weights and I see that the resulting weights are NAN.
Thanks.
Can you tell me what example you are running? The ones under the "examples", and "tutorial" folders should be updated.
The original LeNet under the "examples" uses "mnist.jl" under the "data". But it is not exist.
Because of that, I updated LeNet (attached) by using MLDatasets.
However, any learning is not taking place (see the following output). The weights always stay the same.
(:epoch, 0, :trn, 0.11217948717948718, :tst, 0.11338141025641026)
12.182926 seconds (1.62 M allocations: 263.846 MiB, 14.39% gc time)
(:epoch, 1, :trn, 0.11231303418803419, :tst, 0.11298076923076923)
11.094744 seconds (1.47 M allocations: 256.223 MiB, 13.59% gc time)
(:epoch, 2, :trn, 0.11227964743589744, :tst, 0.11328125)
10.550422 seconds (1.47 M allocations: 255.971 MiB, 13.62% gc time)
(:epoch, 3, :trn, 0.11227964743589744, :tst, 0.11288060897435898)
Thanks for the catch: I updated lenet.jl but apparently forgot lenet.ipynb. Just delete the line that includes "mnist.jl" it should work. Let me know if it doesn't.
Dear Deniz,
When I checked your Github site, I could not see the current version of the LeNet application. My request from you is to put the current working version on your Github site.
Because unfortunately, when I did what you said (deleted the mnist line) it still didn't work. I think the accuracy function is updated and the classic Lenet program uses the old accuracy function.
Thank you for your support and wish you good work.
I fixed it in the latest master, please test.
Dear Deniz,
I just downloaded your code (master) from GitHub and tested it.
The code works fine, but unfortunately learning does not occur.
Accuracy values are not improving.
The output is following as:
opts=(:batchsize, 128)(:fast, false)(:iters, 9223372036854775807)(:atype, "KnetArray{Float32,N} where N")(:epochs, 3)(:gcheck, 0)(:lr, 0.1)(:seed, -1)
(:epoch, 0, :trn, 0.11239650106837606, :tst, 0.11348157051282051)
34.803004 seconds (26.22 M allocations: 1.507 GiB, 1.21% gc time)
(:epoch, 1, :trn, 0.11239650106837606, :tst, 0.11348157051282051)
7.044556 seconds (1.45 M allocations: 255.612 MiB, 1.39% gc time)
(:epoch, 2, :trn, 0.11224626068376069, :tst, 0.11348157051282051)
6.522182 seconds (1.44 M allocations: 255.496 MiB, 0.98% gc time)
(:epoch, 3, :trn, 0.11239650106837606, :tst, 0.11348157051282051)
Thanks.
Dear Deniz,
I have some suggestions for quick training of deep networks. I want to use Julia-Knet, not Python for this. But first of all I have to run the Lenet-MNIST application fully. Not updating the accuracy value is a major problem. Although I changed parameters such as learning rate, it did not improve. Are you facing similar problem?
Thanks
Dear Deniz,
In this case, I have to say goodbye to Julia-Knet now.
Because I can't even classify MNIST with a simple CNN architecture (LeNet).
Unfortunately, accuracy is not improving, it stays in place.
Thanks for your help so far.
@fatihtalu I can not replicate this issue on Julia v1.5.1 and Knet#master. Are you sure that you installed Knet master code (with pkg> add Knet#master) ?
Here is my output with a fresh installation:
julia> include(Knet.dir("examples/lenet/lenet.jl"))
julia> LeNet.main()
lenet.jl (c) Deniz Yuret, 2016. The LeNet model on the MNIST handwritten digit recognition problem from http://yann.lecun.com/exdb/mnist.
opts=(:batchsize, 128)(:fast, false)(:iters, 9223372036854775807)(:atype, "Array{Float32,N} where N")(:epochs, 3)(:gcheck, 0)(:lr, 0.1)(:seed, -1)
┌ Warning: CPU conv4 support is experimental and very slow.
└ @ Main.LeNet ~/.julia/packages/Knet/0arKM/examples/lenet/lenet.jl:90
This program has requested access to the data dependency MNIST.
which is not currently installed. It can be installed automatically, and you will not see this message again.
Dataset: THE MNIST DATABASE of handwritten digits
Authors: Yann LeCun, Corinna Cortes, Christopher J.C. Burges
Website: http://yann.lecun.com/exdb/mnist/
[LeCun et al., 1998a]
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.
"Gradient-based learning applied to document recognition."
Proceedings of the IEEE, 86(11):2278-2324, November 1998
The files are available for download at the offical
website linked above. Note that using the data
responsibly and respecting copyright remains your
responsibility. The authors of MNIST aren't really
explicit about any terms of use, so please read the
website to make sure you want to download the
dataset.
Do you want to download the dataset from ["http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz", "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz", "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz", "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz"] to "/Users/akyurek/.julia/datadeps/MNIST"?
[y/n]
y
(:epoch, 0, :trn, 0.12111044337606838, :tst, 0.11868990384615384)
39.640590 seconds (20.66 M allocations: 20.812 GiB, 9.19% gc time)
(:epoch, 1, :trn, 0.9582999465811965, :tst, 0.9610376602564102)
The problem is most likely a GPU installation issue with @fatihtalu -- you can test this either by using another GPU machine from e.g. Amazon, Google Colab etc., or by testing on your machine with the cpu (you can use the --atype Array{Float32} option). Here is my output:
julia> include(Knet.dir("examples/lenet/lenet.jl"))
julia> LeNet.main("--atype Array{Float32}")
lenet.jl (c) Deniz Yuret, 2016. The LeNet model on the MNIST handwritten digit recognition problem from http://yann.lecun.com/exdb/mnist.
opts=(:batchsize, 128)(:fast, false)(:iters, 9223372036854775807)(:atype, "Array{Float32}")(:epochs, 3)(:gcheck, 0)(:lr, 0.1)(:seed, -1)
┌ Warning: CPU conv4 support is experimental and very slow.
└ @ Main.LeNet /userfiles/dyuret/.julia/dev/Knet/examples/lenet/lenet.jl:90
(:epoch, 0, :trn, 0.07495325854700854, :tst, 0.07271634615384616)
76.761784 seconds (20.04 M allocations: 21.032 GiB, 9.24% gc time, 15.78% compilation time)
(:epoch, 1, :trn, 0.9612546741452992, :tst, 0.9651442307692307)
Dear Deniz and Ekin
Thank you for your answers.
Deniz is right. The problem is solved when I assign Array{Float32} to the atype variable. Working on the CPU is enough for me.
I can pick up where I left off in Julia-Knet.
Thank you.