Implement dropout layer

Question

Implement dropout layer

milancurcic opened this issue 2 years ago · 17 comments

milancurcic commented 2 years ago

References:

Answer 1 · 2024-12-24T21:39:04.000Z

Hello, i'd like to work on this

Answer 2 · 2024-12-26T23:26:03.000Z

Hi @ricor07, that'd be fantastic, thank you!

To start, take a look at how a concrete layer type extends the abstract base layer, for example nf_dense_layer. Then follow the same pattern by defining a nf_dropout_layer type in src/nf/nf_dropout_layer.f90.
nf_dropout_layer will need the init, forward, and backward methods.
In nf_layer_constructors, you'll need to add a constructor function dropout that returns an instance of nf_dropout_layer type. You can model this function based on the corresponding counterparts in Keras and PyTorch.
Create a test program in test/test_dropout_layer.f90. We'd first want to ensure that the dropout layer behaves the way we expect (e.g. creating a dropout layer given inputs should have a specific known shape etc.).
Once we're happy with the above, we can proceed to add the forward and backward invocations of dropout in the network % forward and network % backward implementations in nf_network_submodule.

Let me know if you have any questions. If you'd like, we could also meet on video to discuss this after January 6.

Answer 3 · 2024-12-27T09:35:59.000Z

Thank you for your kindness. I'll start to work.

Answer 4 · 2024-12-27T21:08:50.000Z

@milancurcic Sorry for disturbing you, but i can't compile the repo. I downloaded fpm and gfortran, i included them in the path as well. After running fpm build --profile release, this is what I get:

`PS C:\Users\angel\neural-fortran> fpm build --profile release

mkdir build\dependencies
nf_random.f90 done.
nf_optimizers.f90 done.
nf_parallel.f90 done.
nf_base_layer.f90 done.
nf_metrics.f90 done.
nf_activation.f90 done.
nf_datasets_mnist.f90 done.
nf_datasets.f90 done.
nf_io_binary.f90 done.
nf_layer.f90 failed.
nf_input3d_layer.f90 failed.
nf_dense_layer.f90 failed.
nf_input1d_layer.f90 failed.
nf_datasets_submodule.f90 failed.
nf_datasets_mnist_submodule.f90 failed.
nf_flatten_layer.f90 failed.
nf_conv2d_layer.f90 failed.
[ 33%] Compiling...
f951.exe: Fatal Error: Reading module 'nf_activation.mod' at line 22 column 34: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'build\gfortran_2654F75F5833692A/nf_datasets_mnist.smod' at line 63 column 62: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'build\gfortran_2654F75F5833692A/nf_datasets.smod' at line 41 column 56: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'nf_activation.mod' at line 22 column 34: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'build\gfortran_2654F75F5833692A/nf_base_layer.mod' at line 17 column 21: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'build\gfortran_2654F75F5833692A/nf_base_layer.mod' at line 17 column 21: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'build\gfortran_2654F75F5833692A/nf_base_layer.mod' at line 17 column 21: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'build\gfortran_2654F75F5833692A/nf_base_layer.mod' at line 17 column 21: Unexpected EOF
compilation terminated.
Compilation failed for object " src_nf_nf_conv2d_layer.f90.o "
Compilation failed for object " src_nf_nf_datasets_mnist_submodule.f90.o "
Compilation failed for object " src_nf_nf_datasets_submodule.f90.o "
Compilation failed for object " src_nf_nf_dense_layer.f90.o "
Compilation failed for object " src_nf_nf_flatten_layer.f90.o "
Compilation failed for object " src_nf_nf_input1d_layer.f90.o "
Compilation failed for object " src_nf_nf_input3d_layer.f90.o "
Compilation failed for object " src_nf_nf_layer.f90.o "
stopping due to failed compilation
STOP 1
PS C:\Users\angel\neural-fortran> fpm build --profile release
nf_datasets_mnist_submodule.f90 failed.
nf_input1d_layer.f90 failed.
nf_conv2d_layer.f90 failed.
nf_datasets_submodule.f90 failed.
nf_dense_layer.f90 failed.
nf_input3d_layer.f90 failed.
nf_flatten_layer.f90 failed.
nf_layer.f90 failed.
[ 19%] Compiling...
f951.exe: Fatal Error: Reading module 'nf_activation.mod' at line 22 column 34: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'build\gfortran_2654F75F5833692A/nf_datasets_mnist.smod' at line 63 column 62: Unexpected EOF
compilation terminated.
f951.exe: Fatal Error: Reading module 'build\gfortran_2654F75F5833692A/nf_datasets.smod' at line 41 column 56: Unexpected EOF
compilation terminated.`

Could you help me fixing this issue?

Answer 5 · 2024-12-27T21:17:53.000Z

Not clear to me yet why this happens. May be Windows-specific. What version of gfortran is this?

Answer 6 · 2024-12-27T21:35:16.000Z

14.2.0. I think it is the last one

Answer 7 · 2024-12-27T22:01:44.000Z

OK, I just built and tested main with gfortran-14.2.0, albeit on Linux, with no issues.

Just to be sure, can you try removing the build directory and running fpm build again?

If the issue still appears the same way, let's try building with CMake.

Answer 8 · 2024-12-27T22:08:40.000Z

Yes, already done, the issue persists.
Before trying FPM, i already tried Cmake.

PS C:\Users\angel\neural-fortran\build> cd nf
PS C:\Users\angel\neural-fortran\build\nf> cmake ..
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631.
-- The Fortran compiler identification is unknown
-- Detecting Fortran compiler ABI info
CMake Error: Generator: execution of make failed. Make command was: devenv.com CMAKE_TRY_COMPILE.sln /build Debug /project cmTC_9ce8e
-- Detecting Fortran compiler ABI info - failed
-- Check for working Fortran compiler: C:/msys64/mingw64/bin/gfortran.exe
CMake Error: Generator: execution of make failed. Make command was: devenv.com CMAKE_TRY_COMPILE.sln /build Debug /project cmTC_3566c
-- Check for working Fortran compiler: C:/msys64/mingw64/bin/gfortran.exe - broken
CMake Error at C:/Program Files/CMake/share/cmake-3.31/Modules/CMakeTestFortranCompiler.cmake:59 (message):
The Fortran compiler

"C:/msys64/mingw64/bin/gfortran.exe"

is not able to compile a simple test program.

It fails with the following output:

Change Dir: 'C:/Users/angel/neural-fortran/build/CMakeFiles/CMakeScratch/TryCompile-zciw85'

Run Build Command(s): devenv.com CMAKE_TRY_COMPILE.sln /build Debug /project cmTC_3566c
no such file or directory
Generator: execution of make failed. Make command was: devenv.com CMAKE_TRY_COMPILE.sln /build Debug /project cmTC_3566c

CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:9 (project)

Don't mind the nf in the first line, the output is the same even without it

Answer 9 · 2024-12-27T22:17:28.000Z

OK, cool, thanks. I think this points toward the mingw or its gfortran distribution, or how they're configured. I'm not familiar with compiling on Windows or mingw but I know others who are and we can ask.

When you get a chance, can you try compiling a simple hello world program, and then a module source file, with gfortran directly?

Answer 10 · 2024-12-27T22:22:33.000Z

Sure. I'll do it tomorrow morning (I think 5.00 am in Miami). Are you active also on Saturdays?

Answer 11 · 2024-12-28T03:44:28.000Z

No rush and whenever it's convenient for you. I'm off until January 6 but read email and can respond on most days.

Answer 12 · 2024-12-28T10:38:53.000Z

Hello. The hello world program and the module source file run perfectly. I even tried building with cmake again but it stops reading some code chunks, as i posted before with fpm. These are the logs i get with the make command:

[ 86%] Building Fortran object example/CMakeFiles/dense_mnist.dir/dense_mnist.f90.obj
C:\Users\angel\neural-fortran\example\dense_mnist.f90:3:74:

3 |   use nf, only: dense, input, network, sgd, label_digits, load_mnist, corr
  |                                                                          1

Error: Unexpected USE statement at (1)
C:\Users\angel\neural-fortran\example\dense_mnist.f90:5:15:

5 |   implicit none
  |               1

Error: Unexpected IMPLICIT NONE statement at (1)
C:\Users\angel\neural-fortran\example\dense_mnist.f90:7:16:

7 |   type(network) :: net
  |                1

Error: Derived type 'network' at (1) is being used before it is defined
C:\Users\angel\neural-fortran\example\dense_mnist.f90:8:63:

8 |   real, allocatable :: training_images(:,:), training_labels(:)
  |                                                               1

Error: Unexpected data declaration statement at (1)
C:\Users\angel\neural-fortran\example\dense_mnist.f90:9:67:

9 |   real, allocatable :: validation_images(:,:), validation_labels(:)
  |                                                                   1

Error: Unexpected data declaration statement at (1)
C:\Users\angel\neural-fortran\example\dense_mnist.f90:10:26:

10 | integer :: n, num_epochs
| 1
Error: Unexpected data declaration statement at (1)
C:\Users\angel\neural-fortran\example\dense_mnist.f90:25:10:

25 | call net % print_info()
| 1
Error: VARIABLE attribute of 'net' conflicts with PROCEDURE attribute at (1)
C:\Users\angel\neural-fortran\example\dense_mnist.f90:32:12:

32 | call net % train( &
| 1
Error: VARIABLE attribute of 'net' conflicts with PROCEDURE attribute at (1)
C:\Users\angel\neural-fortran\example\dense_mnist.f90:44:29:

44 | output_metrics = net % evaluate(validation_images, label_digits(validation_labels), metric=corr())
| 1
Error: Symbol 'net' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:56:18:

18 | net = network([ &
| 2
......
56 | type(network), intent(in out) :: net
| 1
Error: Type name 'network' at (1) conflicts with previously declared entity at (2), which has the same name
C:\Users\angel\neural-fortran\example\dense_mnist.f90:61:27:

61 | if (all(maxloc(net % predict(x(:,i))) == maxloc(y(:,i)))) then
| 1
Error: Symbol 'net' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:63:9:

63 | end if
| 1
Error: Expecting END DO statement at (1)
C:\Users\angel\neural-fortran\example\dense_mnist.f90:55:28:

55 | real function accuracy(net, x, y)
| 1
Error: Symbol 'net' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:30:14:

30 | epochs: do n = 1, num_epochs
| 1
Error: Symbol 'n' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:18:5:

18 | net = network([ &
| 1
Error: Symbol 'net' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:23:12:

23 | num_epochs = 10
| 1
Error: Symbol 'num_epochs' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:12:33:

12 | call load_mnist(training_images, training_labels, &
| 1
Error: Symbol 'training_images' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:12:50:

12 | call load_mnist(training_images, training_labels, &
| 1
Error: Symbol 'training_labels' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:13:35:

13 | validation_images, validation_labels)
| 1
Error: Symbol 'validation_images' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:13:54:

13 | validation_images, validation_labels)
| 1
Error: Symbol 'validation_labels' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:12:33:

12 | call load_mnist(training_images, training_labels, &
| 1
Error: Symbol 'training_images' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:18:17:

18 | net = network([ &
| 1
Error: Function 'input' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:19:15:

19 | input(784), &
| 1
Error: Function 'dense' at (1) has no IMPLICIT type
C:\Users\angel\neural-fortran\example\dense_mnist.f90:20:14:

20 | dense(30), &
| 1
Error: Function 'dense' at (1) has no IMPLICIT type
mingw32-make[3]: *** [example\CMakeFiles\dense_mnist.dir\build.make:77: example/CMakeFiles/dense_mnist.dir/dense_mnist.f90.obj] Error 1
mingw32-make[2]: *** [CMakeFiles\Makefile2:735: example/CMakeFiles/dense_mnist.dir/all] Error 2
mingw32-make[1]: *** [Makefile:145: all] Error 2
mingw32-make[1]: Leaving directory 'C:/Users/angel/neural-fortran/build'