sn.set_random_seed doesn't make initialized weights and training results reproducible

[pradhyumna85]

The code below doesn't reproduce the same weights when model is initialized multiple times, even after using sn.set_random_seed function.

import sciann as sn
from sciann.utils.math import diff

seed = 43634
sn.set_random_seed(seed)

for i in range(4):

    x = sn.Variable('x',dtype='float64')
    y = sn.Variable('y',dtype='float64')

    inputs = [x,y]
    p = sn.Functional('p', inputs,2*[10], activation='tanh')

    ## L1-loss
    L1 = diff(p, x, order=2) + diff(p, y, order=2)

    losses = [L1]

    m = sn.SciModel(inputs, losses)

    weights = p.get_weights()

    ## just talking a slice to compare
    compare_weights = weights[0][0][0]

    print(f'############## Weights Iter: {i}')
    print(compare_weights)

Output:

############## Weights Iter: 0
[-0.08364122 -0.39584261  0.46033181  0.699524   -0.15388536  1.13492848
  0.97746673  0.0638296   0.22659807 -0.36424939]
############## Weights Iter: 1
[-0.57727796 -0.53107439 -0.36321291 -0.17676498 -0.00334409 -0.71008476
  0.98622227 -0.13798297  0.09670978 -1.08998305]
############## Weights Iter: 2
[-1.49435514  0.80398993  0.89099648 -0.35270435 -0.87543759 -1.57591196
  0.3990877   0.57710672  0.60861149  0.06177852]
############## Weights Iter: 3
[-0.4822131   0.8055504   0.2928848   1.15362153  0.95912567  0.30233269
  0.41268821  0.85532438 -0.36524137 -0.71060004]

[pradhyumna85]

I was able to fix this by adding a parameter init_seed to the sciann.Functional and passing this init_seed to the "seed" parameter of prepare_default_activations_and_initializers (utils/utilities.py) method called inside the sciann.Functional.

And now if we use this init_seed then the weights initialized become reproducible (and also the training results).

import sciann as sn
from sciann.utils.math import diff

seed = 43634
sn.set_random_seed(seed)

for i in range(4):

    x = sn.Variable('x',dtype='float64')
    y = sn.Variable('y',dtype='float64')

    inputs = [x,y]
    p = sn.Functional('p', inputs,2*[10], activation='tanh',init_seed=seed) ## passing the init_seed

    ## L1-loss
    L1 = diff(p, x, order=2) + diff(p, y, order=2)

    losses = [L1]

    m = sn.SciModel(inputs, losses)

    weights = p.get_weights()

    ## just talking a slice to compare
    compare_weights = weights[0][0][0]

    print(f'############## Weights Iter: {i}')
    print(compare_weights)

Output:

############## Weights Iter: 0
[ 0.39380369  0.35950971  0.74893839  1.33222068  0.5684224  -0.50967855
 -0.63144903  0.12257833 -0.33385106  0.08463333]
############## Weights Iter: 1
[ 0.39380369  0.35950971  0.74893839  1.33222068  0.5684224  -0.50967855
 -0.63144903  0.12257833 -0.33385106  0.08463333]
############## Weights Iter: 2
[ 0.39380369  0.35950971  0.74893839  1.33222068  0.5684224  -0.50967855
 -0.63144903  0.12257833 -0.33385106  0.08463333]
############## Weights Iter: 3
[ 0.39380369  0.35950971  0.74893839  1.33222068  0.5684224  -0.50967855
 -0.63144903  0.12257833 -0.33385106  0.08463333]

@ehsanhaghighat Let me know your though on this, I can raise a pull request with the required code changes.

[ehsanhaghighat]

you need to move sn.set_random_seed(seed) to after the for loop.

[pradhyumna85]

With reference to your advice, I tried the implementations below but in both the implementations the initialized weights were not reproducible.

1st Variation - sn.set_random_seed in the begining of the for loop

seed = 43634
for i in range(4):
    sn.set_random_seed(seed)
    
    x = sn.Variable('x',dtype='float64')
    y = sn.Variable('y',dtype='float64')

    inputs = [x,y]
    p = sn.Functional('p', inputs,2*[10], activation='tanh')

    ## L1-loss
    L1 = diff(p, x, order=2) + diff(p, y, order=2)

    losses = [L1]

    m = sn.SciModel(inputs, losses)

    weights = p.get_weights()

    ## just talking a slice to compare
    compare_weights = weights[0][0][0]

    print(f'############## Weights Iter: {i}')
    print(compare_weights)

Output:

############## Weights Iter: 0
[ 0.17192684  0.8505472   1.5303545  -0.07131549 -0.86318325 -0.01254475
 -0.95688675 -0.56421035 -0.13802275 -0.3585783 ]
############## Weights Iter: 1
[ 0.30115654  0.45204806  1.37423179  0.07729327 -0.64992897  0.74816071
 -0.23140187  0.54765457 -0.75551972 -0.53503478]
############## Weights Iter: 2
[ 0.26492076  0.88008895 -0.41499606  1.08219736  1.27035521 -1.43900151
 -0.99588373  0.50230552 -1.13693667  0.40157258]
############## Weights Iter: 3
[ 0.31586012 -0.2186777  -0.78905957  0.98990586 -0.3413854   1.00164896
 -0.31305837  0.10532799 -0.73633277 -1.54556423]

2nd Variation - sn.set_random_seed in the end of the for loop

seed = 43634
for i in range(4):
    
    x = sn.Variable('x',dtype='float64')
    y = sn.Variable('y',dtype='float64')

    inputs = [x,y]
    p = sn.Functional('p', inputs,2*[10], activation='tanh')

    ## L1-loss
    L1 = diff(p, x, order=2) + diff(p, y, order=2)

    losses = [L1]

    m = sn.SciModel(inputs, losses)

    weights = p.get_weights()

    ## just talking a slice to compare
    compare_weights = weights[0][0][0]

    print(f'############## Weights Iter: {i}')
    print(compare_weights)
    
    sn.set_random_seed(seed)

Output:

############## Weights Iter: 0
[ 0.62614818  0.24746175  0.3359769  -0.78704616 -0.06134327 -1.02164462
 -0.07365291 -1.04365773  0.11022187  1.09975384]
############## Weights Iter: 1
[-0.98858494  0.6591131   0.00676309  0.07054628  0.34525049  0.070351
 -0.24014073 -0.24590834  0.92014918 -0.11158274]
############## Weights Iter: 2
[ 0.25006986 -0.71967126  0.44816473 -0.31196732  0.31224364  0.67646486
 -0.03281037  0.54980068  0.17476912 -1.2574309 ]
############## Weights Iter: 3
[-0.53281204  0.23683607 -0.76720324 -1.01064904  1.21020394  1.41065052
 -0.30724967 -0.11914628  0.58634421 -0.53657643]

@ehsanhaghighat Am I missing something here ?
I anyways have raised a pull request for this, you can review.