juntang-zhuang/Adabelief-Optimizer

issues on AdaBlief-tensorflow

dusk666 opened this issue · 7 comments

dusk666 commented

HI!
I had some trouble using Adambelief in a simple lstm training.
What could be the reason for this？
CODE：
from adabelief_tf import AdaBeliefOptimizer
tf.keras.backend.clear_session()
multivariate_lstmA = tf.keras.models.Sequential([
LSTM(100, input_shape=input_shape,
return_sequences=True),
Flatten(),
Dense(200, activation='relu'),
Dropout(0.1),
Dense(1)
])
model_checkpoint = tf.keras.callbacks.ModelCheckpoint(
'multivariate_lstmA.h5', monitor=('val_loss'), save_best_only=True)
optimizer = AdaBeliefOptimizer(learning_rate=1e-3, epsilon=1e-14, rectify=False)
multivariate_lstmA.compile(loss=loss,
optimizer=optimizer,
metrics=metric)

RESULT：
Please check your arguments if you have upgraded adabelief-tf from version 0.0.1.
Modifications to default arguments:
eps weight_decouple rectify

adabelief-tf=0.0.1 1e-08 Not supported Not supported
Current version (0.1.0) 1e-14 supported default: True
For a complete table of recommended hyperparameters, see
https://github.com/juntang-zhuang/Adabelief-Optimizer

dusk666 commented

python version：3.7.6
tensoflow version：tensorflow-gpu 2.2.0
adabelief-tf ：0.1.0
model using tf.keras.api

louislbc commented

Hi, does the adabelief-tf support tensorflow 1, for instance, tensorflow-gpu 1.14? I found the adabelief-tf uninstalled my tensorflow 1 and installed tensorflow 2. Thanks in advance.

dusk666 commented

Hi, does the adabelief-tf support tensorflow 1, for instance, tensorflow-gpu 1.14? I found the adabelief-tf uninstalled my tensorflow 1 and installed tensorflow 2. Thanks in advance.

I think u need tensorflow version>=2.1.0 , because neither adabelief-tf 0.0.1 nor 0.1.0 work in my tensorflow 2.0.0 env.
BUT we can run it in tensorflow 2.2 or 2.3.

cryu854 commented

@dusk666 The message is to guide you to set the default arguments for AdaBelief, popping up once you create AdaBeliefOptimizer.

For version, the newest pypi_package cache file of adabelief_tf0.1.0 is not update yet.
Currently, please use the following commands to install:

git clone https://github.com/juntang-zhuang/Adabelief-Optimizer.git
pip install ./Adabelief-Optimizer/pypi_packages/adabelief_tf0.1.0
cryu854 commented

@louislbc Sorry, the package currently only supports tensorflow 2.

juntang-zhuang commented

@louislbc @dusk666 We have released version 0.2.0, please try the new version by pip install adabelief-tf==0.2.0, this version should be slightly faster than 0.1.0.

juntang-zhuang commented

@louislbc Please see https://github.com/juntang-zhuang/Adabelief-Optimizer/tree/update_0.3.0/pypi_packages/adabelief_tf0.3.0, there are two subfolders, one for tf1 the other for tf2. It's not pushed to pip yet, so please download the source and try. Thanks for the implementation by @cryu854.