optuna/optuna-dashboard

Different parameter importance shown in Dashboard vs using the importance evaluator standalone

Opened this issue · 0 comments

jt269 commented

Description

I have being trying to reproduce the importance chart from the dashboard for distribution to colleagues and clients. I wanted it in a format that I could manipulate such as a .csv. However when trying to reproduce the importances using the Ped-Anova evaluator and exporting the output dictionary to a .csv the results are different.
Chart from dashboard:
image
Chart reproduced from .csv:
image

As can be seen the important parameters are different and the magnitude of the importances are different. Both methods are reading from the same journal file.

Thanks

How to Reproduce

Code used to launch dashboard:

from optuna_dashboard import run_server
import webbrowser
import threading
import os

def dash_main():
    current_path = os.path.realpath(__file__)
    current_directory = os.path.dirname(current_path)
    file_path = os.path.join(current_directory, 'journal.log')
    server_thread = threading.Thread(target=run_dashboard_server, args=(file_path,))
    server_thread.daemon = True
    server_thread.start()
    server_thread.join(timeout=5)
    url = 'http://localhost:8080/dashboard/studies/0'
    webbrowser.open(url)

def run_dashboard_server(file_path):
    run_server(file_path)

if __name__ == '__main__':
    dash_main()

Code used to use standalone evaluator:

import optuna
from optuna.importance import PedAnovaImportanceEvaluator
import pandas as pd

journal_path='journal.log'

lock_obj = optuna.storages.JournalFileOpenLock(journal_path)

storage = optuna.storages.JournalStorage(
    optuna.storages.JournalFileStorage(journal_path, lock_obj=lock_obj),
)

study = optuna.load_study(
    study_name="AMC_Optimiser", storage=storage
)

evaluator = PedAnovaImportanceEvaluator()
data=evaluator.evaluate(study)

df_export = pd.DataFrame(list(data.items()), columns=['Parameter', 'Importance'])
df_export.to_csv('importance_data.csv', index=False)

Python version

3.11.9

Optuna version

3.6.1

optuna-dashboard version or git revision

0.15.1

Web browser

Google Chrome