AI4Finance-Foundation/ElegantRL

Suggestions for elegant_helloworld are welcome here. 给helloworld代码提建议

Yonv1943 opened this issue · 7 comments

Yonv1943 commented

If you have any suggestion about ElegantRL Helloworld, you can discuss them here, and we will keep an eye on this issue.
ElegantRL's code, especially the Helloworld, really needs a lot of feedback to be better.

如果大家对ElegantRL 的 Helloworld 有什么意见，可以在此处讨论，我们会持续关注这个问题。
ElegantRL 的代码，尤其是 Helloworld 真的需要很多人的反馈，才能改好。

We hope that the ElegantRL Helloworld would help people who want to learn about reinforcement learning to quickly run a few introductory examples.
Based on previous feedback, we've come up with a few requirements for the Helloworld code, with reasons in parentheses, to discuss with you.

  • Less lines of code. (In order to allow the reader to read it in 10 minutes, we want the code lines <1000)
  • Less packages requirements. (To reduce the installation time and difficulty of the code, only torch for deep learning, and gym gym[Box2D]` for RL learning environment should be installed)
  • have a gentle learning curve for reader.. (The reader can learn about reinforcement learning through the code, or learn about the core code of ElegantRL, so we show the DRL algorithm in order: DQN -> DDPG -> PPO)
  • Helloworld code should be in a consistent style with the full version of ElegantRL. (To help the reader's understanding of ElegantRL through ElegantRL_helloworld)

我们想要改写 ElegantRL 的 Helloworld 代码，希望它能帮助想要了解 强化学习的人快速运行几个入门例子。
根据大家的反馈，我们提出了对 Helloworld 代码的几点要求，并在括号内给出原因，同大家讨论：

  • 代码行数要少。（为了方便读者能在10分钟内读完，helloworld版本的代码量将维持在1000行以内）
  • 代码依赖的安装的包要足够少。（减少代码的安装时间，降低安装难度，只需要安装深度学习必备的 torch, 以及标准学习环境 gym，可选的 gym[Box2D]）
  • 代码要有平缓的学习曲线。（读者可以通过代码了解强化学习，或者了解ElegantRL的核心代码，所以我们按顺序展示了DRL算法：DQN -> DDPG -> PPO）
  • 代码要与 ElegantRL 完整版的代码 保持统一的风格。（方便读者通过 ElegantRL_helloworld 了解 ElegantRL）

If you are just starting learining about reinforcement learning, or coding, then your suggestions will be even more valuable to us (the helloworld code is positioned to help most people). Feel free to leave your suggestions here.

如果你对强化学习不够了解，或是对代码编写不够了解，那么你的建议对我们来说会更加宝贵（helloworld代码的定位就是要帮助大多数人）。欢迎在这里留下你的建议。

Yonv1943 commented

The following show our discussion, including What are changed? and Why change them?:
下面简单地展示一下我们的讨论，包括改了什么，以及为何要改：

net.py

  • mid_layer_num -> num_layer, the number layer of multilayer perceptron
  • def get_action(self, state): return a int variable, add comment
  • rd.rand() > self.explore_rate: --> self.explore_rate < rd.rand():
  • complete the variable names: inp --> input, out --> output, a_std_log -> action_std_log
  • explore_noise --> explore_noise_std, standard deviation of exploration action noise
  • dist_entropy --> entropy
  • def get_a_to_e(action): is a bad function name. This function converts action of network output to the action of environment input. So we plan to change get_a_to_e to convert_action_for_env
Yonv1943 commented

agent.py

  • remove env_num, it is the code of vectorized env. And it should not be here (ElegantRL helloworld)
  • ten --> tensor
  • self.convert_trajectory(traj_list) --> convert_trajectory_transition()

env.py

  • move function def kwargs_filter() to def build_env(). The reader don't need to know about it.

run.py

  • move class Arguments to config.py. The code that appears in run.py should be related to the training pipeline.
  • add torch.set_grad_enabled(False)
jaried commented

运行tutorial_helloworld_DQN.py时，出现如下的信息：

| Steps 1.03e+03 ExpR 1.00 | ObjC 0.03 ObjA 1.51
| Steps 1.46e+04 ExpR 1.00 | ObjC 0.25 ObjA 10.95

DQN有AC网络吗？

jaried commented

这里提到hello world有sac的demo，但是代码里面并没有sac。
https://elegantrl.readthedocs.io/en/latest/helloworld/intro.html#net-agent-env-run-file-structure

moon195 commented

I suggest that the tutorial documentation can be written by referring to pycaret, very fast, simple and effective, and hopefully with a few more complete examples, very useful for beginners

emmanuelsalawu commented

There is an Error 404 for
!wget https://github.com/AI4Finance-Foundation/ElegantRL/raw/master/elegantrl_helloworld -P /content/

--2022-12-11 16:38:21-- https://github.com/AI4Finance-Foundation/ElegantRL/raw/master/elegantrl_helloworld
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-12-11 16:38:21 ERROR 404: Not Found.

Has it been removed/replaced? This makes some of the examples, such as tutorial_helloworld_DQN_DDPG_PPO.ipynb, unrunnable.