I get the following error when visiting the digital twin virtual scene, can someone help me please to solve it

Question

I get the following error when visiting the digital twin virtual scene, can someone help me please to solve it

Opened this issue a year ago · 17 comments

Hello!
When running the command "python inference.py --host=localhost:30001 --action_nums=8", I get the following error:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30001: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30001: Failed to connect to remote host: Connection refused {created_time:"2023-07-09T15:48:41.970507598+08:00", grpc_status:14}"

I get the similar error when running the command "python SYSU_sim_test.py":

Answer 1 · 2023-07-09T07:53:36.000Z

The error I get when running the command "python SYSU_sim_test.py" is：
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:30001: WSA Error"
debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:30001: WSA Error {grpc_status:14, created_time:"2023-07-09T07:15:06.520195799+00:00"}"

Answer 2 · 2023-07-10T05:38:44.000Z

I got the same error:
(RM-PRT) liyun@DESKTOP-0NAM80S:~/RM-PRT$ python inference.py --host=172.23.159.81:30001 --action_nums=8
/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindformers/core/clip_grad.py:39: RuntimeWarning: divide by zero encountered in log
inf = Tensor(np.log(0.0), mstype.float32)
start
Traceback (most recent call last):
File "inference.py", line 55, in
initworld = stub.Init(GrabSim_pb2.Count(value = 2))
File "/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/grpc/_channel.py", line 1030, in call
return _end_unary_response_blocking(state, call, False, None)
File "/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.23.159.81:30001: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.23.159.81:30001: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-07-10T14:24:38.07754811+09:00"}"

Answer 3 · 2023-07-10T07:59:35.000Z

You need to open the simulator first on the computer

Answer 4 · 2023-07-10T11:20:03.000Z

你好！我是华工九月入学的研究生感谢您的解答，在github上提出的问题已经解决了另外，我对您的研究方向很感兴趣，方便加一下微信吗？

…

------------------ 原始邮件 ------------------ 发件人: "Necolizer/RM-PRT" ***@***.***>; 发送时间: 2023年7月10日(星期一) 下午3:59 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Necolizer/RM-PRT] I get the following error when visiting the digital twin virtual scene, can someone help me please to solve it (Issue #1) You need to open the simulator first on the computer — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 5 · 2023-07-11T08:41:31.000Z

你好，你是怎么解决的？能加微型交流一下吗。我的微信：670028676

Answer 6 · 2023-07-11T08:42:53.000Z

你好！我是华工九月入学的研究生感谢您的解答，在github上提出的问题已经解决了另外，我对您的研究方向很感兴趣，方便加一下微信吗？
…
------------------ 原始邮件 ------------------ 发件人: "Necolizer/RM-PRT" @.>; 发送时间: 2023年7月10日(星期一) 下午3:59 @.>; @.@.>; 主题: Re: [Necolizer/RM-PRT] I get the following error when visiting the digital twin virtual scene, can someone help me please to solve it (Issue #1) You need to open the simulator first on the computer — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

你好，你是怎么解决的？能加微型交流一下吗。我的微信：670028676

Answer 7 · 2023-07-11T13:52:07.000Z

Can you provide more details about how to use this code? The current instruction is so less information. Thank you

Answer 8 · 2023-07-11T13:53:56.000Z

You need to open the simulator first on the computer

Can you provide more details about how to use this code? The current instruction is so less information. Thank you

Answer 9 · 2023-07-11T14:02:25.000Z

需要先在电脑上打开模拟器

您能否提供有关如何使用此代码的更多详细信息？当前的指令信息很少。谢谢

I'm sorry for forgetting to explain how to use the simulator. You need to download the simulator and just run GrabSimClient.exe or GrabSimClient.sh. You can download the simulator and get its usage through this website.

Answer 10 · 2023-07-11T15:07:09.000Z

需要先在电脑上打开模拟器

您能否提供有关如何使用此代码的更多详细信息？当前的指令信息很少。谢谢

I'm sorry for forgetting to explain how to use the simulator. You need to download the simulator and just run GrabSimClient.exe or GrabSimClient.sh. You can download the simulator and get its usage through this website.

兄弟，能加我微信不？或者给个微信，上面那个哥们也都是国人，快速群聊一下解决一下问题啊~ 感谢！

Answer 11 · 2023-07-12T08:26:24.000Z

你好，simulator已经打开了也看到了机器人，但是该如何交互呢？如何操作？能出一个更详细的Instructin吗？现在的操作说明实在太少了。期待尽快回复，感谢。

Answer 12 · 2023-07-12T09:09:44.000Z

你好，simulator已经打开了也看到了机器人，但是该如何交互呢？如何操作？能出一个更详细的Instructin吗？现在的操作说明实在太少了。期待尽快回复，感谢。

SYSU_sim_test.py已经提供了相关API接口的使用示例

Answer 13 · 2023-07-12T09:19:58.000Z

你好，simulator已经打开了也看到了机器人，但是该如何交互呢？如何操作？能出一个更详细的Instructin吗？现在的操作说明实在太少了。期待尽快回复，感谢。

SYSU_sim_test.py已经提供了相关API接口的使用示例

能提供个详细点的Instructin吗？主要是如何用用语言通过LLM和机器人交互这块儿？感谢。最好能加个微信：670028676 交流一下啊？我是大阪大学的博三，也是做这块儿，想用你这个base做进一步的实现，方便交流下吗？

Answer 14 · 2023-07-12T12:42:53.000Z

inference.py文件的执行一直在报错：

(RM-PRT) liyun@DESKTOP-0NAM80S:~/RM-PRT$ python inference.py
/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindformers/core/clip_grad.py:39: RuntimeWarning: divide by zero encountered in log
inf = Tensor(np.log(0.0), mstype.float32)
start
------------------show_env_info----------------------
sceneID:0, location:[-2150.0, -1350.0], rotation:
joints number:21, fingers number:10
objects number: 505
velocity:0.0, rotation:0.0, timestep:0

successfully initialized

Traceback (most recent call last):
File "inference.py", line 70, in
predictor=predictor_construct()
File "/home/liyun/RM-PRT/Predictor.py", line 64, in predictor_construct
model = RT1(
File "<@beartype(RT1_ms.RT1.init) at 0x7fe2d9e3ef70>", line 43, in init
File "/home/liyun/RM-PRT/RT1_ms.py", line 583, in init
self.conditioner = conditioner_klass(
File "<@beartype(classifier_free_guidance_ms.TextConditioner.init) at 0x7fe2d9e2fa60>", line 37, in init
File "/home/liyun/RM-PRT/classifier_free_guidance_ms.py", line 254, in init
model = klass(model_name)
File "/home/liyun/RM-PRT/textEncoder.py", line 40, in init
model, tokenizer = get_model_and_tokenizer(name)
File "/home/liyun/RM-PRT/textEncoder.py", line 26, in get_model_and_tokenizer
model = AutoModel.from_pretrained(name)
File "/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindformers/auto_class.py", line 322, in from_pretrained
if cls.invalid_model_name(pretrained_model_name_or_dir):
File "/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindformers/auto_class.py", line 209, in invalid_model_name
local_value = cls.support_list[pretrained_model_name_or_dir.split('')[cls._model_type]]
KeyError: './bert'

将存放predictor.ckpt的文件夹./checkpoints设定后依然报错：

model_type is: bert, model_name is: ./checkpoints
Traceback (most recent call last):
File "inference.py", line 70, in
predictor=predictor_construct()
File "/home/liyun/RM-PRT/Predictor.py", line 64, in predictor_construct
model = RT1(
File "<@beartype(RT1_ms.RT1.init) at 0x7f112a259f70>", line 43, in init
File "/home/liyun/RM-PRT/RT1_ms.py", line 583, in init
self.conditioner = conditioner_klass(
File "<@beartype(classifier_free_guidance_ms.TextConditioner.init) at 0x7f112a244a60>", line 37, in init
File "/home/liyun/RM-PRT/classifier_free_guidance_ms.py", line 254, in init
model = klass(model_name)
File "/home/liyun/RM-PRT/textEncoder.py", line 40, in init
model, tokenizer = get_model_and_tokenizer(name)
File "/home/liyun/RM-PRT/textEncoder.py", line 26, in get_model_and_tokenizer
model = AutoModel.from_pretrained(name)
File "/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindformers/auto_class.py", line 333, in from_pretrained
raise FileNotFoundError(f"there is no yaml file for model config or ckpt file"
FileNotFoundError: there is no yaml file for model config or ckpt file for model weights in ./checkpoints

请问这是什么原因？

Answer 15 · 2023-07-12T13:54:48.000Z

这里需要放Bert的模型可以从huggingface下载bert-base-uncased，参考网址https://huggingface.co/bert-base-uncased

Answer 16 · 2023-07-12T16:42:19.000Z

上面的已经改好了但仍然有以下报错，请帮忙看一下，感谢：

Traceback (most recent call last):
File "/home/liyun/RM-PRT/inference.py", line 80, in
action = predictor.predict(obs)[0]
File "/home/liyun/RM-PRT/Predictor.py", line 39, in predict
train_logits = self.RT1(head_rgb, text_embeds=text_embeds)
File "/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindspore/nn/cell.py", line 645, in call
raise err
File "/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindspore/nn/cell.py", line 641, in call
output = self._run_construct(args, kwargs)
File "/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindspore/nn/cell.py", line 429, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/home/liyun/RM-PRT/classifier_free_guidance_ms.py", line 130, in inner
logits = fn_maybe_with_text(self, *args, **kwargs_without_cond_dropout)
File "/home/liyun/RM-PRT/classifier_free_guidance_ms.py", line 116, in fn_maybe_with_text
return fn(self, *args, **kwargs)
TypeError: construct() got multiple values for argument 'cond_drop_prob'

Answer 17 · 2023-07-13T05:55:26.000Z

python inference.py的预期结果是什么？我修改了一些代码中的运行bug但最后的运行效果没看到什么仅仅只是：Fail to grasp？如果可以的话能否抽时间写一个更详细的操作流程吗？尤其是LLM和robot如何实现交互和操作的？感谢。

(RM-PRT) liyun@DESKTOP-0NAM80S:~/RM-PRT$ python inference.py
/home/liyun/anaconda3/envs/RM-PRT/lib/python3.8/site-packages/mindformers/core/clip_grad.py:39: RuntimeWarning: divide by zero encountered in log
inf = Tensor(np.log(0.0), mstype.float32)
start
------------------show_env_info----------------------
sceneID:0, location:[-2150.0, -1350.0], rotation:
joints number:21, fingers number:10
objects number: 505
velocity:0.0, rotation:0.0, timestep:0

successfully initialized
2023-07-13 14:50:12,767 - mindformers - INFO - start to read the ckpt file: 534077599
[WARNING] ME(16630:140030269297472,MainProcess):2023-07-13-14:50:13.794.329 [mindspore/train/serialization.py:1110] For 'load_param_into_net', remove parameter prefix name: bert., continue to load.
[WARNING] ME(16630:140030269297472,MainProcess):2023-07-13-14:50:13.795.351 [mindspore/train/serialization.py:1085] For 'load_param_into_net', 3 parameters in the 'net' are not loaded, because they are not in the 'parameter_dict', please check whether the network structure is consistent when training and loading checkpoint.
[WARNING] ME(16630:140030269297472,MainProcess):2023-07-13-14:50:13.795.455 [mindspore/train/serialization.py:1090] mlmloss.output_bias is not loaded.
[WARNING] ME(16630:140030269297472,MainProcess):2023-07-13-14:50:13.795.480 [mindspore/train/serialization.py:1090] nsploss.dense.weight is not loaded.
[WARNING] ME(16630:140030269297472,MainProcess):2023-07-13-14:50:13.795.508 [mindspore/train/serialization.py:1090] nsploss.dense.bias is not loaded.
2023-07-13 14:50:13,795 - mindformers - INFO - weights in ./checkpoint_download/bert/bert_base_uncased.ckpt are loaded
2023-07-13 14:50:13,795 - mindformers - INFO - model built successfully!
2023-07-13 14:50:13,798 - mindformers - INFO - config in the yaml file ./checkpoint_download/bert/bert_base_uncased.yaml are used for tokenizer building.
2023-07-13 14:50:13,800 - mindformers - WARNING - Can't find the tokenizer_config.json in the file_dict. The content of file_dict is : {}
2023-07-13 14:50:13,800 - mindformers - INFO - build tokenizer class name is: BertTokenizer using args {'cls_token': '[CLS]', 'do_basic_tokenize': True, 'do_lower_case': True, 'mask_token': '[MASK]', 'pad_token': '[PAD]', 'sep_token': '[SEP]', 'unk_token': '[UNK]', 'vocab_file': './checkpoint_download/bert/vocab.txt'}.
Fail to grasp