What is the problem with installing console errors?

Question

What is the problem with installing console errors?

AbelRubick opened this issue 2 years ago · 27 comments

My version of nebula is:3.0.0
My version of console is:3.0.0 and 2.6.0(Have tested for the same error to occur)
error:
2022/06/03 22:11:29 Fail to initialize the connection pool, host: 127.0.0.1, port: 9669, failed to open connection, error: failed to verify client version: verifyClientVersion failed: wrong method name
panic: Fail to initialize the connection pool, host: 127.0.0.1, port: 9669, failed to open connection, error: failed to verify client version: verifyClientVersion failed: wrong method name

goroutine 1 [running]:
log.Panicf(0xc0003a6000, 0xb6, 0x0, 0x0, 0x0)
/opt/hostedtoolcache/go/1.16.4/x64/src/log/log.go:361 +0xc5
main.main()
/home/runner/work/nebula-console/nebula-console/main.go:535 +0xb10

Answer 1 · 2022-06-07T01:49:41.000Z

@Aiee do you have more ideas on this please, it seems version verification failed even when 3.0 console talking to 3.0 nebula graphD.

@AbelRubick could you help double-check its server-side version?

from its binary like nebula-graphd :

bin]# ./nebula-graphd --version
nebula-graphd version 3.1.0, Git: 33fd35e, Build Time: Apr 23 2022 22:03:50

show graph from its studio console if you have it

(root@nebula) [(none)]> show hosts graph
+-----------+------+----------+---------+--------------+---------+
| Host      | Port | Status   | Role    | Git Info Sha | Version |
+-----------+------+----------+---------+--------------+---------+
| "graphd"  | 9669 | "ONLINE" | "GRAPH" | "33fd35e"    | "3.1.0" |
| "graphd1" | 9669 | "ONLINE" | "GRAPH" | "33fd35e"    | "3.1.0" |
| "graphd2" | 9669 | "ONLINE" | "GRAPH" | "33fd35e"    | "3.1.0" |
+-----------+------+----------+---------+--------------+---------+

Answer 2 · 2022-06-08T06:01:11.000Z

@wey-gu This seems to be the typical error due to the version mismatch between the client-side and server-side.

Answer 3 · 2022-06-08T07:54:04.000Z

I've tested it and the version is 3.0.0

@Aiee do you have more ideas on this please, it seems version verification failed even when 3.0 console talking to 3.0 nebula graphD.

@AbelRubick could you help double-check its server-side version?

from its binary like nebula-graphd :
bin]# ./nebula-graphd --version
nebula-graphd version 3.1.0, Git: 33fd35e, Build Time: Apr 23 2022 22:03:50
show graph from its studio console if you have it
(root@nebula) [(none)]> show hosts graph
+-----------+------+----------+---------+--------------+---------+
| Host      | Port | Status   | Role    | Git Info Sha | Version |
+-----------+------+----------+---------+--------------+---------+
| "graphd"  | 9669 | "ONLINE" | "GRAPH" | "33fd35e"    | "3.1.0" |
| "graphd1" | 9669 | "ONLINE" | "GRAPH" | "33fd35e"    | "3.1.0" |
| "graphd2" | 9669 | "ONLINE" | "GRAPH" | "33fd35e"    | "3.1.0" |
+-----------+------+----------+---------+--------------+---------+

I've tested it and all of the version is 3.0.0

Answer 4 · 2022-06-08T09:29:52.000Z

I've tested it and all of the version is 3.0.0

@Aiee where could go wrong then, please? Anything that @AbelRubick could provide to look into this?

Is it possible to be a network issue just with the wrong error being raised by the console?

Answer 5 · 2022-12-26T09:42:21.000Z

Sorry for the late reply. The information you provided has not been reproduced locally. Do you have any new information you can provide us to reproduce? Thanks a lot for your contribution.

Answer 6 · 2023-01-29T06:18:39.000Z

We have noticed that the issue you created hasn’t been updated for nearly a month, so we have to close it for now. If you have any new updates, you are welcome to reopen this issue anytime. Thanks a lot for your contribution.

Answer 7 · 2023-05-07T17:06:54.000Z

How to solve this problem？

Answer 8 · 2023-05-08T02:14:05.000Z

How to solve this problem？

Have you encountered this, too? this error refers to version-hand-shake failure, which normally could be caused by two reasons:

console-db version mismatch, please double check this
console trying to access non-graphd thrift ports (i.e. metad /storaged ...)

could you double-check against the above cases?

Answer 9 · 2023-05-08T03:42:20.000Z

root@1f69b77c5572:/usr/local/nebula/bin# ./nebula-graphd --version
nebula-graphd version 3.4.0, Git: db3c1b3, Build Time: Feb 9 2023 12:09:22
This source code is licensed under Apache 2.0 License.

nebula-console-linux-amd64-v3.4.0
My nebula db and console are both 3.4.0.

Answer 10 · 2023-05-08T05:47:11.000Z

@fxb392 could you help double confirm:

your console cli line: i.e. nebula-console-3.4 -addr 192.168.8.128 -port 9669 -user root -p nebula and error
graphd is listening on the port your are accessing to, i.e. ss -plunt | grep 9669

Answer 11 · 2023-05-08T06:22:47.000Z

/usr/local/nebula/scripts/nebula.service status all

[INFO] nebula-metad(db3c1b3): Exited
[INFO] nebula-graphd(db3c1b3): Running as 565, Listening on 9669
[WARN] nebula-storaged after v3.0.0 will not start service until it is added to cluster.
[WARN] See Manage Storage hosts:ADD HOSTS in https://docs.nebula-graph.io/
[INFO] nebula-storaged(db3c1b3): Running as 611, Listening on 9779

./nebula-console-linux-amd64-v3.4.0 -addr 192.168.8.128 -port 9669 -user root -p nebula

2023/05/08 06:16:38 Fail to initialize the connection pool, host: 192.168.8.128, port: 9669, failed to open connection, error: failed to open transport, error: dial tcp 192.168.8.128:9669: i/o timeout
panic: Fail to initialize the connection pool, host: 192.168.8.128, port: 9669, failed to open connection, error: failed to open transport, error: dial tcp 192.168.8.128:9669: i/o timeout

goroutine 1 [running]:
log.Panicf(0xc0000aa0c0, 0xb5, 0x0, 0x0, 0x0)
/opt/hostedtoolcache/go/1.16.4/x64/src/log/log.go:361 +0xc5
main.main()
/home/runner/work/nebula-console/nebula-console/main.go:532 +0xb10

Answer 12 · 2023-05-08T06:23:56.000Z

root@1f69b77c5572:/nebula# ss -plunt | grep 9669
tcp LISTEN 0 128 0.0.0.0:19669 0.0.0.0:* users:(("nebula-graphd",pid=1012,fd=28))

Answer 13 · 2023-05-08T08:04:53.000Z

Is there a problem with my installation?

Answer 14 · 2023-05-08T12:11:03.000Z

It's related the metad not being up and running, could you check metad's log?

[INFO] nebula-metad(db3c1b3): Exited

Answer 15 · 2023-05-08T14:35:39.000Z

Three error logs:
1、nebula-metad.ERROR
Log file created at: 2023/05/08 14:23:52
Running on machine: 7fdf26acdc79
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
F20230508 14:23:52.975773 55 DiskManager.cpp:39] DataPath invalid: boost::filesystem::status: Function not implemented: "/usr/local/nebula/data/meta"

2、nebula-graphd.ERROR
Log file created at: 2023/05/08 14:23:55
Running on machine: 7fdf26acdc79
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E20230508 14:23:55.987087 203 MetaClient.cpp:772] Send request to "127.0.0.1":9559, exceed retry limit
E20230508 14:23:55.987252 203 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230508 14:23:55.987308 122 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
E20230508 14:24:08.997056 216 MetaClient.cpp:772] Send request to "127.0.0.1":9559, exceed retry limit
E20230508 14:24:08.997126 216 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230508 14:24:08.998440 122 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
E20230508 14:24:22.001811 218 MetaClient.cpp:772] Send request to "127.0.0.1":9559, exceed retry limit
E20230508 14:24:22.001852 218 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230508 14:24:22.001892 122 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
E20230508 14:24:35.006040 220 MetaClient.cpp:772] Send request to "127.0.0.1":9559, exceed retry limit
E20230508 14:24:35.006086 220 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230508 14:24:35.006141 122 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
E20230508 14:24:38.009608 283 MetaClient.cpp:772] Send request to "127.0.0.1":9559, exceed retry limit
E20230508 14:24:38.009657 283 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230508 14:24:38.009712 122 MetaClient.cpp:150] RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
E20230508 14:24:38.009747 122 GraphService.cpp:49] Failed to wait for meta service ready synchronously.
E20230508 14:24:38.009774 122 GraphServer.cpp:39] Failed to wait for meta service ready synchronously.
E20230508 14:24:38.011329 122 GraphDaemon.cpp:156] The graph server start failed

3、nebula-storaged.ERROR
Log file created at: 2023/05/08 14:23:55
Running on machine: 7fdf26acdc79
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E20230508 14:23:55.998018 210 MetaClient.cpp:772] Send request to "127.0.0.1":9559, exceed retry limit
E20230508 14:23:55.998123 210 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230508 14:23:55.998186 165 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
E20230508 14:24:09.003250 217 MetaClient.cpp:772] Send request to "127.0.0.1":9559, exceed retry limit
......

what should i do next?

Answer 16 · 2023-05-09T01:57:16.000Z

boost::filesystem::status: Function not implemented

@fxb392 it's a known issue vesoft-inc/nebula#4205 may I know what OS are you using?

uname -a

For now, I suggest using one with a newer version of the linux kernel.

Answer 17 · 2023-05-09T03:30:26.000Z

@wey-gu
uname -a
Linux 7fdf26acdc79 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64 GNU/Linux
I run this in docker.

Answer 18 · 2023-05-09T09:57:24.000Z

3.10.0

The root cause is the upstream boost::filesystem::status() doesn't have implementation under 3.10.0 kernal, which was released 10 years ago T__T.

Answer 19 · 2023-05-10T03:03:50.000Z

Thank you very much for your help.
Can this problem be avoided if Nebula db 3.3.0 is used?

Answer 20 · 2023-05-10T03:37:05.000Z

No, it's been there for more than one year as I observed, sorry about this, could you spawn some VM with a higher version of kernel?

Answer 21 · 2023-05-10T06:09:15.000Z

Ok.If i want upgrade the kernal version,which kernal version is suitable.

Answer 22 · 2023-05-10T06:15:06.000Z

Ok.If i want upgrade the kernal version,which kernal version is suitable.

According to similar issues

~~4.11 onward would be good enough for x86_64, but I haven't tried it yet~~

It's strange enough that someone with a 4.15 kernel is with this issue, but I could see one of my machines with an old kernel worked fine Linux nebula1 3.10.0-1160.76.1.el7.x86_64

It's highly recommended to try spawning a new VM, which should be working fine.

Answer 23 · 2023-05-11T03:15:50.000Z

I used version 3.0.2 and successfully installed it.
root@83311f6a550a:/opt/nebula# /usr/local/nebula/scripts/nebula.service status all
[INFO] nebula-metad(4805f8c): Running as 144, Listening on 9559
[INFO] nebula-graphd(4805f8c): Running as 212, Listening on 9669
[INFO] nebula-storaged(4805f8c): Running as 251, Listening on 9779

But i encountered another problem...
root@83311f6a550a:/opt/nebula# dpkg -i nebula-graph-studio-3.2.5.x86_64.deb
(Reading database ... 5855 files and directories currently installed.)
Preparing to unpack nebula-graph-studio-3.2.5.x86_64.deb ...
Unpacking nebula-graph-studio (3.2.5) over (3.2.5) ...
Setting up nebula-graph-studio (3.2.5) ...
ERROR:systemctl: dbus.service: Executable path is not absolute, ignoring: @/usr/bin/dbus-daemon @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only

Answer 24 · 2023-05-11T03:46:16.000Z

ok, that's a quite outdated version though :)

Answer 25 · 2023-05-11T06:02:11.000Z

ERROR:systemctl: dbus.service: Executable path is not absolute, ignoring: @/usr/bin/dbus-daemon @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
@wey-gu Does this error report have any impact？I can open the studio now.

Answer 26 · 2023-05-11T06:34:30.000Z

ERROR:systemctl: dbus.service: Executable path is not absolute, ignoring: @/usr/bin/dbus-daemon @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only @wey-gu Does this error report have any impact？I can open the studio now.

This is a problem with systemd's dependency on dbus. Perhaps you don't have dbus installed. You can manually start the studio process without using systemd.

--
这是 systemd 对 dbus 的依赖的问题，是不是你没有 dbus，你可以手动启动 studio 进程，不用 systemd。

Answer 27 · 2023-05-11T06:41:28.000Z

这是 systemd 对 dbus 的依赖的问题，是不是你没有 dbus，你可以手动启动 studio 进程，不用 systemd。

好的，不是nebula studio的问题就行，我现在能打开nebula studio的ui界面。非常感谢您这两天的帮助！！！