volume start does not start the glusterfsd processes

[sac]

root@mgr:/home/sac/work/moana/mgr# ./bin/kadalu volume start FOO/testvol
Volume testvol started
root@mgr:/home/sac/work/moana/mgr# ./bin/kadalu volume list --status --detail 
Name                    : FOO/testvol
Type                    : Replicate
ID                      : bf0c1fb7-812c-43fd-b926-de65cf035aeb
State                   : Started (Down)
Size                    : 50.0GiB
Inodes                  : 26.2M
Utilization             : 730MiB/50.0GiB
Utilization (Inodes)    : 9.02k/26.2M
Options                 : -
Number of storage units : 3
Distribute group 1      : Health: Partial
    Storage unit 1      : vm1:/data/t0 (Port: 49252, Health: Down)
    Storage unit 2      : vm2:/data/t0 (Port: 49252, Health: Down)
    Storage unit 3      : vm3:/data/t0 (Port: 49252, Health: Up)

root@mgr:/home/sac/work/moana/mgr# 

[sac]

Should we have something like --force to brute force the volume start?

[aravindavk]

Volume start starts the glusterfsd process if not started. Any errors seen in Storage unit logs?

[sac]

Last few lines from the log (/var/log/kadalu/storage_units/*.log
There isn't much in the mgr.log

[2022-07-18 07:51:48.259332 +0000] W [socket.c:751:__socket_rwv] 0-tcp.testvol-server: readv on 192.168.122.68:49151 failed (No data available)
[2022-07-18 07:51:48.259388 +0000] I [MSGID: 115036] [server.c:494:server_rpc_notify] 0-testvol-server: disconnecting connection [{client-uid=CTX_ID:2a36d000-3afa-4f9b-bd7f-cee61c068bef-GRAPH_ID:0-PID:101339-HOST:mgr-PC_NAME:testvol-replicate-0-client-0-RECON_NO:-0}] 
[2022-07-18 07:51:48.259541 +0000] I [MSGID: 101054] [client_t.c:395:gf_client_unref] 0-testvol-server: Shutting down connection CTX_ID:2a36d000-3afa-4f9b-bd7f-cee61c068bef-GRAPH_ID:0-PID:101339-HOST:mgr-PC_NAME:testvol-replicate-0-client-0-RECON_NO:-0 
[2022-07-18 07:52:12.151938 +0000] I [addr.c:52:compare_addr_and_update] 0-/data/t0: allowed = "*", received addr = "192.168.122.68"
[2022-07-18 07:52:12.152006 +0000] I [MSGID: 115029] [server-handshake.c:563:server_setvolume] 0-testvol-server: accepted client from CTX_ID:c56960c6-77c3-4a05-a04c-1a5bf6b5e4bd-GRAPH_ID:0-PID:102032-HOST:mgr-PC_NAME:testvol-replicate-0-client-0-RECON_NO:-0 (version: 11dev) with subvol /data/t0 
[2022-07-18 07:53:51.136917 +0000] W [socket.c:751:__socket_rwv] 0-tcp.testvol-server: readv on 192.168.122.68:49148 failed (No data available)
[2022-07-18 07:53:51.140180 +0000] I [MSGID: 115036] [server.c:494:server_rpc_notify] 0-testvol-server: disconnecting connection [{client-uid=CTX_ID:c56960c6-77c3-4a05-a04c-1a5bf6b5e4bd-GRAPH_ID:0-PID:102032-HOST:mgr-PC_NAME:testvol-replicate-0-client-0-RECON_NO:-0}] 
[2022-07-18 07:53:51.140333 +0000] I [MSGID: 101054] [client_t.c:395:gf_client_unref] 0-testvol-server: Shutting down connection CTX_ID:c56960c6-77c3-4a05-a04c-1a5bf6b5e4bd-GRAPH_ID:0-PID:102032-HOST:mgr-PC_NAME:testvol-replicate-0-client-0-RECON_NO:-0 
[2022-07-18 10:52:40.733800 +0000] W [glusterfsd.c:1415:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x94947) [0x7f6029b8f947] -->glusterfsd(+0x129bd) [0x5652a2f169bd] -->glusterfsd(cleanup_and_exit+0x64) [0x5652a2f0f8b4] ) 0-: received signum (15), shutting down