Storage List Command Fails
Closed this issue · 18 comments
slimandslam commented
M1 Macbook Pro with MacOS 13.2.1
$ mkdir AIR
$ cd AIR
$ python3.10 -m venv venv
$ source venv/bin/activate
(venv) $ pip3 install airavata-mft-cli
Collecting airavata-mft-cli
Using cached airavata_mft_cli-0.1.9-py3-none-any.whl (21 kB)
Collecting typer[all]<0.8.0,>=0.7.0
Using cached typer-0.7.0-py3-none-any.whl (38 kB)
Collecting pick==2.2.0
Using cached pick-2.2.0-py3-none-any.whl (4.9 kB)
Collecting airavata_mft_sdk==0.0.1-alpha27
Using cached airavata_mft_sdk-0.0.1a27-py3-none-any.whl (96 kB)
Collecting grpcio==1.47.0rc1
Using cached grpcio-1.47.0rc1-cp310-cp310-macosx_13_0_arm64.whl
Collecting grpcio-tools==1.47.0rc1
Using cached grpcio_tools-1.47.0rc1-cp310-cp310-macosx_13_0_arm64.whl
Collecting google-api-python-client>=2.0.0
Downloading google_api_python_client-2.83.0-py2.py3-none-any.whl (11.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/11.2 MB 3.4 MB/s eta 0:00:00
Collecting six>=1.5.2
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from grpcio-tools==1.47.0rc1->airavata-mft-cli) (67.2.0)
Collecting protobuf<4.0dev,>=3.12.0
Using cached protobuf-3.20.3-py2.py3-none-any.whl (162 kB)
Collecting click<9.0.0,>=7.1.1
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting shellingham<2.0.0,>=1.3.0
Using cached shellingham-1.5.0.post1-py2.py3-none-any.whl (9.4 kB)
Collecting colorama<0.5.0,>=0.4.3
Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting rich<13.0.0,>=10.11.0
Using cached rich-12.6.0-py3-none-any.whl (237 kB)
Collecting uritemplate<5,>=3.0.1
Using cached uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
Collecting google-auth<3.0.0dev,>=1.19.0
Downloading google_auth-2.17.0-py2.py3-none-any.whl (178 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 178.1/178.1 kB 13.5 MB/s eta 0:00:00
Collecting google-auth-httplib2>=0.1.0
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.11.0-py3-none-any.whl (120 kB)
Collecting httplib2<1dev,>=0.15.0
Using cached httplib2-0.22.0-py3-none-any.whl (96 kB)
Collecting pygments<3.0.0,>=2.6.0
Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting commonmark<0.10.0,>=0.9.0
Using cached commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
Collecting requests<3.0.0dev,>=2.18.0
Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting googleapis-common-protos<2.0dev,>=1.56.2
Using cached googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
Collecting cachetools<6.0,>=2.0.0
Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.9-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting pyasn1<0.5.0,>=0.4.6
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting charset-normalizer<4,>=2
Using cached charset_normalizer-3.1.0-cp310-cp310-macosx_11_0_arm64.whl (123 kB)
Installing collected packages: pyasn1, commonmark, urllib3, uritemplate, six, shellingham, rsa, pyparsing, pygments, pyasn1-modules, protobuf, pick, idna, colorama, click, charset-normalizer, certifi, cachetools, typer, rich, requests, httplib2, grpcio, googleapis-common-protos, google-auth, grpcio-tools, google-auth-httplib2, google-api-core, google-api-python-client, airavata_mft_sdk, airavata-mft-cli
Successfully installed airavata-mft-cli-0.1.9 airavata_mft_sdk-0.0.1a27 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.1.0 click-8.1.3 colorama-0.4.6 commonmark-0.9.1 google-api-core-2.11.0 google-api-python-client-2.83.0 google-auth-2.17.0 google-auth-httplib2-0.1.0 googleapis-common-protos-1.59.0 grpcio-1.47.0rc1 grpcio-tools-1.47.0rc1 httplib2-0.22.0 idna-3.4 pick-2.2.0 protobuf-3.20.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pygments-2.14.0 pyparsing-3.0.9 requests-2.28.2 rich-12.6.0 rsa-4.9 shellingham-1.5.0.post1 six-1.16.0 typer-0.7.0 uritemplate-4.1.1 urllib3-1.26.15
(venv) $ mft init
Setting up MFT Services
Consul process id: 74795
Standalone Service stoping ...
./standalone-service-daemon.sh: line 55: kill: (62156) - No such process
Standalone Service stopped ...
Starting Standalone Service ...
Standalone Service started ...
MFT Started
(venv) $ mft storage list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/storage/__init__.py:6 │
│ 6 in list_storage │
│ │
│ 63 │ │ │ │ │ │ │ │ secret_service_host = configcli.secret_service_host, │
│ 64 │ │ │ │ │ │ │ │ secret_service_port = configcli.secret_service_port) │
│ 65 │ list_req = StorageCommon_pb2.StorageListRequest() │
│ ❱ 66 │ list_response = client.common_api.listStorages(list_req) │
│ 67 │ │
│ 68 │ console = Console() │
│ 69 │ table = Table(show_header=True, header_style='bold #2070b2') │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ client = <airavata_mft_sdk.mft_client.MFTClient object at 0x10796d8a0> │ │
│ │ list_req = <class 'rich.pretty.Node'>.__repr__ returned empty string │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/myuser/venv/lib/python3.10/site-packages/grpc/_channel.py:946 in __call__ │
│ │
│ 943 │ │ │ │ compression=None): │
│ 944 │ │ state, call, = self._blocking(request, timeout, metadata, credentials, │
│ 945 │ │ │ │ │ │ │ │ │ wait_for_ready, compression) │
│ ❱ 946 │ │ return _end_unary_response_blocking(state, call, False, None) │
│ 947 │ │
│ 948 │ def with_call(self, │
│ 949 │ │ │ │ request, │
│ │
│ ╭──────────────────────────────────── locals ─────────────────────────────────────╮ │
│ │ call = <grpc._cython.cygrpc.SegregatedCall object at 0x1076085c0> │ │
│ │ compression = None │ │
│ │ credentials = None │ │
│ │ metadata = None │ │
│ │ request = <class 'rich.pretty.Node'>.__repr__ returned empty string │ │
│ │ self = <grpc._channel._UnaryUnaryMultiCallable object at 0x107adde70> │ │
│ │ state = <grpc._channel._RPCState object at 0x107adf4f0> │ │
│ │ timeout = None │ │
│ │ wait_for_ready = None │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/myuser/venv/lib/python3.10/site-packages/grpc/_channel.py:849 in │
│ _end_unary_response_blocking │
│ │
│ 846 │ │ else: │
│ 847 │ │ │ return state.response │
│ 848 │ else: │
│ ❱ 849 │ │ raise _InactiveRpcError(state) │
│ 850 │
│ 851 │
│ 852 def _stream_unary_invocation_operationses(metadata, initial_metadata_flags): │
│ │
│ ╭──────────────────────────────── locals ────────────────────────────────╮ │
│ │ call = <grpc._cython.cygrpc.SegregatedCall object at 0x1076085c0> │ │
│ │ deadline = None │ │
│ │ state = <grpc._channel._RPCState object at 0x107adf4f0> │ │
│ │ with_call = False │ │
│ ╰────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1680205081.363435000","description":"Failed to pick
subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3261,"referenced_errors":
[{"created":"@1680205081.363434000","description":"failed to connect to all
addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>
(venv) $
DImuthuUpe commented
@slimandslam Can you post the output of mft log ?
slimandslam commented
Here it is:
$ mft log
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/base.py:47 in │
│ init_mft │
│ │
│ 44 │
│ 45 @app.command("log") │
│ 46 def init_mft(): │
│ ❱ 47 bootstrap.print_log() │
│ │
│ /Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/bootstrap.py:158 in │
│ print_log │
│ │
│ 155 │
│ 156 def print_log(): │
│ 157 log_file_path = os.path.join(os.path.expanduser('~'), ".mft", "Standalone-Service-0.01 │
│ ❱ 158 log_file = open(log_file_path,"r") │
│ 159 lines = follow_file(log_file) │
│ 160 for line in lines: │
│ 161 │ print(line) │
│ │
│ ╭────────────────────────────────────── locals ───────────────────────────────────────╮ │
│ │ log_file_path = '/Users/myuser/.mft/Standalone-Service-0.01/logs/airavata.log' │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory:
'/Users/myuser/.mft/Standalone-Service-0.01/logs/airavata.log'
$
DImuthuUpe commented
@slimandslam I think the latest mft cli is having a conflict with an older mft version you had earlier. Can you try running following commands to make sure that you have a clean installation?
rm ~/mft_db.*
mft stop
rm -rf ~/.mft
mft init
Please post the output of mft init. It should download binaries from GitHub release
slimandslam commented
Ok.
$ rm ~/mft_db.*
rm: /Users/myuser/mft_db.*: No such file or directory
$ mft stop
Stopping MFT Services
Standalone Service stoping ...
./standalone-service-daemon.sh: line 55: kill: (83776) - No such process
Standalone Service stopped ...
MFT Stopped....
$ rm -rf ~/.mft
$ mft init
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 1372, in _path_importer_cache
KeyError: '.'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/myuser/venv/bin/mft", line 5, in <module>
from airavata_mft_cli.main import app
File "/Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/main.py", line 21, in <module>
import airavata_mft_cli.base
File "/Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/base.py", line 21, in <module>
import airavata_mft_cli.bootstrap as bootstrap
File "/Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/bootstrap.py", line 20, in <module>
import requests
File "/Users/myuser/venv/lib/python3.10/site-packages/requests/__init__.py", line 43, in <module>
import urllib3
File "/Users/myuser/venv/lib/python3.10/site-packages/urllib3/__init__.py", line 13, in <module>
from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
File "Users/myuser/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 12, in <module>
from .connection import (
File "Users/myuser/venv/lib/python3.10/site-packages/urllib3/connection.py", line 15, in <module>
from .util.proxy import create_proxy_ssl_context
File "Users/myuser/venv/lib/python3.10/site-packages/urllib3/util/__init__.py", line 5, in <module>
from .request import SKIP_HEADER, SKIPPABLE_HEADERS, make_headers
File "Users/myuser/venv/lib/python3.10/site-packages/urllib3/util/request.py", line 18, in <module>
import brotlicffi as _unused_module_brotli # noqa: F401
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 945, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1439, in find_spec
File "<frozen importlib._bootstrap_external>", line 1408, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1374, in _path_importer_cache
File "<frozen importlib._bootstrap_external>", line 1350, in _path_hooks
File "<frozen importlib._bootstrap_external>", line 1632, in path_hook_for_FileFinder
File "<frozen importlib._bootstrap_external>", line 1505, in __init__
FileNotFoundError: [Errno 2] No such file or directory
$
DImuthuUpe commented
It seems like some issue in the virtual environment you created. The line which is getting failed is https://github.com/apache/airavata-mft/blob/master/python-cli/mft_cli/airavata_mft_cli/bootstrap.py#L20 which is the import of requests module. Can you try creating a new virtual environment and repeating the installation steps?
slimandslam commented
What are you using to check if your daemon is running? Every time I've done an install, I see this (below) but there is never anything running.
$ mft init
Setting up MFT Services
Consul process id: 41547
Standalone Service stoping ...
./standalone-service-daemon.sh: line 55: kill: (41361) - No such process
Standalone Service stopped ...
Starting Standalone Service ...
Standalone Service started ...
MFT Started
$
DImuthuUpe commented
I think the process created for the previous mft server run was abruptly killed or exited. That's why you are seeing " No such process" error. One possibility is that you do not have a compatible java runtime installed on your machine. We can try the attached mode in mft to narrow down the issue. Please send the output after running the following commands.
cd ~/.mft/Standalone-Service-0.01/bin
chmod +x standalone-service.sh
./standalone-service.sh
slimandslam commented
What are you using to check to see if the daemon is running? Doe CLI commands depend on the daemon process?
J
DImuthuUpe commented
mft init command kills the currently running mft service daemon process (if exists) and starts a new one. When the daemon process starts, it saves a process id file at file ~/.mft/Standalone-Service-0.01/bin/service-pid. When a daemon restart was invoked, it looks for a process with the id in service-pid file to kill first. However, in your case, this process exited due to some error, and logs were not even created. I assume that you do not have a compatible java virtual machine in your computer. The commands I mentioned earlier should reveal what the error is
slimandslam commented
Do the mft CLI commands depend on the daemon in order to work?
J
…On Tue, Apr 4, 2023 at 9:20 AM Dimuthu Wannipurage ***@***.***> wrote:
mft init command kills the currently running mft service daemon process
(if exists) and starts a new one. When the daemon process starts, it saves
a process id file at file ~/.mft/Standalone-Service-0.01/bin/service-pid.
When a daemon restart was invoked, it looks for a process with the id in
service-pid file to kill first. However, in your case, this process exited
due to some error, and logs were not even created. I assume that you do not
have a compatible java virtual machine in your computer. The commands I
mentioned earlier should reveal what the error is
—
Reply to this email directly, view it on GitHub
<#93 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABYBR225HME6BPKGWO2LN3W7QUZBANCNFSM6AAAAAAWNXYORA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
DImuthuUpe commented
Yes, it requires the daemon to be up and running. CLI talks to the daemon through its APIs
slimandslam commented
Is there a way for the CLI to return a sensible error message like: “cannot
connect to demon” ?
J
…On Tue, Apr 4, 2023 at 10:03 AM Dimuthu Wannipurage < ***@***.***> wrote:
Yes, it requires the daemon to be up and running. CLI talks to the daemon
through its APIs
—
Reply to this email directly, view it on GitHub
<#93 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABYBR4MWR6FXRUQIM5DDHDW7QZ3LANCNFSM6AAAAAAWNXYORA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
DImuthuUpe commented
Good suggestion. I created the issue #94 to track this enhancement
slimandslam commented
Thanks!
…On Tue, Apr 4, 2023 at 12:23 PM Dimuthu Wannipurage < ***@***.***> wrote:
Good suggestion. I created the issue #94
<#94> to track this
enhancement
—
Reply to this email directly, view it on GitHub
<#93 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABYBRYIPW2MYR2IXRAEJWLW7RKJRANCNFSM6AAAAAAWNXYORA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
slimandslam commented
In my case, I had Java 11 installed, but Java 8 was the selected runtime (my bad). However, there were two things that made it harder to debug than it should have been. One thing was that the CLI wasn't responding with "cannot contact daemon" (or something similar). I guess that will be handled in #94
The other thing is that your script said:
Standalone Service stopped ...
Starting Standalone Service ...
Standalone Service started ...
MFT Started
But it almost certainly exited immediately after this. You can test it by running it with the Java 8 runtime.
Perhaps your script should check for the version of the Java runtime just before starting the
service since having multiple Java runtimes is not uncommon.
DImuthuUpe commented
These are very good suggestions @slimandslam. Especially checking the java runtime before starting the script is very important. Did you manage to start MFT after changing the java version to 11?
DImuthuUpe commented
@slimandslam I created another issue to track your suggestion. Please feel free to create issues you encounter on both the usability and operational level. We are actively working on improving the usability of the CLI as it is still at the very early stage of development
DImuthuUpe commented
Closing as the issue no longer appears. Please reopen if required