Error while running examples on README
timoderbeste opened this issue · 3 comments
I am trying to run GNES locally on my Linux machine. However, when I run bash run.sh
, I received the following errors.
E:EncoderService:[bas:run:305]:could not determine a constructor for the tag '!GPT2Encoder'
in "gpt2.yml", line 3, column 5
Namely, there is something wrong with gpt2.yml. I replaced the value of model_path with a real path to a downloaded GPT2 model, but it still did not work.
I installed gnes and also was running the docker service. Is there anything else we need to do to get it working? Thanks!
Hi, thanks again ❤️ for trying GNES and giving feedback at the early stage. I'd like to introduce you the new GNES Flow API (available since v0.0.46
), enables a pythonic and intuitive way for building workflow in GNES. As an example, an indexing workflow can be simply defined as:
flow = (Flow(check_version=False, ctrl_with_ipc=True)
.add_preprocessor(name='prep', yaml_path='yaml/prep.yml', replicas=3)
.add_encoder(yaml_path='yaml/incep.yml', replicas=6)
.add_indexer(name='vec_idx', yaml_path='yaml/vec.yml')
.add_indexer(name='doc_idx', yaml_path='yaml/doc.yml', recv_from='prep')
.add_router(name='sync', yaml_path='BaseReduceRouter', num_part=2, recv_from=['vec_idx', 'doc_idx']))
# then use it for indexing
with flow(backend='process') as fl:
fl.index(bytes_gen=read_flowers(), batch_size=64)
🔰 You can find some resources here to help you getting started quickly:
- Image search demo using GNES Flow, Jupyter Notebook
- GNES Flow module documentation
- Getting started in GNES readme
🙇 Give it a try and we welcome your feedback and contribution.
Hi @hanxiao , Thanks for this great contribution.
I was running the very initial sample (flower classification via query).
I did pip install only. I ran in terminal (not in jupyter-notebook).
I came up with this error,
>>> flow = (Flow(check_version=False)
... .add_preprocessor(name='prep', yaml_path='yaml/prep.yml')
... .add_encoder(yaml_path='yaml/incep.yml')
... .add_indexer(name='vec_idx', yaml_path='yaml/vec.yml')
... .add_indexer(name='doc_idx', yaml_path='yaml/doc.yml', recv_from='prep')
... .add_router(name='sync', yaml_path='BaseReduceRouter', num_part=2, recv_from=['vec_idx', 'doc_idx']))
W:Flow:[__i:_po:173]:this object is not named ("name" is not found under "gnes_config" in YAML config), i will call it "Flow-e96c35ae". naming the object is important as it provides an unique identifier when serializing/deserializing this object.
usage: [-h] [-v] [--verbose] [--port_in PORT_IN] [--port_out PORT_OUT]
[--host_in HOST_IN] [--host_out HOST_OUT]
[--socket_in {PULL_BIND,PULL_CONNECT,PUSH_BIND,PUSH_CONNECT,SUB_BIND,SUB_CONNECT,PUB_BIND,PUB_CONNECT,PAIR_BIND,PAIR_CONNECT}]
[--socket_out {PULL_BIND,PULL_CONNECT,PUSH_BIND,PUSH_CONNECT,SUB_BIND,SUB_CONNECT,PUB_BIND,PUB_CONNECT,PAIR_BIND,PAIR_CONNECT}]
[--port_ctrl PORT_CTRL] [--timeout TIMEOUT]
[--dump_interval DUMP_INTERVAL] [--read_only]
[--parallel_backend {thread,process}] [--num_parallel NUM_PARALLEL]
[--parallel_type {PUSH_BLOCK,PUSH_NONBLOCK,PUB_BLOCK,PUB_NONBLOCK}]
[--check_version] [--identity IDENTITY] [--route_table] [--squeeze_pb]
[--ctrl_with_ipc] --yaml_path YAML_PATH
[--py_path PY_PATH [PY_PATH ...]]
: error: argument --yaml_path: yaml/prep.yml can not be resolved, it should be a readable stream, or a valid file path, or a supported class name.
Traceback (most recent call last):
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2294, in _get_value
result = type_func(arg_string)
File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/cli/parser.py", line 63, in resolve_yaml_path
' or a valid file path, or a supported class name.' % path)
argparse.ArgumentTypeError: yaml/prep.yml can not be resolved, it should be a readable stream, or a valid file path, or a supported class name.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1766, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1972, in _parse_known_args
start_index = consume_optional(start_index)
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1912, in consume_optional
take_action(action, args, option_string)
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1824, in take_action
argument_values = self._get_values(action, argument_strings)
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2265, in _get_values
value = self._get_value(action, arg_string)
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2300, in _get_value
raise ArgumentError(action, msg)
argparse.ArgumentError: argument --yaml_path: yaml/prep.yml can not be resolved, it should be a readable stream, or a valid file path, or a supported class name.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/flow/__init__.py", line 707, in _get_parsed_args
p_args, unknown_args = service_arg_parser().parse_known_args(args)
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1773, in parse_known_args
self.error(str(err))
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2393, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2380, in exit
_sys.exit(status)
SystemExit: 2
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/flow/__init__.py", line 444, in add_preprocessor
return self.add(Service.Preprocessor, *args, **kwargs)
File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/flow/__init__.py", line 638, in add
args, p_args, unk_args = op_flow._get_parsed_args(op_flow, name, service_map[service]['parser'], kwargs)
File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/flow/__init__.py", line 712, in _get_parsed_args
'you may want to double check your args "%s"' % (name, args))
ValueError: bad arguments for service "prep", you may want to double check your args "['--yaml_path', 'yaml/prep.yml', '--no_check_version']"
Some sort of arg error in yml file? is it that i missed to install some packages? because i just install with pip installl gnes
but not pip install gnes[all]
opps, my bad. I was running under different repo, so it couldn't find the yml files under yaml directory.
Can ignore the my above comment