gnes-ai/gnes

Exemples of use cases

bratao opened this issue · 3 comments

bratao commented

Hi,
This is a VERY GREAT looking project. Thank you for your effort.

But from the documentation is not clear what kind of uses are possible with this project.
For example, in a text search, I would imagine that gnes would enable me to find similar documents. Is that right?

And how it compared with Vespa.ai (https://github.com/vespa-engine) ?

Some examples of what is possible with Gnes would be nice!

Ravikiran2611 commented

yeah some more examples with explanation of what is happening (how data is loaded and how model is loaded , how the path for model is specified )could be even more useful for beginners like me . Hope you take this asa a request
Thanks

  • 👍6
hanxiao commented

@bratao @Ravikiran2611 thanks again ❤️ for trying GNES and giving feedback at the early stage. I'd like to introduce you the new GNES Flow API (available since v0.0.46), enables a pythonic and intuitive way for building workflow in GNES. As an example, an indexing workflow can be simply defined as:

flow = (Flow(check_version=False, ctrl_with_ipc=True)
        .add_preprocessor(name='prep', yaml_path='yaml/prep.yml', replicas=3)
        .add_encoder(yaml_path='yaml/incep.yml', replicas=6)
        .add_indexer(name='vec_idx', yaml_path='yaml/vec.yml')
        .add_indexer(name='doc_idx', yaml_path='yaml/doc.yml', recv_from='prep')
        .add_router(name='sync', yaml_path='BaseReduceRouter', num_part=2, recv_from=['vec_idx', 'doc_idx']))

# then use it for indexing
with flow(backend='process') as fl:
    fl.index(bytes_gen=read_flowers(), batch_size=64)

🔰 You can find some resources here to help you getting started quickly:

🙇 Give it a try and we welcome your feedback and contribution.

  • 👍1
Ravikiran2611 commented

@hanxiao can i add a new model to gnes so that i can use it in a yaml file that can be specified in gnes.flow