Q-Future/Q-Instruct

http api test

xu455255849 opened this issue · 3 comments

xu455255849 commented

if i clone the repo , how can i export http api to the web to use

teowu commented

Hi,

You may need to deploy it somewhere on cloud gpu providers (which also supports https API). There are several third-party providers available (e.g. runpod.io, studio.lightning.ai) at present. Or, the simplest way is to deploy an inference endpoint on Hugging Face.

Best Regards
Haoning

xu455255849 commented

My current approach: I have cloned the project on a machine, and successfully run the local demo instance using Python 3. However, when attempting to access it through the web page, I am getting a 500 Internal Server Error. I am now looking to deploy it and make HTTP API calls. How should I proceed with this?
///////// error stderr/ ////////
2023-12-18 03:03:33 | ERROR | stderr |
Some weights of MPLUGOwl2LlamaForCausalLM were not initialized from the model checkpoint at ../../../mplug_owl_2_qinstruct_preview_v0.1 and are newly initialized: ['model.visual_abstractor.encoder.layers.0.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.2.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.1.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.0.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.5.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.3.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.4.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.4.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.3.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.1.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.5.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.2.crossattention.attention.k_pos_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2023-12-18 03:03:33 | INFO | gradio_web_server_local | Namespace(host='0.0.0.0', port=7860, concurrency_count=10, model_list_mode='once', model_path='../../../mplug_owl_2_qinstruct_preview_v0.1/', device='cuda', load_8bit=False, load_4bit=False, moderate=False, embed=False)
2023-12-18 03:03:35 | INFO | stdout | Running on local URL: http://0.0.0.0:7860
2023-12-18 03:03:44 | INFO | gradio_web_server_local | load_demo. ip: 192.168.1.174. params: None
2023-12-18 03:03:58 | INFO | gradio_web_server_local | add_text. ip: 192.168.1.174. len: 30
2023-12-18 03:03:58 | INFO | stdout | ('<|image|>Rate the quality of the image.', <PIL.Image.Image image mode=RGB size=960x540 at 0x7F1F76723A90>, 'Default')
2023-12-18 03:03:58 | ERROR | stderr | ERROR: Exception in ASGI application
2023-12-18 03:03:58 | ERROR | stderr | Traceback (most recent call last):
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
2023-12-18 03:03:58 | ERROR | stderr | result = await app( # type: ignore[func-returns-value]
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
2023-12-18 03:03:58 | ERROR | stderr | return await self.app(scope, receive, send)
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1106, in call
2023-12-18 03:03:58 | ERROR | stderr | await super().call(scope, receive, send)
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 122, in call
2023-12-18 03:03:58 | ERROR | stderr | await self.middleware_stack(scope, receive, send)
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in call
2023-12-18 03:03:58 | ERROR | stderr | raise exc
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in call
2023-12-18 03:03:58 | ERROR | stderr | await self.app(scope, receive, _send)
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 91, in call
2023-12-18 03:03:58 | ERROR | stderr | await self.simple_response(scope, receive, send, request_headers=headers)
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 146, in simple_response
2023-12-18 03:03:58 | ERROR | stderr | await self.app(scope, receive, send)
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 79, in call
2023-12-18 03:03:58 | ERROR | stderr | raise exc
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 68, in call
2023-12-18 03:03:58 | ERROR | stderr | await self.app(scope, receive, sender)
2023-12-18 03:03:58 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 20, in call
2023-12-18 03:03:58 | ERROR | stderr | raise e
.....

xu455255849 commented

i has resolved， my gradio version is low ，upgrade is OK