modelscope/evalscope

evalscope perf --url 'our_url/v1/completions' --parallel 128 --model 'Qwen2-72B-Instruct' --log-every-n-query 10 --read-timeout=120 --dataset-path './data/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --stop '<|im_end|>' --dataset openqa --debug

zll0000 opened this issue · 7 comments

zll0000 commented

vllm 部署的服务
为啥请求不通
evalscope perf --url 'our_url/v1/completions' --parallel 128 --model 'Qwen2-72B-Instruct' --log-every-n-query 10 --read-timeout=120 --dataset-path './data/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --stop '<|im_end|>' --dataset openqa --debug
在本地机器上 利用vllm 启动服务利用下面命令可以正常

evalscope perf --url 'http://127.0.0.1:65000/v1/chat/completions' --parallel 128 --model 'Qwen2-72B-Instruct' --log-every-n-query 10 --read-timeout=120 --dataset-path './data/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --stop '<|im_end|>' --dataset openqa --debug`

liuyhwangyh commented

报错信息是什么？您网络是通的吗？

zll0000 commented

报错信息是什么？您网络是通的吗？

通的 利用下面代码请求相同的url 是通的
payload = {
"model": 'Qwen2-72B-Instruct',
"prompt": "",
"stream": False,
"temperature": 0.0,
#"top_k":-1,
#"top_p":1,
#"presence_penalty": 0.0,
#"frequency_penalty": 0.0,
"max_tokens": 2048,
#"stop": ["<|im_end|>"],
#'stop_token_ids': [7],
#"useSearch":False,
#"ignore_eos": True,
}
headers = {
"X-Trace-Id": recordId,
"Cache-Control": "no-cache",
"Accept": "text/event-stream",
'content-type': "application/json"
}
start_time = perf_counter()
first_token_time = 0
with httpx.stream("POST", url=url, data=json.dumps(payload), headers=headers, timeout=9999) as r:
i = 0
for text in r.iter_text():
if i == 0:
first_token_time = perf_counter() - start_time
i += 1
try:
t = json.loads(text)
except BaseException as e:
print('The file contains invalid JSON')
pass
return json.loads(text)["choices"][0]["text"]

liuyhwangyh commented

报错信息方便贴下吗？

zll0000 commented

报错信息方便贴下吗？

evalscope perf --url 'http://andesinfer-api-2.local/api//docqa_translate/v1/completions' --parallel 128 --model 'Qwen2-72B-Instruct' --log-every-n-query 10 --read-timeout=120 --dataset-path './data/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --stop '<|im_end|>' --dataset openqa --debug
Save the result to : /home/notebook/data/group/zhangxiaolei/vllm-server/eval-scope/Qwen2-72B-Instruct_benchmark_2024_08_07_08_36_21_074402.db
2024-08-07 08:36:21,088 - perf - http_client.py - on_request_start - 54 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://our_url/docqa_translate/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench')>)>
2024-08-07 08:36:21,146 - perf - http_client.py - on_request_chunk_sent - 58 - DEBUG - Request body: TraceRequestChunkSentParams(method='POST', url=URL('http:/our_url/v1/completions'), chunk=b'{"messages": [{"role": "user", "content": "\u76d7\u8d3c\u5929\u8d4b\u76d7\u8d3c\u600e\u4e48\u52a0\u5929\u8d4b?\u77e5\u9053\u544a\u8bc9\u4e00\u4e0b\u4e0b\u5566~~"}], "model": "Qwen2-72B-Instruct", "stop": ["<|im_end|>"], "stream": true, "stream_options": {"include_usage": true}}')
2024-08-07 08:36:21,149 - perf - http_client.py - on_response_chunk_received - 62 - DEBUG - Response info: <TraceResponseChunkReceivedParams(method='POST', url=URL('our_url/v1/completions'), chunk=b'{"object":"error","message":"[{'type': 'missing', 'loc': ('body', 'prompt'), 'msg': 'Field required', 'input': {'messages': [{'content': '\xe7\x9b\x97\xe8\xb4\xbc\xe5\xa4\xa9\xe8\xb5\x8b\xe7\x9b\x97\xe8\xb4\xbc\xe6\x80\x8e\xe4\xb9\x88\xe5\x8a\xa0\xe5\xa4\xa9\xe8\xb5\x8b?\xe7\x9f\xa5\xe9\x81\x93\xe5\x91\x8a\xe8\xaf\x89\xe4\xb8\x80\xe4\xb8\x8b\xe4\xb8\x8b\xe5\x95\xa6~~', 'role': 'user'}], 'model': 'Qwen2-72B-Instruct', 'stop': ['<|im_end|>'], 'stream': True}}]","type":"BadRequestError","param":null,"code":400}')>
2024-08-07 08:36:21,149 - perf - http_client.py - send_requests_worker - 564 - ERROR - Request: {'messages': [{'role': 'user', 'content': '盗贼天赋盗贼怎么加天赋?知道告诉一下下啦~~'}], 'model': 'Qwen2-72B-Instruct', 'stop': ['<|im_end|>'], 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'missing', 'loc': ('body', 'prompt'), 'msg': 'Field required', 'input': {'messages': [{'content': '\u76d7\u8d3c\u5929\u8d4b\u76d7\u8d3c\u600e\u4e48\u52a0\u5929\u8d4b?\u77e5\u9053\u544a\u8bc9\u4e00\u4e0b\u4e0b\u5566~~', 'role': 'user'}], 'model': 'Qwen2-72B-Instruct', 'stop': ['<|im_end|>'], 'stream': True}}]", "type": "BadRequestError", "param": null, "code": 400}
Benchmarking summary:
Time taken for tests: 1.001 seconds
Expected number of requests: 1
Number of concurrency: 128
Total requests: 1
Succeed requests: 0
Failed requests: 1
Average QPS: 0.000
Average latency: -1.000
Throughput(average output tokens per second): -1.000
Average time to first token: -1.000
Average input tokens per request: -1.000
Average output tokens per request: -1.000
Average time per output token: -1.00000
Average package per request: -1.000
Average package latency: -1.000
Too little data to calculate quantiles!

liuyhwangyh commented

辛苦看下：--url 'http://andesinfer-api-2.local/api//docqa_translate/v1/completions' 为什么变成 url=URL('http://our_url/docqa_translate/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench')>)>

zll0000 commented

辛苦看下：--url 'http://andesinfer-api-2.local/api//docqa_translate/v1/completions' 为什么变成 url=URL('http://our_url/docqa_translate/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench')>)>

改回取是一样的
evalscope perf --url 'http://andesinfer-api-2.oppo.local/api/xiaobu/docqa_translate/v1/completions' --parallel 128 --model 'Qwen2-72B-Instruct' --log-every-n-query 10 --read-timeout=120 --dataset-path './data/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --stop '<|im_end|>' --dataset openqa --debug
Save the result to : /home/notebook/data/group/zhangxiaolei/vllm-server/eval-scope/Qwen2-72B-Instruct_benchmark_2024_08_07_08_36_21_074402.db
2024-08-07 08:36:21,088 - perf - http_client.py - on_request_start - 54 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://andesinfer-api-2.oppo.local/api/xiaobu/docqa_translate/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench')>)>
2024-08-07 08:36:21,146 - perf - http_client.py - on_request_chunk_sent - 58 - DEBUG - Request body: TraceRequestChunkSentParams(method='POST', url=URL('http://andesinfer-api-2.oppo.local/api/xiaobu/docqa_translate/v1/completions'), chunk=b'{"messages": [{"role": "user", "content": "\u76d7\u8d3c\u5929\u8d4b\u76d7\u8d3c\u600e\u4e48\u52a0\u5929\u8d4b?\u77e5\u9053\u544a\u8bc9\u4e00\u4e0b\u4e0b\u5566~~"}], "model": "Qwen2-72B-Instruct", "stop": ["<|im_end|>"], "stream": true, "stream_options": {"include_usage": true}}')
2024-08-07 08:36:21,149 - perf - http_client.py - on_response_chunk_received - 62 - DEBUG - Response info: <TraceResponseChunkReceivedParams(method='POST', url=URL('http://andesinfer-api-2.oppo.local/api/xiaobu/docqa_translate/v1/completions'), chunk=b'{"object":"error","message":"[{'type': 'missing', 'loc': ('body', 'prompt'), 'msg': 'Field required', 'input': {'messages': [{'content': '\xe7\x9b\x97\xe8\xb4\xbc\xe5\xa4\xa9\xe8\xb5\x8b\xe7\x9b\x97\xe8\xb4\xbc\xe6\x80\x8e\xe4\xb9\x88\xe5\x8a\xa0\xe5\xa4\xa9\xe8\xb5\x8b?\xe7\x9f\xa5\xe9\x81\x93\xe5\x91\x8a\xe8\xaf\x89\xe4\xb8\x80\xe4\xb8\x8b\xe4\xb8\x8b\xe5\x95\xa6~~', 'role': 'user'}], 'model': 'Qwen2-72B-Instruct', 'stop': ['<|im_end|>'], 'stream': True}}]","type":"BadRequestError","param":null,"code":400}')>
2024-08-07 08:36:21,149 - perf - http_client.py - send_requests_worker - 564 - ERROR - Request: {'messages': [{'role': 'user', 'content': '盗贼天赋盗贼怎么加天赋?知道告诉一下下啦~~'}], 'model': 'Qwen2-72B-Instruct', 'stop': ['<|im_end|>'], 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'missing', 'loc': ('body', 'prompt'), 'msg': 'Field required', 'input': {'messages': [{'content': '\u76d7\u8d3c\u5929\u8d4b\u76d7\u8d3c\u600e\u4e48\u52a0\u5929\u8d4b?\u77e5\u9053\u544a\u8bc9\u4e00\u4e0b\u4e0b\u5566~~', 'role': 'user'}], 'model': 'Qwen2-72B-Instruct', 'stop': ['<|im_end|>'], 'stream': True}}]", "type": "BadRequestError", "param": null, "code": 400}
Benchmarking summary:
Time taken for tests: 1.001 seconds
Expected number of requests: 1
Number of concurrency: 128
Total requests: 1
Succeed requests: 0
Failed requests: 1
Average QPS: 0.000
Average latency: -1.000
Throughput(average output tokens per second): -1.000
Average time to first token: -1.000
Average input tokens per request: -1.000
Average output tokens per request: -1.000
Average time per output token: -1.00000
Average package per request: -1.000
Average package latency: -1.000
Too little data to calculate quantiles!

liuyhwangyh commented

state_code: 400, data: {"object": "error", "message": "[{'type': 'missing', 'loc': ('body', 'prompt'), 'msg': 'Field required', 'input': {'messages': [{'content': '\u76d7\u8d3c\u5929\u8d4b\u76d7\u8d3c\u600e\u4e48\u52a0\u5929\u8d4b?\u77e5\u9053\u544a\u8bc9\u4e00\u4e0b\u4e0b\u5566~~', 'role': 'user'}], 'model': 'Qwen2-72B-Instruct', 'stop': ['<|im_end|>'], 'stream': True}}]", "type": "BadRequestError", "param": null, "code": 400}
看起来是服务端问题，先用curl试下是否通