codefuse-ai/codefuse-chatbot

gpt-4 chatbot timeout!

maxenergy opened this issue · 2 comments

maxenergy commented

Checked other resources

  • I searched the Codefuse documentation with the integrated search.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in Codefuse-Repos rather than my code.
  • I added a very descriptive title to this issue.

System Info

Windows

Code Version

Latest Release

Description

python start.py
2024-02-02 14:13:14.623 | INFO | main:start_sandbox_service:118 - start container sandbox service
2024-02-02 14:13:14.994 | INFO | main:start_docker:81 - docker id: 7473979b00
2024-02-02 14:13:14.994 | DEBUG | main:start_docker:90 - cd /home/user/chatbot/jupyter_work && nohup jupyter-notebook --NotebookApp.token=mytoken --port=5050 --allow-root --ip=0.0.0.0 --notebook-dir=/home/user/chatbot/jupyter_work --no-browser --ServerApp.disable_check_xsrf=True &
2024-02-02 14:13:17.041 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b"[W 2024-02-02 06:13:15.626 ServerApp] A _jupyter_server_extension_points function was not found in notebook_shim. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.\n[I 2024-02-02 06:13:15.627 ServerApp] jupyter_lsp | extension was successfully linked.\n[I 2024-02-02 06:13:15.630 ServerApp] jupyter_server_terminals | extension was successfully linked.\n[W 2024-02-02 06:13:15.631 LabApp] 'token' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.\n[W 2024-02-02 06:13:15.634 ServerApp] ServerApp.token config is deprecated in 2.0. Use IdentityProvider.token.\n[I 2024-02-02 06:13:15.634 ServerApp] jupyterlab | extension was successfully linked.\n[I 2024-02-02 06:13:15.637 ServerApp] notebook | extension was successfully linked.\n[I 2024-02-02 06:13:15.639 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret\n[I 2024-02-02 06:13:15.872 ServerApp] notebook_shim | extension was successfully linked.\n[I 2024-02-02 06:13:15.887 ServerApp] notebook_shim | extension was successfully loaded.\n[I 2024-02-02 06:13:15.891 ServerApp] jupyter_lsp | extension was successfully loaded.\n[I 2024-02-02 06:13:15.892 ServerApp] jupyter_server_terminals | extension was successfully loaded.\n[I 2024-02-02 06:13:15.893 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.9/site-packages/jupyterlab\n[I 2024-02-02 06:13:15.893 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab\n[I 2024-02-02 06:13:15.894 LabApp] Extension Manager is 'pypi'.\n[I 2024-02-02 06:13:15.896 ServerApp] jupyterlab | extension was successfully loaded.\n[I 2024-02-02 06:13:15.898 ServerApp] notebook | extension was successfully loaded.\n[I 2024-02-02 06:13:15.898 ServerApp] Serving notebooks from local directory: /home/user/chatbot/jupyter_work\n[I 2024-02-02 06:13:15.898 ServerApp] Jupyter Server 2.12.5 is running at:\n[I 2024-02-02 06:13:15.898 ServerApp] http://7473979b0007:5050/tree?token=...\n[I 2024-02-02 06:13:15.898 ServerApp] http://127.0.0.1:5050/tree?token=...\n[I 2024-02-02 06:13:15.898 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).\n[I 2024-02-02 06:13:15.926 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server\n")
2024-02-02 14:13:22.047 | INFO | main:start_sandbox_service:128 - http://localhost:5050
2024-02-02 14:13:22.069 | INFO | main:start_sandbox_service:131 - container & notebook init success
2024-02-02 14:13:22.100 | INFO | main:start_api_service:151 - start container service
2024-02-02 14:13:22.600 | INFO | main:start_docker:81 - docker id: 540c1c697c
2024-02-02 14:13:22.600 | DEBUG | main:start_docker:90 - mkdir -p /home/user/logs
2024-02-02 14:13:22.668 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'')
2024-02-02 14:13:22.669 | DEBUG | main:start_docker:90 -
if [ -d "/home/user/chatbot/data/nebula_data/data/meta" ]; then
cp -r /home/user/chatbot/data/nebula_data/data /usr/local/nebula/
fi

2024-02-02 14:13:22.729 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'')
2024-02-02 14:13:22.729 | DEBUG | main:start_docker:90 - /usr/local/nebula/scripts/nebula.service start all
2024-02-02 14:13:22.834 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'[INFO] Starting nebula-metad...\n[INFO] Done\n[INFO] Starting nebula-graphd...\n[INFO] Done\n[INFO] Starting nebula-storaged...\n[INFO] Done\n')
2024-02-02 14:13:22.835 | DEBUG | main:start_docker:90 - /usr/local/nebula/scripts/nebula.service status all
2024-02-02 14:13:22.965 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'[INFO] nebula-metad(de9b3ed): Running as 42, Listening on 9559\n[INFO] nebula-graphd(de9b3ed): Running as 65, Listening on 9669\n[INFO] nebula-storaged(de9b3ed): Running as 94, Listening on 9779\n')
2024-02-02 14:13:22.965 | DEBUG | main:start_docker:90 - sleep 2
2024-02-02 14:13:25.017 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'')
2024-02-02 14:13:25.017 | DEBUG | main:start_docker:90 - curl -X PUT -H "Content-Type: application/json" -d'{"heartbeat_interval_secs":"2"}' -s "http://127.0.0.1:19559/flags"
2024-02-02 14:13:25.078 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'{\n "errCode": 0\n}')
2024-02-02 14:13:25.078 | DEBUG | main:start_docker:90 - curl -X PUT -H "Content-Type: application/json" -d'{"heartbeat_interval_secs":"2"}' -s "http://127.0.0.1:19669/flags"
2024-02-02 14:13:25.126 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'{\n "errCode": 0\n}')
2024-02-02 14:13:25.126 | DEBUG | main:start_docker:90 - curl -X PUT -H "Content-Type: application/json" -d'{"heartbeat_interval_secs":"2"}' -s "http://127.0.0.1:19779/flags"
2024-02-02 14:13:25.170 | DEBUG | main:start_docker:92 - ExecResult(exit_code=7, output=b'')
2024-02-02 14:13:25.171 | DEBUG | main:start_docker:90 - pip install zdatafront-sdk-python==0.1.2 -i https://artifacts.antgroup-inc.cn/simple

2024-02-02 14:14:03.381 | DEBUG | main:start_docker:92 - ExecResult(exit_code=1, output=b"Looking in indexes: https://artifacts.antgroup-inc.cn/simple\nWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))': /simple/zdatafront-sdk-python/\nWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))': /simple/zdatafront-sdk-python/\nWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))': /simple/zdatafront-sdk-python/\nWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))': /simple/zdatafront-sdk-python/\nWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))': /simple/zdatafront-sdk-python/\nCould not fetch URL https://artifacts.antgroup-inc.cn/simple/zdatafront-sdk-python/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='artifacts.antgroup-inc.cn', port=443): Max retries exceeded with url: /simple/zdatafront-sdk-python/ (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))) - skipping\nERROR: Could not find a version that satisfies the requirement zdatafront-sdk-python==0.1.2 (from versions: none)\nERROR: No matching distribution found for zdatafront-sdk-python==0.1.2\n\n[notice] A new release of pip is available: 23.0.1 -> 23.3.2\n[notice] To update, run: pip install --upgrade pip\n")
2024-02-02 14:14:03.382 | DEBUG | main:start_docker:90 - pip install jieba
2024-02-02 14:14:05.898 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b"Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\nRequirement already satisfied: jieba in /usr/local/lib/python3.9/site-packages (0.42.1)\nWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n\n[notice] A new release of pip is available: 23.0.1 -> 23.3.2\n[notice] To update, run: pip install --upgrade pip\n")
2024-02-02 14:14:05.898 | DEBUG | main:start_docker:90 - pip install duckduckgo-search
2024-02-02 14:14:08.428 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b"Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\nRequirement already satisfied: duckduckgo-search in /usr/local/lib/python3.9/site-packages (4.4)\nRequirement already satisfied: lxml>=4.9.3 in /usr/local/lib/python3.9/site-packages (from duckduckgo-search) (4.9.4)\nRequirement already satisfied: click>=8.1.7 in /usr/local/lib/python3.9/site-packages (from duckduckgo-search) (8.1.7)\nRequirement already satisfied: nest-asyncio>=1.6.0 in /usr/local/lib/python3.9/site-packages (from duckduckgo-search) (1.6.0)\nRequirement already satisfied: curl-cffi>=0.6.0b7 in /usr/local/lib/python3.9/site-packages (from duckduckgo-search) (0.6.0b9)\nRequirement already satisfied: docstring-inheritance>=2.1.2 in /usr/local/lib/python3.9/site-packages (from duckduckgo-search) (2.1.2)\nRequirement already satisfied: cffi>=1.12.0 in /usr/local/lib/python3.9/site-packages (from curl-cffi>=0.6.0b7->duckduckgo-search) (1.16.0)\nRequirement already satisfied: certifi in /usr/local/lib/python3.9/site-packages (from curl-cffi>=0.6.0b7->duckduckgo-search) (2023.11.17)\nRequirement already satisfied: pycparser in /usr/local/lib/python3.9/site-packages (from cffi>=1.12.0->curl-cffi>=0.6.0b7->duckduckgo-search) (2.21)\nWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n\n[notice] A new release of pip is available: 23.0.1 -> 23.3.2\n[notice] To update, run: pip install --upgrade pip\n")
2024-02-02 14:14:08.429 | DEBUG | main:start_docker:90 - nohup python chatbot/examples/sdfile_api.py > /home/user/logs/sdfile_api.log 2>&1 &
2024-02-02 14:14:08.478 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'')
2024-02-02 14:14:08.478 | DEBUG | main:start_docker:90 - export DUCKDUCKGO_PROXY=socks5://host.docker.internal:13659 && export SANDBOX_HOST=172.18.0.2 && nohup python chatbot/examples/api.py > /home/user/logs/api.log 2>&1 &
2024-02-02 14:14:08.526 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'')
2024-02-02 14:14:08.527 | DEBUG | main:start_docker:90 - export DUCKDUCKGO_PROXY=socks5://host.docker.internal:13659 && export SANDBOX_HOST=172.18.0.2 && cd chatbot/examples && nohup streamlit run webui.py > /home/user/logs/start_webui.log 2>&1 &
2024-02-02 14:14:08.580 | DEBUG | main:start_docker:92 - ExecResult(exit_code=0, output=b'')
image

Example Code

python start.py

Error Message and Stack Trace (if applicable)

聊天界面没有任何返回信息，控制台也没有错误信息：
model_config.py
import os
import sys
import logging
import torch
import openai
import base64
from .utils import is_running_in_docker
from .default_config import *

日志格式

LOG_FORMAT = "%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s"
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging.basicConfig(format=LOG_FORMAT)

VERSION = "v0.1.0"

import platform
system_name = platform.system()

try:
# ignore these content
from zdatafront import client, monkey, OPENAI_API_BASE
# patch openai sdk
monkey.patch_openai()
secret_key = base64.b64decode('xx').decode('utf-8')
# zdatafront 提供的统一加密密钥
client.aes_secret_key = secret_key
# zdatafront 分配的业务标记
client.visit_domain = os.environ.get("visit_domain")
client.visit_biz = os.environ.get("visit_biz")
client.visit_biz_line = os.environ.get("visit_biz_line")
except:
pass

add your openai key

OPENAI_API_BASE = "http://openai.com/v1"
os.environ["API_BASE_URL"] = OPENAI_API_BASE
os.environ["OPENAI_API_KEY"] = "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"
openai.api_key = "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"

os.environ["OPENAI_PROXY"] = "socks5h://127.0.0.1:13659"

os.environ["DUCKDUCKGO_PROXY"] = os.environ.get("DUCKDUCKGO_PROXY") or "socks5://127.0.0.1:13659"

ignore if you dont's use baidu_ocr_api

os.environ["BAIDU_OCR_API_KEY"] = "xx"
os.environ["BAIDU_OCR_SECRET_KEY"] = "xx"

os.environ["log_verbose"] = "2"

LLM 名称

EMBEDDING_ENGINE = 'model' # openai or model
EMBEDDING_MODEL = "text2vec-base"
LLM_MODEL = "gpt-4-1106-preview"
LLM_MODELs = ["gpt-4-1106-preview"]
USE_FASTCHAT = "gpt" not in LLM_MODEL # 判断是否进行fastchat

LLM 运行设备

LLM_DEVICE = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"

在以下字典中修改属性值，以指定本地embedding模型存储位置

如将 "text2vec": "GanymedeNil/text2vec-large-chinese" 修改为 "text2vec": "User/Downloads/text2vec-large-chinese"

此处请写绝对路径

embedding_model_dict = {
"ernie-tiny": "nghuyong/ernie-3.0-nano-zh",
"ernie-base": "nghuyong/ernie-3.0-base-zh",
"text2vec-base": "shibing624/text2vec-base-chinese",
"text2vec": "GanymedeNil/text2vec-large-chinese",
"text2vec-paraphrase": "shibing624/text2vec-base-chinese-paraphrase",
"text2vec-sentence": "shibing624/text2vec-base-chinese-sentence",
"text2vec-multilingual": "shibing624/text2vec-base-multilingual",
"m3e-small": "moka-ai/m3e-small",
"m3e-base": "moka-ai/m3e-base",
"m3e-large": "moka-ai/m3e-large",
"bge-small-zh": "BAAI/bge-small-zh",
"bge-base-zh": "BAAI/bge-base-zh",
"bge-large-zh": "BAAI/bge-large-zh"
}

LOCAL_MODEL_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(file))), "embedding_models")
embedding_model_dict = {k: f"/home/rogers/chatbot/embedding_models/{v}" if is_running_in_docker() else f"{LOCAL_MODEL_DIR}/{v}" for k, v in embedding_model_dict.items()}

Embedding 模型运行设备

EMBEDDING_DEVICE = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"

ONLINE_LLM_MODEL = {
# 线上模型。请在server_config中为每个在线API设置不同的端口

"openai-api": {
    "model_name": "gpt-4-1106-preview",
    "api_base_url": "https://api.openai.com/v1",
    "api_key": "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",
    "openai_proxy": "",
},
"example": {
    "version": "gpt-4-1106-preview",  # 采用openai接口做示例
    "api_base_url": "https://api.openai.com/v1",
    "api_key": "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",
    "provider": "ExampleWorker",
},

}

建议使用chat模型，不要使用base，无法获取正确输出

llm_model_dict = {
"chatglm-6b": {
"local_model_path": "THUDM/chatglm-6b",
"api_base_url": "http://localhost:8888/v1", # "name"修改为fastchat服务中的"api_base_url"
"api_key": "EMPTY"
},
# 以下模型经过测试可接入，配置仿照上述即可
# 'codellama_34b', 'Baichuan2-13B-Base', 'Baichuan2-13B-Chat', 'baichuan2-7b-base', 'baichuan2-7b-chat',
# 'internlm-7b-base', 'internlm-chat-7b', 'chatglm2-6b', 'qwen-14b-base', 'qwen-14b-chat', 'qwen-1-8B-Chat',
# 'Qwen-7B', 'Qwen-7B-Chat', 'qwen-7b-base-v1.1', 'qwen-7b-chat-v1.1', 'chatglm3-6b', 'chatglm3-6b-32k',
# 'chatglm3-6b-base', 'Qwen-72B-Chat-Int4'
# 调用chatgpt时如果报出： urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.openai.com', port=443):
# Max retries exceeded with url: /v1/chat/completions
# 则需要将urllib3版本修改为1.25.11
# 如果依然报urllib3.exceptions.MaxRetryError: HTTPSConnectionPool，则将https改为http
# 参考https://zhuanlan.zhihu.com/p/350015032

# 如果报出：raise NewConnectionError(
# urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x000001FE4BDB85E0>:
# Failed to establish a new connection: [WinError 10060]
# 则是因为内地和香港的IP都被OPENAI封了，需要切换为日本、新加坡等地
"gpt-3.5-turbo": {
    "local_model_path": "gpt-3.5-turbo",
    "api_base_url": os.environ.get("API_BASE_URL"),
    "api_key": os.environ.get("OPENAI_API_KEY")
},
"gpt-3.5-turbo-16k": {
    "local_model_path": "gpt-3.5-turbo-16k",
    "api_base_url": os.environ.get("API_BASE_URL"),
    "api_key": os.environ.get("OPENAI_API_KEY")
},
"gpt-3.5-turbo-0613": {
    "local_model_path": "gpt-3.5-turbo-0613",
    "api_base_url": os.environ.get("API_BASE_URL"),
    "api_key": os.environ.get("OPENAI_API_KEY")
},
"gpt-4": {
    "local_model_path": "gpt-4",
    "api_base_url": os.environ.get("API_BASE_URL"),
    "api_key": os.environ.get("OPENAI_API_KEY")
},
"gpt-4-1106-preview": {
    "local_model_path": "gpt-4",
    "api_base_url": os.environ.get("API_BASE_URL"),
    "api_key": os.environ.get("OPENAI_API_KEY")
},
"gpt-3.5-turbo-1106": {
    "local_model_path": "gpt-3.5-turbo-1106",
    "api_base_url": os.environ.get("API_BASE_URL"),
    "api_key": os.environ.get("OPENAI_API_KEY")
},

}

建议使用chat模型，不要使用base，无法获取正确输出

VLLM_MODEL_DICT = {
'chatglm2-6b': "THUDM/chatglm-6b",
}

以下模型经过测试可接入，配置仿照上述即可

'codellama_34b', 'Baichuan2-13B-Base', 'Baichuan2-13B-Chat', 'baichuan2-7b-base', 'baichuan2-7b-chat',

'internlm-7b-base', 'internlm-chat-7b', 'chatglm2-6b', 'qwen-14b-base', 'qwen-14b-chat', 'qwen-1-8B-Chat',

'Qwen-7B', 'Qwen-7B-Chat', 'qwen-7b-base-v1.1', 'qwen-7b-chat-v1.1', 'chatglm3-6b', 'chatglm3-6b-32k',

'chatglm3-6b-base', 'Qwen-72B-Chat-Int4'

LOCAL_LLM_MODEL_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(file))), "llm_models")

模型路径重置

llm_model_dict_c = {}
for k, v in llm_model_dict.items():
v_c = {}
for kk, vv in v.items():
if k=="local_model_path":
v_c[kk] = f"/home/user/chatbot/llm_models/{vv}" if is_running_in_docker() else f"{LOCAL_LLM_MODEL_DIR}/{vv}"
else:
v_c[kk] = vv
llm_model_dict_c[k] = v_c

llm_model_dict = llm_model_dict_c

VLLM_MODEL_DICT_c = {}
for k, v in VLLM_MODEL_DICT.items():
VLLM_MODEL_DICT_c[k] = f"/home/user/chatbot/llm_models/{v}" if is_running_in_docker() else f"{LOCAL_LLM_MODEL_DIR}/{v}"
VLLM_MODEL_DICT = VLLM_MODEL_DICT_c

server_config.py:
from .model_config import LLM_MODEL, LLM_DEVICE
import os

API 是否开启跨域，默认为False，如果需要开启，请设置为True

is open cross domain

OPEN_CROSS_DOMAIN = False

是否用容器来启动服务

DOCKER_SERVICE = True

是否采用容器沙箱

SANDBOX_DO_REMOTE = True

是否采用api服务来进行

NO_REMOTE_API = True

各服务器默认绑定host

DEFAULT_BIND_HOST = "127.0.0.1"
os.environ["DEFAULT_BIND_HOST"] = DEFAULT_BIND_HOST

CONTRAINER_NAME = "devopsgpt_webui"
IMAGE_NAME = "devopsgpt:py39"

webui.py server

WEBUI_SERVER = {
"host": DEFAULT_BIND_HOST,
"port": 8501,
"docker_port": 8501
}

api.py server

API_SERVER = {
"host": DEFAULT_BIND_HOST,
"port": 7861,
"docker_port": 7861
}

sdfile_api.py server

SDFILE_API_SERVER = {
"host": DEFAULT_BIND_HOST,
"port": 7862,
"docker_port": 7862
}

fastchat openai_api server

FSCHAT_OPENAI_API = {
"host": DEFAULT_BIND_HOST,
"port": 8888, # model_config.llm_model_dict中模型配置的api_base_url需要与这里一致。
"docker_port": 8888, # model_config.llm_model_dict中模型配置的api_base_url需要与这里一致。
}

nebula conf

NEBULA_HOST = DEFAULT_BIND_HOST
NEBULA_PORT = 9669
NEBULA_STORAGED_PORT = 9779
NEBULA_USER = 'root'
NEBULA_PASSWORD = ''
NEBULA_GRAPH_SERVER = {
"host": DEFAULT_BIND_HOST,
"port": NEBULA_PORT,
"docker_port": NEBULA_PORT
}

chroma conf

CHROMA_PERSISTENT_PATH = '/home/user/chatbot/data/chroma_data'

sandbox api server

SANDBOX_CONTRAINER_NAME = "devopsgpt_sandbox"
SANDBOX_IMAGE_NAME = "devopsgpt:py39"
SANDBOX_HOST = os.environ.get("SANDBOX_HOST") or DEFAULT_BIND_HOST # "172.25.0.3"
SANDBOX_SERVER = {
"host": f"http://{SANDBOX_HOST}",
"port": 5050,
"docker_port": 5050,
"url": f"http://{SANDBOX_HOST}:5050",
"do_remote": SANDBOX_DO_REMOTE,
}

fastchat model_worker server

这些模型必须是在model_config.llm_model_dict中正确配置的。

在启动startup.py时，可用通过--model-worker --model-name xxxx指定模型，不指定则为LLM_MODEL

FSCHAT_MODEL_WORKERS = {
"default": {
"host": DEFAULT_BIND_HOST,
"port": 20002,
"device": LLM_DEVICE,
# todo: 多卡加载需要配置的参数
"gpus": None,
"numgpus": 1,
# 以下为非常用参数，可根据需要配置
# "max_gpu_memory": "20GiB",
# "load_8bit": False,
# "cpu_offloading": None,
# "gptq_ckpt": None,
# "gptq_wbits": 16,
# "gptq_groupsize": -1,
# "gptq_act_order": False,
# "awq_ckpt": None,
# "awq_wbits": 16,
# "awq_groupsize": -1,
# "model_names": [LLM_MODEL],
# "conv_template": None,
# "limit_worker_concurrency": 5,
# "stream_interval": 2,
# "no_register": False,
},
'codellama_34b': {'host': DEFAULT_BIND_HOST, 'port': 20002},
'Baichuan2-13B-Base': {'host': DEFAULT_BIND_HOST, 'port': 20003},
'Baichuan2-13B-Chat': {'host': DEFAULT_BIND_HOST, 'port': 20004},
'baichuan2-7b-base': {'host': DEFAULT_BIND_HOST, 'port': 20005},
'baichuan2-7b-chat': {'host': DEFAULT_BIND_HOST, 'port': 20006},
'internlm-7b-base': {'host': DEFAULT_BIND_HOST, 'port': 20007},
'internlm-chat-7b': {'host': DEFAULT_BIND_HOST, 'port': 20008},
'chatglm2-6b': {'host': DEFAULT_BIND_HOST, 'port': 20009},
'qwen-14b-base': {'host': DEFAULT_BIND_HOST, 'port': 20010},
'qwen-14b-chat': {'host': DEFAULT_BIND_HOST, 'port': 20011},
'qwen-1-8B-Chat': {'host': DEFAULT_BIND_HOST, 'port': 20012},
'Qwen-7B': {'host': DEFAULT_BIND_HOST, 'port': 20013},
'Qwen-7B-Chat': {'host': DEFAULT_BIND_HOST, 'port': 20014},
'qwen-7b-base-v1.1': {'host': DEFAULT_BIND_HOST, 'port': 20015},
'qwen-7b-chat-v1.1': {'host': DEFAULT_BIND_HOST, 'port': 20016},
'chatglm3-6b': {'host': DEFAULT_BIND_HOST, 'port': 20017},
'chatglm3-6b-32k': {'host': DEFAULT_BIND_HOST, 'port': 20018},
'chatglm3-6b-base': {'host': DEFAULT_BIND_HOST, 'port': 20019},
'Qwen-72B-Chat-Int4': {'host': DEFAULT_BIND_HOST, 'port': 20020},
'gpt-3.5-turbo': {'host': DEFAULT_BIND_HOST, 'port': 20021}
}

fastchat multi model worker server

FSCHAT_MULTI_MODEL_WORKERS = {
# todo
}

fastchat controller server

FSCHAT_CONTROLLER = {
"host": DEFAULT_BIND_HOST,
"port": 20001,
"dispatch_method": "shortest_queue",
}

以下不要更改

def fschat_controller_address() -> str:
host = FSCHAT_CONTROLLER["host"]
port = FSCHAT_CONTROLLER["port"]
return f"http://{host}:{port}"

def fschat_model_worker_address(model_name: str = LLM_MODEL) -> str:
if model := FSCHAT_MODEL_WORKERS.get(model_name):
host = model["host"]
port = model["port"]
return f"http://{host}:{port}"

def fschat_openai_api_address() -> str:
host = FSCHAT_OPENAI_API["host"]
port = FSCHAT_OPENAI_API["port"]
return f"http://{host}:{port}"

def api_address() -> str:
host = API_SERVER["host"]
port = API_SERVER["port"]
return f"http://{host}:{port}"

def webui_address() -> str:
host = WEBUI_SERVER["host"]
port = WEBUI_SERVER["port"]
return f"http://{host}:{port}"

lightislost commented

openai接口调用的实现是基于langchain和openai的py依赖，他们能接入，那我们也能接入

lightislost commented 
  from langchain.chat_models import ChatOpenAI
  model = ChatOpenAI(
      streaming=True,
      verbose=True,
      openai_api_key="dsdadas",
      model_name="gpt-4"
  )
  print(model.predict("hi!"))

langchain的调用示例