请教一个问题。如何才能喂饱多个GPU
leonunix opened this issue · 3 comments
leonunix commented
提交前必须检查以下项目
- 请确保使用的是仓库最新代码（git pull），一些问题已被解决和修复。
- 我已阅读项目文档和FAQ章节并且已在Issue中对问题进行了搜索，没有找到相似问题和解决方案。
- 第三方插件问题：例如llama.cpp、LangChain、text-generation-webui等，同时建议到对应的项目中查找解决方案。
问题类型
模型训练与精调
基础模型
Chinese-LLaMA-2 (7B/13B)
操作系统
Linux
详细描述问题
你好，开发组，我想请教一个问题。如何才能在单机上把多GPU撑满。情况是这样的，我现在有一台机器上有8块GPU，用你们的脚本跑pt，但是我发现每个gpu的利用率有点低迷，这个需要什么特别的配置吗？
per_device_train_batch_size我设置的是7，超过这个数会报显存不足。
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA H100 80GB HBM3 Off | 00000000:19:00.0 Off | 0 |
| N/A 59C P0 300W / 700W | 80865MiB / 81559MiB | 28% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA H100 80GB HBM3 Off | 00000000:3B:00.0 Off | 0 |
| N/A 48C P0 293W / 700W | 80961MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA H100 80GB HBM3 Off | 00000000:4C:00.0 Off | 0 |
| N/A 46C P0 366W / 700W | 80991MiB / 81559MiB | 29% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA H100 80GB HBM3 Off | 00000000:5D:00.0 Off | 0 |
| N/A 56C P0 318W / 700W | 80577MiB / 81559MiB | 27% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 4 NVIDIA H100 80GB HBM3 Off | 00000000:9B:00.0 Off | 0 |
| N/A 58C P0 366W / 700W | 80561MiB / 81559MiB | 33% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 5 NVIDIA H100 80GB HBM3 Off | 00000000:BB:00.0 Off | 0 |
| N/A 46C P0 326W / 700W | 80989MiB / 81559MiB | 1% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 6 NVIDIA H100 80GB HBM3 Off | 00000000:CB:00.0 Off | 0 |
| N/A 55C P0 336W / 700W | 80961MiB / 81559MiB | 27% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 7 NVIDIA H100 80GB HBM3 Off | 00000000:DB:00.0 Off | 0 |
| N/A 43C P0 331W / 700W | 80481MiB / 81559MiB | 32% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 25052 C /usr/bin/python3 80856MiB |
| 1 N/A N/A 25053 C /usr/bin/python3 80952MiB |
| 2 N/A N/A 25054 C /usr/bin/python3 80982MiB |
| 3 N/A N/A 25055 C /usr/bin/python3 80568MiB |
| 4 N/A N/A 25056 C /usr/bin/python3 80552MiB |
| 5 N/A N/A 25057 C /usr/bin/python3 80980MiB |
| 6 N/A N/A 25058 C /usr/bin/python3 80952MiB |
| 7 N/A N/A 25059 C /usr/bin/python3 80472MiB |
+-----------------------------------------------------------------------------------------+
依赖情况（代码类问题务必提供）
# 请在此处粘贴依赖情况（请粘贴在本代码块里）
运行日志或截图
# 请在此处粘贴运行日志（请粘贴在本代码块里）
iMountTai commented
这有多个原因，比如CPU和GPU之间的数据通信瓶颈、模型本身规模小不足以充分利用显卡的能力等，如果您有代码方面的改进和完善，欢迎提出讨论。
github-actions commented
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your consideration.
github-actions commented
Closing the issue, since no updates observed. Feel free to re-open if you need any further assistance.