Does ceph-ansible 6.0 support bcache?
xudehua opened this issue · 6 comments
Bug Report
What happened:
Deploy a ceph cluster with ceph-ansible 6.0,To accelerate Ceph OSD performance with using Bcache.
Does ceph-ansible 6.0 support bcache to deploy ceph cluster?
and How do I config the osds.yml file?
What you expected to happen:
How to reproduce it (minimal and precise):
lsblk
[root@ceph02 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 40G 0 disk
├─vda1 253:1 0 1G 0 part /boot
├─vda2 253:2 0 20G 0 part /
├─vda3 253:3 0 10G 0 part /home
├─vda4 253:4 0 1K 0 part
├─vda5 253:5 0 6G 0 part /recovery
└─vda6 253:6 0 3G 0 part [SWAP]
vdb 253:16 0 20G 0 disk
├─vdb1 253:17 0 5G 0 part
├─vdb2 253:18 0 5G 0 part
└─vdb3 253:19 0 10G 0 part
└─bcache0 252:0 0 20G 0 disk
vdc 253:32 0 20G 0 disk
└─bcache0 252:0 0 20G 0 disk
edit ceph-ansible/group_vars/osds.yml, add the following content
lvm_volumes:
- data: /dev/bcache0
wal: /dev/vdb1
db: /dev/vdb2
then run "ansible-playbook site.yml"
It error:
TASK [ceph-osd : use ceph-volume to create bluestore osds] **********************************************************************************************
Monday 10 January 2022 10:08:27 +0800 (0:00:00.085) 0:03:37.943 ********
failed: [ceph04] (item={'data': '/dev/bcache0', 'wal': '/dev/vdb1', 'db': '/dev/vdb2'}) => changed=true
ansible_loop_var: item
cmd:
- ceph-volume
- --cluster
- ceph
- lvm
- create
- --bluestore
- --data
- /dev/bcache0
- --block.db
- /dev/vdb2
- --block.wal
- /dev/vdb1
delta: '0:01:35.762529'
end: '2022-01-10 10:10:03.274638'
item:
data: /dev/bcache0
db: /dev/vdb2
wal: /dev/vdb1
msg: non-zero return code
Share your group_vars files, inventory and full ceph-ansibe log
Environment:
- OS (e.g. from /etc/os-release): centos 8.0
- Kernel (e.g.
uname -a): 4.19.113-10 - Docker version if applicable (e.g.
docker version): - Ansible version (e.g.
ansible-playbook --version): 2.9.21 - ceph-ansible version (e.g.
git head or tag or stable branch): 6.0 - Ceph version (e.g.
ceph -v): 16.2.7
Can someone answer that or provide some steps?
Thanks.
@xudehua I think ceph-ansible itself would support bcache.
Looks like you are facing a ceph-volume limitation here.
can you go on node ceph04 and share the output of :
# CEPH_VOLUME_DEBUG=1 ceph-volume --cluster ceph lvm create --bluestore --data /dev/bcache0 --block.db /dev/vdb2 --block.wal /dev/vdb1@guits
the log :
[root@ceph04 ~]# CEPH_VOLUME_DEBUG=1 ceph-volume --cluster ceph lvm create --bluestore --data /dev/bcache0 --block.db /dev/vdb2 --block.wal /dev/vdb1
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 811a5e64-bd6a-49e5-a58a-66a27f19ad1e
Running command: /usr/sbin/vgcreate --force --yes ceph-a6236d75-aecb-4695-8df9-50e769ff7cd0 /dev/bcache0
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: purged osd.0
......
File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 943, in create_lv
vg = create_vg(device, name_prefix='ceph')
File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 650, in create_vg
name] + devices
File "/usr/lib/python3.6/site-packages/ceph_volume/process.py", line 153, in run
raise RuntimeError(msg)
RuntimeError: command returned non-zero exit status: -11
The reason should be that the bcache0 is not done correctly,I will debug it.
"I think ceph-ansible itself would support bcache." is very useful to me.
thanks
@guits
...... The ellipsis part is some code for ceph_volume.
The resason is that bcache.ko version does not match the centos kernel version.
when running separately the command "/usr/sbin/vgcreate --force --yes ceph-a6236d75-aecb-4695-8df9-50e769ff7cd0 /dev/bcache0",The terminal displays a segment fault.
I rebuild the kernel 4.19.113 with bcache,and redeployed the ceph cluster with ceph-ansible, it's successful.
Is the osds.yml content I modified correct?
lvm_volumes:
- data: /dev/bcache0
wal: /dev/vdb1
db: /dev/vdb2
thanks again
@guits ...... The ellipsis part is some code for ceph_volume. The resason is that bcache.ko version does not match the centos kernel version. when running separately the command "/usr/sbin/vgcreate --force --yes ceph-a6236d75-aecb-4695-8df9-50e769ff7cd0 /dev/bcache0",The terminal displays a segment fault. I rebuild the kernel 4.19.113 with bcache,and redeployed the ceph cluster with ceph-ansible, it's successful.
Is the osds.yml content I modified correct?
lvm_volumes: - data: /dev/bcache0 wal: /dev/vdb1 db: /dev/vdb2thanks again
yes it looks good