在虚拟机的XML配置文件中添加新disk配置，指定刚才配置好的secet和ceph集群的monitor IP和pool_name/rbd_block_name即可。
详细步骤：
- 配置第一个ceph集群的认证和挂载rbd盘
cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
<usage type='ceph'>
<name>client.libvirt secret</name>
</usage>
</secret>
EOF
virsh secret-define --file secret.xml #定义出来,会返回Secret 2c8b6fa3-1dc7-4107-a0e0-8c7688a7bf5f created 返回的数值部分就是libvirt_uuid
#列出
virsh secret-list --可查出uuid
#用ceph用户来签发libvirt用户的secret
配置ceph认证
ceph auth list
ceph auth get-or-create client.libvirt mon 'allow *' osd 'allow *' mgr 'allow *'
ceph auth caps client.libvirt mon 'allow *' osd 'allow *' mgr 'allow *'
virsh secret-set-value --secret 0b2a0d5c-4259-4d6f-b8d8-68e4daf0ee64 AQB/ygdgqMUyMBAALj0U8aK5XKfr3Ydms5uPZQ==
error: Passing secret value as command-line argument is insecure! ---已经设置成功了，只是提示这样设置不安全而已！
Secret value setsecret_uuid_libvirt
ceph auth get-key client.libvirt | tee client.libvirt.key #从ceph获取libvirt用户的认证key值
sudo virsh secret-set-value --secret $(cat secret_uuid_libvirt) --base64 $(cat client.libvirt.key)
# virsh secret-get-value $(cat secret_uuid_libvirt)
AQCB3ghgnQR0FRAAe8NYru1g7aW+cCyH6uTrPw==
# virsh start wcy_mig_vm
error: Failed to start domain wcy_mig_vm
error: internal error: qemu unexpectedly closed the monitor: 2021-01-21T02:30:35.205424Z qemu-kvm: -blockdev {"driver":"rbd","pool":"test_pool","
image":"wcy_rbd_8g","server":[{"host":"192.168.122.11","port":"6789"},{"host":"192.168.122.12","port":"6789"},{"host":"192.168.122.13","port":"6789"}],"
user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-2-storage-auth-secret0","node-name":"libvirt-2-storage","cache":{"direct":
false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error connecting: Operation not permitted
2021-01-21T03:18:35.351274Z qemu-kvm: -blockdev {"driver":"rbd","pool":"test_pool","image":"wcy_rbd_8g","server":[{"host":"192.168.122.11","port":"
6789"},{"host":"192.168.122.12","port":"6789"},{"host":"192.168.122.13","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key
secret":"libvirt-2-storage-auth-secret0","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"
unmap"}: error connecting: Operation not permitted
2021-01-21 03:18:35.362+0000: shutting down, reason=failed
2021-01-21 03:28:22.166+0000: 53573: warning : qemuDomainObjTaint:7157 : Domain id=61 name='wcy_mig_vm' uuid=83a50eaf-d83b-49aa-86b5-
bf9a570d6872 is tainted: high-privileges
2021-01-21 03:28:43.216+0000: 53735: error : qemuMonitorIORead:489 : Unable to read from monitor: Connection reset by peer
2021-01-21 03:28:43.217+0000: 53735: error : qemuProcessReportLogError:2103 : internal error: qemu unexpectedly closed the monitor: 2021-01-
21T03:28:43.202624Z qemu-kvm: -blockdev {"driver":"rbd","pool":"test_pool","image":"wcy_rbd_8g","server":[{"host":"192.168.122.11","port":"6789"},
{"host":"192.168.122.12","port":"6789"},{"host":"192.168.122.13","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"
libvirt-2-storage-auth-secret0","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error
connecting: Operation not permitted
xml配置auth有问题！
解决：
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writethrough' discard='unmap'/>
<auth username='libvirt'>
<secret type='ceph' uuid='0b2a0d5c-4259-4d6f-b8d8-68e4daf0ee64'/>
</auth>
<source protocol='rbd' name='test_pool/wcy_rbd_8g'>
<host name='192.168.122.11' port='6789'/>
<host name='192.168.122.12' port='6789'/>
<host name='192.168.122.13' port='6789'/>
</source>
<target dev='vdb' bus='virtio'/>
<serial>348d2c47-b468-4fe3-8a78-f0a96f21f68d</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
client.libvirt
key: AQCB3ghgnQR0FRAAe8NYru1g7aW+cCyH6uTrPw==
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
在/etc/ceph目录下添加libvirt用户的keyring
# cat ceph.client.libvirt.keyring
[client.libvirt]
key = AQCB3ghgnQR0FRAAe8NYru1g7aW+cCyH6uTrPw==
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
ceph --name client.libvirt -s 有正常返回！
- 配置第二个ceph集群的认证和挂载该集群的rbd盘
cat > secret2.xml <<EOF
<secret ephemeral='no' private='no'>
<usage type='ceph'>
<name>client.libvirt2 secret</name>
</usage>
</secret>
EOF#在此固定住uuuid --在目的主机针对同一个集群时，需要指定和源一样的uuid!!!!!否则不能够迁移！
cat << EOF > /etc/ceph/secret.xml
<secret ephemeral='no' private='no'>
<uuid>6a085c23-2177-242d-7661-c785df7f6230</uuid>
<usage type='ceph'>
<name>client.admin secret</name>
</usage>
</secret>
EOF
virsh secret-define --file secret2.xml #命令成功后会返回该secret的uuid
Secret 3f277a88-94cc-41c9-8a22-f1564ecfc1ed created
#列出
virsh secret-list --可查出uuid
UUID Usage
---------------------------------------------------------------------
0b2a0d5c-4259-4d6f-b8d8-68e4daf0ee64 ceph client.libvirt secret
3f277a88-94cc-41c9-8a22-f1564ecfc1ed ceph client.libvirt2 secret
#用ceph用户来签发libvirt用户的secret
配置ceph认证
ceph auth list
ceph auth get-or-create client.libvirt2 mon 'allow *' osd 'allow *' mgr 'allow *'
ceph auth caps client.libvirt mon 'allow *' osd 'allow *' mgr 'allow *'
[client.libvirt2]
key = AQAudgpg2gyAORAA/dizbAvV9xagY4WqsEog4Q==
virsh secret-set-value --secret 3f277a88-94cc-41c9-8a22-f1564ecfc1ed AQAudgpg2gyAORAA/dizbAvV9xagY4WqsEog4Q==
virsh secret-get-value 3f277a88-94cc-41c9-8a22-f1564ecfc1ed
在第二个集群创建pool和rbd块：
ceph osd pool create test_pool2 32 32
rados lspools
rbd create wcy_rbd_cluster2_6g -p test_pool2 --size 6G
rbd info wcy_rbd_cluster2_6g -p test_pool
修改VM XML添加：
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writethrough' discard='unmap'/>
<auth username='libvirt2'>
<secret type='ceph' uuid='3f277a88-94cc-41c9-8a22-f1564ecfc1ed'/>
</auth>
<source protocol='rbd' name='test_pool2/wcy_rbd_cluster2_6g'>
<host name='192.168.122.15' port='6789'/>
<host name='192.168.122.16' port='6789'/>
<host name='192.168.122.17' port='6789'/>
</source>
<target dev='vdc' bus='virtio'/>
<serial>348d2c47-b468-4fe3-8a78-f0a96f21f688</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</disk>
# ceph auth get-or-create client.libvirt mon 'allow *' osd 'allow *' mgr 'allow *'
[client.libvirt]
key = AQAPMRFgA0ppNxAAq1VXZlpnY+61CDI0BKW9hQ==
2f79bd85-afe3-4be9-a25c-9d15d6e7676a
