1. 配置
a. xml
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB' nodeset='0'/>
</hugepages>
</memoryBacking>
<cpu mode='custom' match='exact'>
<model fallback='allow'>kvm64</model>
<feature policy='require' name='avx'/>
....
<feature policy='require' name='ssse3'/>
<numa>
<cell id='0' cpus='0-3' memory='16777216' unit='KiB' memAccess='shared'/>
</numa>
<interface type='vhostuser'>
<mac address='52:54:11:88:2:61'/>
<source type='unix' path='/tmp/vhostuser.sock' mode='client'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</interface>
b. qemu
-chardev socket,id=char0,path=test/vhostuser.sock -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce -device virtio-net-pci,mac=52:54:11:88:2:61,netdev=mynet1,object memory-backend-file,id=mem,size=$GUEST_MEM,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc
由于vhost_user进程与qemu并没有在同一个进程中, 所以数据的传递就需要内存共享的方式. 所以通常虚拟机内存的分配需要使用hugepage方式.
2.
在vhost模式下, 整个虚拟网网卡包括三部分
a. virtio_net: 虚拟机里面的网卡驱动
b. virtio_user: qemu里面模拟的virtio device后端
c. vhost_user: 数据面接收转发模块
vhost_user主要由dpdk提供的支持
vhost_user
1). create unix_socket and listen on it
rte_vhost_driver_register()-->vhost_user_create_server()
{
建立unix_socket, 设置fd的callback为vhost_user_server_new_connection:a. accept b. accept_fd 建立信息通道
}
2). 创建线程接收fd的消息, 设置创建/删除device(vhost_user) callback
vhost_driver_session_start()
virtio_user
3). 初始化backend
virtio_user_dev_init()
{
a. 连接到unix sock, vhost user会accept 然后建立消息通道
vhost_user_setup
b. 消息VHOST_USER_SET_OWNER: 空设置
c. 消息VHOST_USER_GET_FEATURES: 获取vhost_user feature
virtio_user->features = vhost_user features
}
virtio_net
pcidev: device sub_device vender sub_vender
0x1000 0x1 0xfaf4 0xfaf4
virtiodev: device = pcidev->sub_device vender = pcidev->sub_vender
4). virtio_net init
virtio_pci_probe---->virtio_dev_probe---->virtio_net->probe
virtio_dev_probe()
{
a. set status VIRTIO_CONFIG_S_DRIVER
b. 获取virtio_user(device) feture
dev->config->get_features: 其实就是device_feature = virtio_user->features
driver_feature = virtio_net->feature_table;
c. dev->features = driver_features & device_features;
如果 device_feature 有VIRTIO_RING_F_INDIRECT_DESC, VIRTIO_RING_F_EVENT_IDX. dev->features也设置上(这两个feature由device决定)
d. 通知virtio_user(device) 最终协商的feature
dev->config->finalize_features: 其实就是virtio_user->features = dev->features;
e. drv->probe: 初始化virtio_net, init queues
f. set status VIRTIO_CONFIG_S_DRIVER_OK
}
virtio_user
5). virtio_user set vhost_user
virtio_user_start_device
{
a. virtio_user_create_queue --->VHOST_USER_SET_VRING_CALL
b. set virtio_user->features to vhost_user -----> VHOST_USER_SET_FEATURES
c. VHOST_USER_SET_FEATURES, 告知vhost_user共享内存文件打开的fd. 所以vhost_user也能映射出共享文件的内存
d. virtio_user_kick_queue--->VHOST_USER_SET_VRING_NUM, VHOST_USER_SET_VRING_BASE, VHOST_USER_SET_VRING_ADDR,VHOST_USER_SET_VRING_KICK.
告诉vhost_user每个queue的vring的虚拟地址, 让vhost_user在共享内存中找到并构建出vring
e. new_device: 当最后一个queue kick后就回调new_device
f. VHOST_USER_SET_VRING_ENABLE: enable queues
}
