rdma_cm 和vbers配合使用，rdma_cm主要用于管理连接（建立和断开）、vbers用于管理数据的收发。
对于rdma编程，目前主流实现是利用rdma_cm来建立连接，然后利用verbs来传输数据。
rdma_cm和ibverbs分别会创建一个fd，这两个fd的分工不同。rdma_cm fd主要用于通知建连相关的事件，verbs fd则主要通知有新的cqe发生。当直接对rdma_cm fd进行poll/epoll监听时，此时只能监听到POLLIN事件，这意味着有rdma_cm事件发生。当直接对verbs fd进行poll/epoll监听时，同样只能监听到POLLIN事件，这意味着有新的cqe。
-接受连接请求、错误处理、打印调试信息等功能。rdmacm接口使用TCP/IP协议进行连接管理，它提供了一种简单的方式来管理RDMA传输和通信。
verbs接口则是对RDMA传输和通信进行操作的主要API，它提供了访问设备、内存和网络协议的抽象层。verbs接口是一种底层的编程接口，通过verbs接口可以控制RDMA网络适配器中的硬件资源、创建和管理RDMA操作队列等功能，同时verbs接口也提供了一些原子操作、随机访问、远程直接内存访问等工具，从而实现了高效的、无锁的、直接内存访问。
总之，rdmacm接口和verbs接口配合是比单纯使用verbs接口更方便的方式，其中rdmacm接口主要用于连接管理，并提供了简单易用的接口，verbs接口则是整个RDMA传输和通信的主要API，提供了底层、高效的编程接口。
- RDMACM（RDMA Connection Manager）： RDMACM是一个库，用于管理RDMA连接的建立、维护和关闭。它提供了一组函数，使应用程序能够发现和连接远程节点，并在需要时建立RDMA连接。RDMACM简化了RDMA连接的管理过程，使应用程序可以更方便地使用RDMA功能
不使用CM建连, 使用socket建连, 然后交换QP建连所需的lid, gid, index等信息, 通过ibv_modify_qp转QP的状态机(INIT -> RTR -> RTS)实现QP建连,比较繁琐.
CM通信库简化了应用编程, 以及基于CM事件回调的异步轮询编程模型, 相比ibv_modify_qp等接口更易使用
（看过几个rdma_cm接口的例子，确实如此）
rdma_cm man的解释：
rdma_cm(7): RDMA communication manager - Linux man page）
RDMA CM 可以（用于）控制 RDMA API 的 QP 和通信管理（连接设立/拆毁）部分，或者只有通信管理部分。它与 libibverbs 库定义的erbs API 结合使用。 libibverbs 库提供发送和接收数据所需的底层接口。RDMA CM 可以异步或同步操作。操作模式由用户调用的时候用 rdma_cm 事件通道参数来控制。如果提供了事件通道，则 rdma_cm 标识符 将在该通道上报告其（标识符的）事件数据（例如，连接结果）。（异步）
如果未提供通道，则所选 rdma_cm 标识符的所有 rdma_cm 操作将阻塞，直到它们完成。（同步）
Rdma Verbs
rdma_cm 支持通过 libibverbs 库和接口提供的所有verbs （意思是libibverbs库提供的功能和接口librdmacm库都能提供？）。但是，它还为一些更常用的verbs 功能提供了包装函数：
rdma_reg_msgs - register an array of buffers for sending and receiving
rdma_reg_read - registers a buffer for RDMA read operations
rdma_reg_write - registers a buffer for RDMA write operations
rdma_dereg_mr - deregisters a memory region
rdma_post_recv - post a buffer to receive a message
rdma_post_send - post a buffer to send a message
rdma_post_read - post an RDMA to read data into a buffer
rdma_post_write - post an RDMA to send data from a buffer
rdma_post_recvv - post a vector of buffers to receive a message
rdma_post_sendv - post a vector of buffers to send a message
rdma_post_readv - post a vector of buffers to receive an RDMA read
rdma_post_writev - post a vector of buffers to send an RDMA write
rdma_post_ud_send - post a buffer to send a message on a UD QP
rdma_get_send_comp - get completion status for a send or RDMA operation
rdma_get_recv_comp - get information about a completed receive
对vbers的包装？
例如：
rdma_post_send = ibv_post_send(qp,wr.opcode=IBV_WR_SEND,bad_wr) ?
rdma_post_read = ibv_post_send(qp,wr.opcode=IBV_WR_RDMA_READ ,bad_wr) ?
rdma_post_write = ibv_post_send(qp,wr.opcode=IBV_WR_RDMA_WRITE,bad_wr) ?
libibverbs和librdmacm的区别？
一、rdma_cm依赖ib_verbs？
在infiniband/verbs.h中，定义了ibv_post_send()和ibv_post_recv()操作，分别表示，将wr发布到SQ和RQ中，至于是什么操作(send or write/read)，和wr中的opcode有关。
对ibv_post_send()来说，对应的是struct ibv_send_wr，其中有opcode，表示操作码，有SEND/WRITE/READ等。
对于ibv_post_recv()来说，对应的是struct ibv_recv_wr，没有操作码，因为只有接收一个动作，所以不需要定义其它的操作码。但是发送来说，有三类。
在rdma/rdma_verbs.h中，有rdma_post_send()，rdma_post_recv()，rdma_post_read()，rdma_post_write()。
rdma_post_send()：把wr发布到QP的SQ中，需要mr
rdma_post_recv()：把wr发布到QP的RQ中，需要mr
rdma_post_read()：把wr发布到QP的SQ中，执行RDMA READ操作，需要远程地址和rkey，以及本地存储地址和长度，以及mr
rdma_post_write()：把wr发布到QP的SQ中，RDMA WRITE操作，需要远程的被写入地址和rkey，以及本地要发送数据的地址和长度，以及mr
所以rdma/rdma_verbs.h中的四种通信函数其实和infiniband/verbs.h中的两种方法是一致的。
ibv_post_send()对应rdma_post_send()、rdma_post_read()、rdma_post_write()，ibv_post_recv()对应rdma_post_recv()。
二、其他区别？
在 RDMA 中，有两个选项可以在两端之间建立连接：
- 通过调用 ibv_modify_qp() 在应用程序中显式更改 QP 状态
- 使用 librdmacm（在 iWARP 中这是唯一的方法）
来源：Connecting Queue Pairs - RDMAmojo RDMAmojo
这样来看，说librdmacm是libibverbs的上层封装似乎不正确。有待更多研究。
rdma_cm客户端和服务端操作
（rdma_cm(7): RDMA communication manager - Linux man page）
Rdma Verbs
The rdma_cm supports the full range of verbs available through the libibverbs library and interfaces. However, it also provides wrapper functions for some of the more commonly used verbs funcationality. The full set of abstracted verb calls are:rdma_reg_msgs - register an array of buffers for sending and receiving
rdma_reg_read - registers a buffer for RDMA read operations
rdma_reg_write - registers a buffer for RDMA write operations
rdma_dereg_mr - deregisters a memory region
rdma_post_recv - post a buffer to receive a message
rdma_post_send - post a buffer to send a message
rdma_post_read - post an RDMA to read data into a buffer
rdma_post_write - post an RDMA to send data from a buffer
rdma_post_recvv - post a vector of buffers to receive a message
rdma_post_sendv - post a vector of buffers to send a message
rdma_post_readv - post a vector of buffers to receive an RDMA read
rdma_post_writev - post a vector of buffers to send an RDMA write
rdma_post_ud_send - post a buffer to send a message on a UD QP
rdma_get_send_comp - get completion status for a send or RDMA operation
rdma_get_recv_comp - get information about a completed receive
Client Operation
This section provides a general overview of the basic operation for the active, or client, side of communication. This flow assume asynchronous operation with low level call details shown. For synchronous operation, calls to rdma_create_event_channel, rdma_get_cm_event, rdma_ack_cm_event, and rdma_destroy_event_channel would be eliminated. Abstracted calls, such as rdma_create_ep encapsulate serveral of these calls under a single API. Users may also refer to the example applications for code samples. A general connection flow would be:rdma_getaddrinfo
retrieve address information of the destination
rdma_create_event_channel
create channel to receive events
rdma_create_id
allocate an rdma_cm_id, this is conceptually similar to a socket
rdma_resolve_addr
obtain a local RDMA device to reach the remote address
rdma_get_cm_event
wait for RDMA_CM_EVENT_ADDR_RESOLVED event
rdma_ack_cm_event
ack event
rdma_create_qp
allocate a QP for the communication
rdma_resolve_route
determine the route to the remote address
rdma_get_cm_event
wait for RDMA_CM_EVENT_ROUTE_RESOLVED event
rdma_ack_cm_event
ack event
rdma_connect
connect to the remote server
rdma_get_cm_event
wait for RDMA_CM_EVENT_ESTABLISHED event
rdma_ack_cm_event
ack event
Perform data transfers over connection
rdma_disconnect
tear-down connection
rdma_get_cm_event
wait for RDMA_CM_EVENT_DISCONNECTED event
rdma_ack_cm_event
ack event
rdma_destroy_qp
destroy the QP
rdma_destroy_id
release the rdma_cm_id
rdma_destroy_event_channel
release the event channel
An almost identical process is used to setup unreliable datagram (UD) communication between nodes. No actual connection is formed between QPs however, so disconnection is not needed.
Although this example shows the client initiating the disconnect, either side of a connection may initiate the disconnect.
Server Operation
This section provides a general overview of the basic operation for the passive, or server, side of communication. A general connection flow would be:rdma_create_event_channel
create channel to receive events
rdma_create_id
allocate an rdma_cm_id, this is conceptually similar to a socket
rdma_bind_addr
set the local port number to listen on
rdma_listen
begin listening for connection requests
rdma_get_cm_event
wait for RDMA_CM_EVENT_CONNECT_REQUEST event with a new rdma_cm_id
rdma_create_qp
allocate a QP for the communication on the new rdma_cm_id
rdma_accept
accept the connection request
rdma_ack_cm_event
ack event
rdma_get_cm_event
wait for RDMA_CM_EVENT_ESTABLISHED event
rdma_ack_cm_event
ack event
Perform data transfers over connection
rdma_get_cm_event
wait for RDMA_CM_EVENT_DISCONNECTED event
rdma_ack_cm_event
ack event
rdma_disconnect
tear-down connection
rdma_destroy_qp
destroy the QP
rdma_destroy_id
release the connected rdma_cm_id
rdma_destroy_id
release the listening rdma_cm_id
rdma_destroy_event_channel
release the event channel
Return Codes
= 0
success
= -1
error - see errno for more details
Most librdmacm functions return 0 to indicate success, and a -1 return value to indicate failure. If a function operates asynchronously, a return value of 0 means that the operation was successfully started. The operation could still complete in error; users should check the status of the related event. If the return value is -1, then errno will contain additional information regarding the reason for the failure.
Prior versions of the library would return -errno and not set errno for some cases related to ENOMEM, ENODEV, ENODATA, EINVAL, and EADDRNOTAVAIL codes. Applications that want to check these codes and have compatability with prior library versions must manually set errno to the negative of the return code if it is < -1.
