searchusermenu
  • 发布文章
  • 消息中心
点赞
收藏
评论
分享
原创

rocketmq异地双活最佳实践

2022-12-06 06:25:56
623
0

1.Rocketmq异地双活是什么

Rocketmq是一款高性能,高可用,高可靠消息中间件,具有可靠的消息存储,消息发送的高性能和低延迟,强大的消息堆积能力与消息处理能力,严格的顺序消息模式等核心优势。
通过rocketmq消息中间件,可以快速实现了服务之间的解耦,流程异步化,流量削峰等核心功能,随着业务越来越复杂,集群规模越来越大 ,服务数量越来越多,依赖关系越来越复杂,消息中间件已经成为不可或缺的重要基础组件之一。
 
从rocketmq的架构图中,我们可以看到nameserver,broker,producer,consumer都是支持集群部署的,因此实现了高可用,但是考虑到不同机房之间网络开销,稳定性以及部署成本,这样的架构只支持单机房部署,不同rocketmq集群,数据是彼此独立,也就是说,rocketmq原生架构并不支持异地多活
Rokcetmq的异地双话,简单来说,就是指两个集群分别部署在两个机房,同时对外提供服务,当其中一个机房出现故障,另外一个的机房能接管全部的流量,同时对于故障机房尚未处理的数据,能够接手处理,机房的故障不影响服务的可用性。

2.Rocketmq异地双活需要解决什么问题

1.解决业务跨机房双写的问题
2.解决基于消息的应用零侵入接入
 

3.异地双活消息中间件的解决方案

3.1 异地双活消息中间件架构原理

模块功能说明:
msha-agent
异地多活客户端代理,提供路由传递和消息过滤服务,通过javaagent的方式植入用户应用程序,代码侵入。生产消息的时候为消息打上路由信息,消费消息的时候只消费有当前集群对应路由消息的数据。
msha-mq-sync
提供消息队列异地双向同步服务,以app的方式部署,可集群部署,支持高可用。使得两个broker集群都拥有全量的数据,互为灾备。
msha-manager
用于从整体上管控异地多活服务,主要是切流任务规划和实施,以app的方式部署,单点。基于nacos实现管控指令的下发。
 
说明:
nameserver,broker是rocketmq原生组件
msha-agent,msha-mq-sync,msha-manager是自研组件
 

3.2 接入RocketMQ双活能力应用需要做什么

(一)生产端:修改启动脚本,引入双活包

java -javaagent:lib/msha-agent-1.0.0.jar -Dmsha.commander.namespace=DemoNamespace -Dmsha.commander.addr=127.0.0.1:7810 -Dmsha.unit.name=unit-1 -jar rocketmq-lddl-client-demo-1.0.jar

(二)消费端:修改启动脚本,引入双活包

java -javaagent:lib/msha-agent-1.0.0.jar -Dmsha.commander.namespace=DemoNamespace -Dmsha.commander.addr=127.0.0.1:7810 -Dmsha.unit.name=unit-1 -jar rocketmq-lddl-client-demo-1.0.jar

(三)服务端:部署异地双活RocketMQ版本

部署msha-mq-sync集群,保证高可用,用于同步rocketmq broker数据。

(四)故障切换:集成RocketMQ的切流API

调用切流API
http://ip:port/switch?values=beijing
效果:将routerId为beijing的路由切换到另外一个机房,如果原本是unti-1,则切换到unit-2。
routerId由用户按业务需要配置。
 

4.以短线网关应用场景示例

4.1 业务场景介绍

在互联网服务,政企服务中有大量的发送短信场景,比如手机验证码,通知短信,营销短信等等,短信服务是广大企业客户快速触达手机用户所优先使用的通信能力。越大的平台,对短信服务的要求就越高,高吞吐量,低时延,可靠稳定,实时性高等特点是不可或缺的。
从短信网关需求来看,我们可以发现基于rocketmq实现短信网关是一个很好的选择
 

4.2 基于rocketmq实现短信网关

业务架构图
 
模块简介:
MTR —— 接受用户请求,做必要的业务处理,号码检查,内容检查,通道分配等,封装成消息提交到rocketmq集群。
Channel —— 从rocketmq集群上拉取未消费的消息,按协议提交到运营商网关,完成短信的发送,因为channel是集群部署,可横向扩展,提高短信发送的总速度。
 
因为rocketmq的引入,短信网关很方便就实现了高吞吐量,低时延,可靠稳定,实时性高等要求
 

4.3.短信网关实现异地双活

从上面短信网关的架构来看,我们可以看出这还是一个典型的单机房架构,即使在机房内实现了高可用,但是对于机房级别的故障,其容灾能力是不够的,还可以进一步提升。
基于自研的异地双活能力,我们可以很轻松实现短信网关的异地双活。
 
 
应用模块MTR,channel通过javaagent的方式启动,嵌入msha-agent代码,实现消息标记和消息过滤;
不同rocketmq broker集群之间通过msha-mq-sync同步数据,使得两个broker集群数据互为灾备,都拥有全量的数据。
 

4.4 效果演示

4.4.1 双机房同时对外提供服务。

机房unit-1的消息的生产和消费日志
 
2022-11-24 23:24:09.757  INFO 18664 --- [nio-8090-exec-1] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416393455239168,投递结果:SEND_OK
2022-11-24 23:24:10.509  INFO 18664 --- [nio-8090-exec-2] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416397083312128,投递结果:SEND_OK
2022-11-24 23:24:10.561  INFO 18664 --- [nio-8090-exec-3] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416397301415936,投递结果:SEND_OK
2022-11-24 23:24:10.628  INFO 18664 --- [nio-8090-exec-4] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416397498548224,投递结果:SEND_OK
2022-11-24 23:24:10.683  INFO 18664 --- [nio-8090-exec-5] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416397779566592,投递结果:SEND_OK
2022-11-24 23:24:10.733  INFO 18664 --- [nio-8090-exec-6] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398018641920,投递结果:SEND_OK
2022-11-24 23:24:10.784  INFO 18664 --- [nio-8090-exec-7] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398236745728,投递结果:SEND_OK
2022-11-24 23:24:10.834  INFO 18664 --- [nio-8090-exec-8] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398446460928,投递结果:SEND_OK
2022-11-24 23:24:10.893  INFO 18664 --- [nio-8090-exec-9] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398693924864,投递结果:SEND_OK
2022-11-24 23:24:10.939  INFO 18664 --- [io-8090-exec-10] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398891057152,投递结果:SEND_OK
2022-11-24 23:24:12.036  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416393455239168,消息来自机房:unit-1
2022-11-24 23:24:14.587  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416397083312128,消息来自机房:unit-1
2022-11-24 23:24:14.603  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398446460928,消息来自机房:unit-1
2022-11-24 23:24:14.618  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416397779566592,消息来自机房:unit-1
2022-11-24 23:24:14.634  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416397498548224,消息来自机房:unit-1
2022-11-24 23:24:14.650  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398693924864,消息来自机房:unit-1
2022-11-24 23:24:14.667  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416397301415936,消息来自机房:unit-1
2022-11-24 23:24:14.683  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398236745728,消息来自机房:unit-1
2022-11-24 23:24:14.700  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398018641920,消息来自机房:unit-1
2022-11-24 23:24:14.716  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398891057152,消息来自机房:unit-1
 
机房unit-2的消息的生产和消费日志
2022-11-24 23:24:10.124  INFO 21636 --- [nio-8092-exec-1] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416395149737984,投递结果:SEND_OK
2022-11-24 23:24:10.535  INFO 21636 --- [nio-8092-exec-3] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416397192364032,投递结果:SEND_OK
2022-11-24 23:24:10.584  INFO 21636 --- [nio-8092-exec-2] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416397402079232,投递结果:SEND_OK
2022-11-24 23:24:10.652  INFO 21636 --- [nio-8092-exec-4] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416397683097600,投递结果:SEND_OK
2022-11-24 23:24:10.707  INFO 21636 --- [nio-8092-exec-5] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416397913784320,投递结果:SEND_OK
2022-11-24 23:24:10.758  INFO 21636 --- [nio-8092-exec-6] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398127693824,投递结果:SEND_OK
2022-11-24 23:24:10.808  INFO 21636 --- [nio-8092-exec-7] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398337409024,投递结果:SEND_OK
2022-11-24 23:24:10.861  INFO 21636 --- [nio-8092-exec-8] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398555512832,投递结果:SEND_OK
2022-11-24 23:24:10.915  INFO 21636 --- [nio-8092-exec-9] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398786199552,投递结果:SEND_OK
2022-11-24 23:24:10.963  INFO 21636 --- [io-8092-exec-10] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398987526144,投递结果:SEND_OK
2022-11-24 23:24:24.469  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416395149737984,消息来自机房:unit-2
2022-11-24 23:24:24.486  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416397913784320,消息来自机房:unit-2
2022-11-24 23:24:27.000  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416397192364032,消息来自机房:unit-2
2022-11-24 23:24:27.015  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398337409024,消息来自机房:unit-2
2022-11-24 23:24:27.030  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416397402079232,消息来自机房:unit-2
2022-11-24 23:24:27.046  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416397683097600,消息来自机房:unit-2
2022-11-24 23:24:27.062  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398127693824,消息来自机房:unit-2
2022-11-24 23:24:27.076  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398555512832,消息来自机房:unit-2
2022-11-24 23:24:27.091  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398786199552,消息来自机房:unit-2
2022-11-24 23:24:27.106  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398987526144,消息来自机房:unit-2

 

从两个机房的生产和消费日志可见,短信id一一对应,每个机房只消费本机房生产的数据,数据机房内闭环。
 

4.4.2 发送过程中进行故障切换

当发现有一个机房出现故障,建设为机房unit-2故障,人工执行切流操作。
从日志中可以看到,路由切换以任务的形式(task=263d41ba-96d6-459e-a5c5-20458af54f07)分步骤执行(共7个步骤执行,phase=1到7),从2022-11-24 23:28:15.460开始,2022-11-24 23:28:23.577结束,耗时8.117秒。
 
机房unit-2切流日志 
2022-11-24 23:28:15.460  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=1
2022-11-24 23:28:16.869  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=2
2022-11-24 23:28:17.883  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=3
2022-11-24 23:28:19.003  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=4
2022-11-24 23:28:20.017  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=5
2022-11-24 23:28:21.133  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=6
2022-11-24 23:28:22.252  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=7
2022-11-24 23:28:23.264  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Complete process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07
2022-11-24 23:28:23.577  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Changed MSHA metadata: key=metadata.routers , value=[Metadata.Router: unitName=unit-1 , values=[beijing, guangzhou], Metadata.Router: unitName=unit-2 , values=[]]
 
切换后机房unit-2的生产消费日志
2022-11-24 23:28:23.513  INFO 21636 --- [nio-8092-exec-8] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458263195648,投递结果:SEND_OK
2022-11-24 23:28:23.557  INFO 21636 --- [nio-8092-exec-9] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458447745024,投递结果:SEND_OK
2022-11-24 23:28:23.599  INFO 21636 --- [io-8092-exec-10] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458623905792,投递结果:SEND_OK
2022-11-24 23:28:23.641  INFO 21636 --- [nio-8092-exec-1] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458795872256,投递结果:SEND_OK
2022-11-24 23:28:23.680  INFO 21636 --- [nio-8092-exec-3] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458959450112,投递结果:SEND_OK
2022-11-24 23:28:23.713  INFO 21636 --- [nio-8092-exec-2] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417459102056448,投递结果:SEND_OK
 
从日志中可以看到,切流之后机房unit-2之投递消息到mq,并没有消费。
 
切流后机房1的生产消费日志
2022-11-24 23:28:37.461  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458263195648,消息来自机房:unit-2
2022-11-24 23:28:37.475  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417472561577984,消息来自机房:unit-1
2022-11-24 23:28:37.475  INFO 18664 --- [nio-8090-exec-6] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417516819873792,投递结果:SEND_OK
2022-11-24 23:28:37.490  INFO 18664 --- [nio-8090-exec-7] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417516886982656,投递结果:SEND_OK
2022-11-24 23:28:37.504  INFO 18664 --- [nio-8090-exec-8] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417516941508608,投递结果:SEND_OK
2022-11-24 23:28:37.517  INFO 18664 --- [nio-8090-exec-9] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517000228864,投递结果:SEND_OK
2022-11-24 23:28:37.525  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458623905792,消息来自机房:unit-2
2022-11-24 23:28:37.536  INFO 18664 --- [io-8090-exec-10] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517079920640,投递结果:SEND_OK
2022-11-24 23:28:37.540  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417473811480576,消息来自机房:unit-1
2022-11-24 23:28:37.549  INFO 18664 --- [nio-8090-exec-1] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517134446592,投递结果:SEND_OK
2022-11-24 23:28:37.554  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417456757440512,消息来自机房:unit-1
2022-11-24 23:28:37.563  INFO 18664 --- [nio-8090-exec-2] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517188972544,投递结果:SEND_OK
2022-11-24 23:28:37.578  INFO 18664 --- [nio-8090-exec-3] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517256081408,投递结果:SEND_OK
2022-11-24 23:28:37.592  INFO 18664 --- [nio-8090-exec-4] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517310607360,投递结果:SEND_OK
2022-11-24 23:28:37.592  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458795872256,消息来自机房:unit-2
2022-11-24 23:28:37.606  INFO 18664 --- [nio-8090-exec-5] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517369327616,投递结果:SEND_OK
2022-11-24 23:28:37.619  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417475547922432,消息来自机房:unit-1
2022-11-24 23:28:37.625  INFO 18664 --- [nio-8090-exec-6] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517453213696,投递结果:SEND_OK
2022-11-24 23:28:37.633  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458334498816,消息来自机房:unit-1
2022-11-24 23:28:37.639  INFO 18664 --- [nio-8090-exec-7] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517511933952,投递结果:SEND_OK
2022-11-24 23:28:37.647  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458716180480,消息来自机房:unit-1
2022-11-24 23:28:37.656  INFO 18664 --- [nio-8090-exec-8] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517579042816,投递结果:SEND_OK
2022-11-24 23:28:37.663  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459513098240,消息来自机房:unit-1
2022-11-24 23:28:37.668  INFO 18664 --- [nio-8090-exec-9] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517633568768,投递结果:SEND_OK
2022-11-24 23:28:37.683  INFO 18664 --- [io-8090-exec-10] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517696483328,投递结果:SEND_OK
2022-11-24 23:28:37.690  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459034947584,消息来自机房:unit-1
2022-11-24 23:28:37.697  INFO 18664 --- [nio-8090-exec-1] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517755203584,投递结果:SEND_OK
2022-11-24 23:28:37.711  INFO 18664 --- [nio-8090-exec-2] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517813923840,投递结果:SEND_OK
2022-11-24 23:28:37.719  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417475866689536,消息来自机房:unit-1
2022-11-24 23:28:37.728  INFO 18664 --- [nio-8090-exec-3] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517885227008,投递结果:SEND_OK
2022-11-24 23:28:37.733  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417460884635648,消息来自机房:unit-1
2022-11-24 23:28:37.742  INFO 18664 --- [nio-8090-exec-4] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517943947264,投递结果:SEND_OK
2022-11-24 23:28:37.746  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458447745024,消息来自机房:unit-2
2022-11-24 23:28:37.753  INFO 18664 --- [nio-8090-exec-5] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517990084608,投递结果:SEND_OK
2022-11-24 23:28:37.760  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459164971008,消息来自机房:unit-1
2022-11-24 23:28:37.775  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459102056448,消息来自机房:unit-2
2022-11-24 23:28:37.776  INFO 18664 --- [nio-8090-exec-6] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518086553600,投递结果:SEND_OK
2022-11-24 23:28:37.788  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417476227399680,消息来自机房:unit-1
2022-11-24 23:28:37.788  INFO 18664 --- [nio-8090-exec-7] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518136885248,投递结果:SEND_OK
2022-11-24 23:28:37.802  INFO 18664 --- [nio-8090-exec-8] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518191411200,投递结果:SEND_OK
2022-11-24 23:28:37.802  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417461186625536,消息来自机房:unit-1
2022-11-24 23:28:37.815  INFO 18664 --- [nio-8090-exec-9] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518245937152,投递结果:SEND_OK
2022-11-24 23:28:37.817  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459454377984,消息来自机房:unit-1
2022-11-24 23:28:37.829  INFO 18664 --- [io-8090-exec-10] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518308851712,投递结果:SEND_OK
2022-11-24 23:28:37.832  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417460146438144,消息来自机房:unit-1
2022-11-24 23:28:37.845  INFO 18664 --- [nio-8090-exec-1] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518371766272,投递结果:SEND_OK
2022-11-24 23:28:37.846  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458959450112,消息来自机房:unit-2
从日志中可见,机房2产生的数据,sms id从到483417458263195648483417459102056448,共6条,都由机房1接收到并消费了,期间机房自身生产的消息继续正常消费,异地切流成功。

4.4.3 故障机房unit-2恢复,回切

切流日志,从2022-11-25 00:04:38.541开始,2022-11-25 00:04:51.168完成,耗时12.658秒。
2022-11-25 00:04:38.541  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=1
2022-11-25 00:04:39.349  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=2
2022-11-25 00:04:40.573  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=3
2022-11-25 00:04:41.692  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=4
2022-11-25 00:04:42.708  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=5
2022-11-25 00:04:43.844  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=6
2022-11-25 00:04:44.959  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=7
2022-11-25 00:04:51.161  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Complete process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398
2022-11-25 00:04:51.168  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Changed MSHA metadata: key=metadata.routers , value=[Metadata.Router: unitName=unit-1 , values=[beijing], Metadata.Router: unitName=unit-2 , values=[guangzhou]]

 

机房unit-2正常投入使用,回切成功。
2022-11-25 00:04:51.392  INFO 21636 --- [nio-8092-exec-4] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426634892836864,投递结果:SEND_OK
2022-11-25 00:04:51.431  INFO 21636 --- [nio-8092-exec-5] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635056414720,投递结果:SEND_OK
2022-11-25 00:04:51.473  INFO 21636 --- [nio-8092-exec-6] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635228381184,投递结果:SEND_OK
2022-11-25 00:04:51.514  INFO 21636 --- [nio-8092-exec-7] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635404541952,投递结果:SEND_OK
2022-11-25 00:04:51.560  INFO 21636 --- [nio-8092-exec-8] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635593285632,投递结果:SEND_OK
2022-11-25 00:04:51.600  INFO 21636 --- [nio-8092-exec-9] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635756863488,投递结果:SEND_OK
2022-11-25 00:04:51.637  INFO 21636 --- [io-8092-exec-10] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635920441344,投递结果:SEND_OK
2022-11-25 00:04:51.681  INFO 21636 --- [nio-8092-exec-1] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426636100796416,投递结果:SEND_OK
2022-11-25 00:04:51.725  INFO 21636 --- [nio-8092-exec-3] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426636289540096,投递结果:SEND_OK
2022-11-25 00:04:51.765  INFO 21636 --- [nio-8092-exec-2] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426636457312256,投递结果:SEND_OK
2022-11-25 00:04:58.186  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426634892836864,消息来自机房:unit-2
2022-11-25 00:04:58.275  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635056414720,消息来自机房:unit-2
2022-11-25 00:04:58.200  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635228381184,消息来自机房:unit-2
2022-11-25 00:04:58.068  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635404541952,消息来自机房:unit-2
2022-11-25 00:04:58.082  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426636289540096,消息来自机房:unit-2
2022-11-25 00:04:58.215  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635756863488,消息来自机房:unit-2
2022-11-25 00:04:58.260  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635593285632,消息来自机房:unit-2
2022-11-25 00:04:58.290  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635920441344,消息来自机房:unit-2
2022-11-25 00:04:58.304  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426636457312256,消息来自机房:unit-2
2022-11-25 00:04:58.318  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426636100796416,消息来自机房:unit-2
0条评论
0 / 1000
c****n
4文章数
2粉丝数
c****n
4 文章 | 2 粉丝
c****n
4文章数
2粉丝数
c****n
4 文章 | 2 粉丝
原创

rocketmq异地双活最佳实践

2022-12-06 06:25:56
623
0

1.Rocketmq异地双活是什么

Rocketmq是一款高性能,高可用,高可靠消息中间件,具有可靠的消息存储,消息发送的高性能和低延迟,强大的消息堆积能力与消息处理能力,严格的顺序消息模式等核心优势。
通过rocketmq消息中间件,可以快速实现了服务之间的解耦,流程异步化,流量削峰等核心功能,随着业务越来越复杂,集群规模越来越大 ,服务数量越来越多,依赖关系越来越复杂,消息中间件已经成为不可或缺的重要基础组件之一。
 
从rocketmq的架构图中,我们可以看到nameserver,broker,producer,consumer都是支持集群部署的,因此实现了高可用,但是考虑到不同机房之间网络开销,稳定性以及部署成本,这样的架构只支持单机房部署,不同rocketmq集群,数据是彼此独立,也就是说,rocketmq原生架构并不支持异地多活
Rokcetmq的异地双话,简单来说,就是指两个集群分别部署在两个机房,同时对外提供服务,当其中一个机房出现故障,另外一个的机房能接管全部的流量,同时对于故障机房尚未处理的数据,能够接手处理,机房的故障不影响服务的可用性。

2.Rocketmq异地双活需要解决什么问题

1.解决业务跨机房双写的问题
2.解决基于消息的应用零侵入接入
 

3.异地双活消息中间件的解决方案

3.1 异地双活消息中间件架构原理

模块功能说明:
msha-agent
异地多活客户端代理,提供路由传递和消息过滤服务,通过javaagent的方式植入用户应用程序,代码侵入。生产消息的时候为消息打上路由信息,消费消息的时候只消费有当前集群对应路由消息的数据。
msha-mq-sync
提供消息队列异地双向同步服务,以app的方式部署,可集群部署,支持高可用。使得两个broker集群都拥有全量的数据,互为灾备。
msha-manager
用于从整体上管控异地多活服务,主要是切流任务规划和实施,以app的方式部署,单点。基于nacos实现管控指令的下发。
 
说明:
nameserver,broker是rocketmq原生组件
msha-agent,msha-mq-sync,msha-manager是自研组件
 

3.2 接入RocketMQ双活能力应用需要做什么

(一)生产端:修改启动脚本,引入双活包

java -javaagent:lib/msha-agent-1.0.0.jar -Dmsha.commander.namespace=DemoNamespace -Dmsha.commander.addr=127.0.0.1:7810 -Dmsha.unit.name=unit-1 -jar rocketmq-lddl-client-demo-1.0.jar

(二)消费端:修改启动脚本,引入双活包

java -javaagent:lib/msha-agent-1.0.0.jar -Dmsha.commander.namespace=DemoNamespace -Dmsha.commander.addr=127.0.0.1:7810 -Dmsha.unit.name=unit-1 -jar rocketmq-lddl-client-demo-1.0.jar

(三)服务端:部署异地双活RocketMQ版本

部署msha-mq-sync集群,保证高可用,用于同步rocketmq broker数据。

(四)故障切换:集成RocketMQ的切流API

调用切流API
http://ip:port/switch?values=beijing
效果:将routerId为beijing的路由切换到另外一个机房,如果原本是unti-1,则切换到unit-2。
routerId由用户按业务需要配置。
 

4.以短线网关应用场景示例

4.1 业务场景介绍

在互联网服务,政企服务中有大量的发送短信场景,比如手机验证码,通知短信,营销短信等等,短信服务是广大企业客户快速触达手机用户所优先使用的通信能力。越大的平台,对短信服务的要求就越高,高吞吐量,低时延,可靠稳定,实时性高等特点是不可或缺的。
从短信网关需求来看,我们可以发现基于rocketmq实现短信网关是一个很好的选择
 

4.2 基于rocketmq实现短信网关

业务架构图
 
模块简介:
MTR —— 接受用户请求,做必要的业务处理,号码检查,内容检查,通道分配等,封装成消息提交到rocketmq集群。
Channel —— 从rocketmq集群上拉取未消费的消息,按协议提交到运营商网关,完成短信的发送,因为channel是集群部署,可横向扩展,提高短信发送的总速度。
 
因为rocketmq的引入,短信网关很方便就实现了高吞吐量,低时延,可靠稳定,实时性高等要求
 

4.3.短信网关实现异地双活

从上面短信网关的架构来看,我们可以看出这还是一个典型的单机房架构,即使在机房内实现了高可用,但是对于机房级别的故障,其容灾能力是不够的,还可以进一步提升。
基于自研的异地双活能力,我们可以很轻松实现短信网关的异地双活。
 
 
应用模块MTR,channel通过javaagent的方式启动,嵌入msha-agent代码,实现消息标记和消息过滤;
不同rocketmq broker集群之间通过msha-mq-sync同步数据,使得两个broker集群数据互为灾备,都拥有全量的数据。
 

4.4 效果演示

4.4.1 双机房同时对外提供服务。

机房unit-1的消息的生产和消费日志
 
2022-11-24 23:24:09.757  INFO 18664 --- [nio-8090-exec-1] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416393455239168,投递结果:SEND_OK
2022-11-24 23:24:10.509  INFO 18664 --- [nio-8090-exec-2] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416397083312128,投递结果:SEND_OK
2022-11-24 23:24:10.561  INFO 18664 --- [nio-8090-exec-3] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416397301415936,投递结果:SEND_OK
2022-11-24 23:24:10.628  INFO 18664 --- [nio-8090-exec-4] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416397498548224,投递结果:SEND_OK
2022-11-24 23:24:10.683  INFO 18664 --- [nio-8090-exec-5] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416397779566592,投递结果:SEND_OK
2022-11-24 23:24:10.733  INFO 18664 --- [nio-8090-exec-6] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398018641920,投递结果:SEND_OK
2022-11-24 23:24:10.784  INFO 18664 --- [nio-8090-exec-7] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398236745728,投递结果:SEND_OK
2022-11-24 23:24:10.834  INFO 18664 --- [nio-8090-exec-8] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398446460928,投递结果:SEND_OK
2022-11-24 23:24:10.893  INFO 18664 --- [nio-8090-exec-9] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398693924864,投递结果:SEND_OK
2022-11-24 23:24:10.939  INFO 18664 --- [io-8090-exec-10] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483416398891057152,投递结果:SEND_OK
2022-11-24 23:24:12.036  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416393455239168,消息来自机房:unit-1
2022-11-24 23:24:14.587  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416397083312128,消息来自机房:unit-1
2022-11-24 23:24:14.603  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398446460928,消息来自机房:unit-1
2022-11-24 23:24:14.618  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416397779566592,消息来自机房:unit-1
2022-11-24 23:24:14.634  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416397498548224,消息来自机房:unit-1
2022-11-24 23:24:14.650  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398693924864,消息来自机房:unit-1
2022-11-24 23:24:14.667  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416397301415936,消息来自机房:unit-1
2022-11-24 23:24:14.683  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398236745728,消息来自机房:unit-1
2022-11-24 23:24:14.700  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398018641920,消息来自机房:unit-1
2022-11-24 23:24:14.716  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483416398891057152,消息来自机房:unit-1
 
机房unit-2的消息的生产和消费日志
2022-11-24 23:24:10.124  INFO 21636 --- [nio-8092-exec-1] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416395149737984,投递结果:SEND_OK
2022-11-24 23:24:10.535  INFO 21636 --- [nio-8092-exec-3] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416397192364032,投递结果:SEND_OK
2022-11-24 23:24:10.584  INFO 21636 --- [nio-8092-exec-2] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416397402079232,投递结果:SEND_OK
2022-11-24 23:24:10.652  INFO 21636 --- [nio-8092-exec-4] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416397683097600,投递结果:SEND_OK
2022-11-24 23:24:10.707  INFO 21636 --- [nio-8092-exec-5] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416397913784320,投递结果:SEND_OK
2022-11-24 23:24:10.758  INFO 21636 --- [nio-8092-exec-6] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398127693824,投递结果:SEND_OK
2022-11-24 23:24:10.808  INFO 21636 --- [nio-8092-exec-7] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398337409024,投递结果:SEND_OK
2022-11-24 23:24:10.861  INFO 21636 --- [nio-8092-exec-8] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398555512832,投递结果:SEND_OK
2022-11-24 23:24:10.915  INFO 21636 --- [nio-8092-exec-9] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398786199552,投递结果:SEND_OK
2022-11-24 23:24:10.963  INFO 21636 --- [io-8092-exec-10] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483416398987526144,投递结果:SEND_OK
2022-11-24 23:24:24.469  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416395149737984,消息来自机房:unit-2
2022-11-24 23:24:24.486  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416397913784320,消息来自机房:unit-2
2022-11-24 23:24:27.000  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416397192364032,消息来自机房:unit-2
2022-11-24 23:24:27.015  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398337409024,消息来自机房:unit-2
2022-11-24 23:24:27.030  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416397402079232,消息来自机房:unit-2
2022-11-24 23:24:27.046  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416397683097600,消息来自机房:unit-2
2022-11-24 23:24:27.062  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398127693824,消息来自机房:unit-2
2022-11-24 23:24:27.076  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398555512832,消息来自机房:unit-2
2022-11-24 23:24:27.091  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398786199552,消息来自机房:unit-2
2022-11-24 23:24:27.106  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483416398987526144,消息来自机房:unit-2

 

从两个机房的生产和消费日志可见,短信id一一对应,每个机房只消费本机房生产的数据,数据机房内闭环。
 

4.4.2 发送过程中进行故障切换

当发现有一个机房出现故障,建设为机房unit-2故障,人工执行切流操作。
从日志中可以看到,路由切换以任务的形式(task=263d41ba-96d6-459e-a5c5-20458af54f07)分步骤执行(共7个步骤执行,phase=1到7),从2022-11-24 23:28:15.460开始,2022-11-24 23:28:23.577结束,耗时8.117秒。
 
机房unit-2切流日志 
2022-11-24 23:28:15.460  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=1
2022-11-24 23:28:16.869  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=2
2022-11-24 23:28:17.883  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=3
2022-11-24 23:28:19.003  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=4
2022-11-24 23:28:20.017  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=5
2022-11-24 23:28:21.133  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=6
2022-11-24 23:28:22.252  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07 , phase=7
2022-11-24 23:28:23.264  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Complete process MSHA route switch: task=263d41ba-96d6-459e-a5c5-20458af54f07
2022-11-24 23:28:23.577  INFO 18664 --- [0-DemoNamespace] SERVER                                   : application.unit-1         Changed MSHA metadata: key=metadata.routers , value=[Metadata.Router: unitName=unit-1 , values=[beijing, guangzhou], Metadata.Router: unitName=unit-2 , values=[]]
 
切换后机房unit-2的生产消费日志
2022-11-24 23:28:23.513  INFO 21636 --- [nio-8092-exec-8] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458263195648,投递结果:SEND_OK
2022-11-24 23:28:23.557  INFO 21636 --- [nio-8092-exec-9] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458447745024,投递结果:SEND_OK
2022-11-24 23:28:23.599  INFO 21636 --- [io-8092-exec-10] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458623905792,投递结果:SEND_OK
2022-11-24 23:28:23.641  INFO 21636 --- [nio-8092-exec-1] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458795872256,投递结果:SEND_OK
2022-11-24 23:28:23.680  INFO 21636 --- [nio-8092-exec-3] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417458959450112,投递结果:SEND_OK
2022-11-24 23:28:23.713  INFO 21636 --- [nio-8092-exec-2] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483417459102056448,投递结果:SEND_OK
 
从日志中可以看到,切流之后机房unit-2之投递消息到mq,并没有消费。
 
切流后机房1的生产消费日志
2022-11-24 23:28:37.461  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458263195648,消息来自机房:unit-2
2022-11-24 23:28:37.475  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417472561577984,消息来自机房:unit-1
2022-11-24 23:28:37.475  INFO 18664 --- [nio-8090-exec-6] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417516819873792,投递结果:SEND_OK
2022-11-24 23:28:37.490  INFO 18664 --- [nio-8090-exec-7] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417516886982656,投递结果:SEND_OK
2022-11-24 23:28:37.504  INFO 18664 --- [nio-8090-exec-8] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417516941508608,投递结果:SEND_OK
2022-11-24 23:28:37.517  INFO 18664 --- [nio-8090-exec-9] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517000228864,投递结果:SEND_OK
2022-11-24 23:28:37.525  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458623905792,消息来自机房:unit-2
2022-11-24 23:28:37.536  INFO 18664 --- [io-8090-exec-10] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517079920640,投递结果:SEND_OK
2022-11-24 23:28:37.540  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417473811480576,消息来自机房:unit-1
2022-11-24 23:28:37.549  INFO 18664 --- [nio-8090-exec-1] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517134446592,投递结果:SEND_OK
2022-11-24 23:28:37.554  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417456757440512,消息来自机房:unit-1
2022-11-24 23:28:37.563  INFO 18664 --- [nio-8090-exec-2] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517188972544,投递结果:SEND_OK
2022-11-24 23:28:37.578  INFO 18664 --- [nio-8090-exec-3] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517256081408,投递结果:SEND_OK
2022-11-24 23:28:37.592  INFO 18664 --- [nio-8090-exec-4] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517310607360,投递结果:SEND_OK
2022-11-24 23:28:37.592  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458795872256,消息来自机房:unit-2
2022-11-24 23:28:37.606  INFO 18664 --- [nio-8090-exec-5] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517369327616,投递结果:SEND_OK
2022-11-24 23:28:37.619  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417475547922432,消息来自机房:unit-1
2022-11-24 23:28:37.625  INFO 18664 --- [nio-8090-exec-6] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517453213696,投递结果:SEND_OK
2022-11-24 23:28:37.633  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458334498816,消息来自机房:unit-1
2022-11-24 23:28:37.639  INFO 18664 --- [nio-8090-exec-7] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517511933952,投递结果:SEND_OK
2022-11-24 23:28:37.647  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458716180480,消息来自机房:unit-1
2022-11-24 23:28:37.656  INFO 18664 --- [nio-8090-exec-8] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517579042816,投递结果:SEND_OK
2022-11-24 23:28:37.663  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459513098240,消息来自机房:unit-1
2022-11-24 23:28:37.668  INFO 18664 --- [nio-8090-exec-9] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517633568768,投递结果:SEND_OK
2022-11-24 23:28:37.683  INFO 18664 --- [io-8090-exec-10] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517696483328,投递结果:SEND_OK
2022-11-24 23:28:37.690  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459034947584,消息来自机房:unit-1
2022-11-24 23:28:37.697  INFO 18664 --- [nio-8090-exec-1] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517755203584,投递结果:SEND_OK
2022-11-24 23:28:37.711  INFO 18664 --- [nio-8090-exec-2] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517813923840,投递结果:SEND_OK
2022-11-24 23:28:37.719  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417475866689536,消息来自机房:unit-1
2022-11-24 23:28:37.728  INFO 18664 --- [nio-8090-exec-3] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517885227008,投递结果:SEND_OK
2022-11-24 23:28:37.733  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417460884635648,消息来自机房:unit-1
2022-11-24 23:28:37.742  INFO 18664 --- [nio-8090-exec-4] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517943947264,投递结果:SEND_OK
2022-11-24 23:28:37.746  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458447745024,消息来自机房:unit-2
2022-11-24 23:28:37.753  INFO 18664 --- [nio-8090-exec-5] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417517990084608,投递结果:SEND_OK
2022-11-24 23:28:37.760  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459164971008,消息来自机房:unit-1
2022-11-24 23:28:37.775  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459102056448,消息来自机房:unit-2
2022-11-24 23:28:37.776  INFO 18664 --- [nio-8090-exec-6] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518086553600,投递结果:SEND_OK
2022-11-24 23:28:37.788  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417476227399680,消息来自机房:unit-1
2022-11-24 23:28:37.788  INFO 18664 --- [nio-8090-exec-7] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518136885248,投递结果:SEND_OK
2022-11-24 23:28:37.802  INFO 18664 --- [nio-8090-exec-8] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518191411200,投递结果:SEND_OK
2022-11-24 23:28:37.802  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417461186625536,消息来自机房:unit-1
2022-11-24 23:28:37.815  INFO 18664 --- [nio-8090-exec-9] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518245937152,投递结果:SEND_OK
2022-11-24 23:28:37.817  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417459454377984,消息来自机房:unit-1
2022-11-24 23:28:37.829  INFO 18664 --- [io-8090-exec-10] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518308851712,投递结果:SEND_OK
2022-11-24 23:28:37.832  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417460146438144,消息来自机房:unit-1
2022-11-24 23:28:37.845  INFO 18664 --- [nio-8090-exec-1] com.example.controller.DemoController    : 机房:unit-1 - 投递消息到mq,短信id:483417518371766272,投递结果:SEND_OK
2022-11-24 23:28:37.846  INFO 18664 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-1 从mq消费消息 - 短信id:483417458959450112,消息来自机房:unit-2
从日志中可见,机房2产生的数据,sms id从到483417458263195648483417459102056448,共6条,都由机房1接收到并消费了,期间机房自身生产的消息继续正常消费,异地切流成功。

4.4.3 故障机房unit-2恢复,回切

切流日志,从2022-11-25 00:04:38.541开始,2022-11-25 00:04:51.168完成,耗时12.658秒。
2022-11-25 00:04:38.541  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=1
2022-11-25 00:04:39.349  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=2
2022-11-25 00:04:40.573  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=3
2022-11-25 00:04:41.692  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=4
2022-11-25 00:04:42.708  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=5
2022-11-25 00:04:43.844  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=6
2022-11-25 00:04:44.959  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398 , phase=7
2022-11-25 00:04:51.161  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Complete process MSHA route switch: task=530d585e-1b62-40fe-8356-f8ef8ed37398
2022-11-25 00:04:51.168  INFO 21636 --- [0-DemoNamespace] SERVER                                   : application.unit-2         Changed MSHA metadata: key=metadata.routers , value=[Metadata.Router: unitName=unit-1 , values=[beijing], Metadata.Router: unitName=unit-2 , values=[guangzhou]]

 

机房unit-2正常投入使用,回切成功。
2022-11-25 00:04:51.392  INFO 21636 --- [nio-8092-exec-4] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426634892836864,投递结果:SEND_OK
2022-11-25 00:04:51.431  INFO 21636 --- [nio-8092-exec-5] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635056414720,投递结果:SEND_OK
2022-11-25 00:04:51.473  INFO 21636 --- [nio-8092-exec-6] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635228381184,投递结果:SEND_OK
2022-11-25 00:04:51.514  INFO 21636 --- [nio-8092-exec-7] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635404541952,投递结果:SEND_OK
2022-11-25 00:04:51.560  INFO 21636 --- [nio-8092-exec-8] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635593285632,投递结果:SEND_OK
2022-11-25 00:04:51.600  INFO 21636 --- [nio-8092-exec-9] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635756863488,投递结果:SEND_OK
2022-11-25 00:04:51.637  INFO 21636 --- [io-8092-exec-10] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426635920441344,投递结果:SEND_OK
2022-11-25 00:04:51.681  INFO 21636 --- [nio-8092-exec-1] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426636100796416,投递结果:SEND_OK
2022-11-25 00:04:51.725  INFO 21636 --- [nio-8092-exec-3] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426636289540096,投递结果:SEND_OK
2022-11-25 00:04:51.765  INFO 21636 --- [nio-8092-exec-2] com.example.controller.DemoController    : 机房:unit-2 - 投递消息到mq,短信id:483426636457312256,投递结果:SEND_OK
2022-11-25 00:04:58.186  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426634892836864,消息来自机房:unit-2
2022-11-25 00:04:58.275  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635056414720,消息来自机房:unit-2
2022-11-25 00:04:58.200  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635228381184,消息来自机房:unit-2
2022-11-25 00:04:58.068  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635404541952,消息来自机房:unit-2
2022-11-25 00:04:58.082  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426636289540096,消息来自机房:unit-2
2022-11-25 00:04:58.215  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635756863488,消息来自机房:unit-2
2022-11-25 00:04:58.260  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635593285632,消息来自机房:unit-2
2022-11-25 00:04:58.290  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426635920441344,消息来自机房:unit-2
2022-11-25 00:04:58.304  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426636457312256,消息来自机房:unit-2
2022-11-25 00:04:58.318  INFO 21636 --- [MessageThread_1] com.example.mq.MshaConsumer              : 机房:unit-2 从mq消费消息 - 短信id:483426636100796416,消息来自机房:unit-2
文章来自个人专栏
rocketmq专栏
4 文章 | 1 订阅
0条评论
0 / 1000
请输入你的评论
1
0