Where’s my message? Durability and you-程序员宅基地

技术标签: RabbitMQ  

There’s a dirty secret about creating queues and exchanges in Rabbit: by default they don’t survive reboot. That’s right; restart your RabbitMQ server and watch those queues and exchanges go poof (along with the messages inside). 

The reason is because of a property on every queue and exchange called  durable. It defaults to false, and tells RabbitMQ whether the queue (or exchange) should be re-created after a crash or restart of Rabbit. Set it to true and you won’t have to re-create those queues and exchanges when the power supply in your server dies. You might also think that setting durable to true on the exchanges and queues is all you need to do to make your messages survive a reboot, but you’d be wrong. Whereas queues and exchanges must be durable to allow messages to survive reboot, it isn’t enough on its own. 

 A message that can survive a crash of the AMQP broker is called persistent. You flag a message as persistent by setting the  delivery mode option of the message to 2 (your AMQP client may use a human-friendly constant instead) before publishing it. At this  point, the message is indicated as persistent, but it must be published to an exchange  that’s durable and arrive in a queue that’s durable to survive. If this weren’t the case,  the queue (or exchange) a persistent message was sitting in when Rabbit crashed  wouldn’t exist when Rabbit restarted, thereby orphaning the message. So, for a mes sage that’s in flight inside Rabbit to survive a crash, the message must 

Have its delivery mode option set to 2 (persistent) 
 Be published into a durable exchange 
 Arrive in a durable queue 

Do these three things and you won't have to play Where’s Waldo with your critical
messages. 
 The way that RabbitMQ ensures persistent messages survive a restart is by writing them to the disk inside of a persistency log file. When you publish a persistent message to a durable exchange, Rabbit won’t send the response until the message is committed to the log file. Keep in mind, though, that if it gets routed to a non durable queue after that, it’s automatically removed from the persistency log and won’t survive a restart. When you use persistent messages it’s crucial that you make sure  all three elements required for a message to persist are in place (we can’t stress this enough).
Once you consume a persistent message from a durable queue (and acknowledge it), RabbitMQ flags it in the persistency log for garbage collection. If Rabbit restarts any-time before you consume a persistent message, it’ll automatically re-create theexchanges and queues (and bindings) and replay any messages in the persistency log
into the appropriate queues or exchanges (depending on where in the routing process the messages were when Rabbit died). 


 You might be thinking that you should use persistent messaging for all of your messages. You could do that, but you’d pay a price for ensuring your messages survive Rabbit restarts: performance. The act of writing messages to disk is much slower than just storing them in RAM, and will significantly decrease the number of messages per second your RabbitMQ server can process. It’s not uncommon to see a 10x or more decrease in message throughput when using persistency.

There’s also the issue thatpersistent messages don’t play well with RabbitMQ’s built-in clustering. Though
RabbitMQ clustering allows you to talk to any queue present in the cluster from any
node, those queues are actually evenly distributed among the nodes  without redun-
dancy (there’s no backup copy of any queue on a second node in the cluster). If the
cluster node hosting your seed_bin queue crashes, the queue disappears from the
cluster until the node is restored … if the queue was durable. More important, while
the node is down its queues aren’t available and the durable ones can’t be re-created.
This can lead to black-holing of messages. We’ll cover the behavior in more detail and
show alternate clustering approaches to get around this in chapter 5. 
 Given the trade-offs, when should you use persistent/durable messaging? First, you
need to analyze (and test) your performance needs. Do you need to process 100,000

messages per second on a single Rabbit server? If so, you should probably look at
other ways of ensuring message delivery (or get a very fast storage system). For exam-
ple, your producer could listen to a reply queue on a separate channel. Every time it
publishes a message, it includes the name of the reply queue so that the consumer can
send a reply back to confirm receipt. If a message isn’t replied to within a reasonable
amount of time, the producer can republish the message. That said, the critical
nature of messages requiring guaranteed delivery generally means they’re lower in
volume than other types of messages (such as logging messages). So if persistent mes-
saging meets your performance needs, it’s an excellent way to help ensure delivery.
We use it a lot for critical messages. We’re just selective about what types of content
use persistent messaging. For example, we run two types of Rabbit clusters: traditional
RabbitMQ clustering for nonpersistent messaging, and pairs of active/hot-standby
nonclustered Rabbit servers for persistent messaging (using load balancers). This
ensures the processing load for persistent messaging doesn’t slow down nonpersistent
messages. It also means Rabbit’s built-in clustering won’t black-hole persistent mes-
sages when a node dies. Do keep in mind that while Rabbit can help ensure delivery, it
can never absolutely guarantee it. Hard drive corruption, buggy behavior by a con-
sumer, or other extreme events can trash/black-hole persistent messages. It’s ulti-
mately up to you to ensure your messages arrive where they need to go, and persistent
messaging is a great tool to help you get there. 
 A concept that’s related to the durability of a message is the AMQP transaction. So
far we’ve talked about marking messages, queues, and exchanges as durable. That’s all
well and good for keeping a message safe once RabbitMQ has it in its custody, but
since a publish operation returns no response to the producer, how do you know if
the broker has persisted the durable message to disk? Should the broker die before it
can write the message to disk, the message would be lost and you wouldn’t know.
That’s where transactions come in. When you absolutely need to be sure the broker
has the message in custody (and has routed the message to all matching subscribed
queues) before you move on to another t奋斗ask, you need to wrap it in a transaction. If 
you come from a database background, it’s important not to confuse AMQP transactions with what “transaction” means in most databases. In  AMQP, after you place a channel into transaction mode, you send it the publish you want to confirm, followed

by zero or more other AMQP commands that should be executed or ignored depend-
ing on whether the initial publish succeeded. Once you’ve sent all of the commands,
you commit the transaction. If the transaction’s initial publish succeeds, then the chan-
nel will complete the other AMQP commands in the transaction. If the publish fails,
none of the other  AMQP commands will be executed. Transactions close the “last
mile” gap between producers publishing messages and RabbitMQ committing them
to disk, but there’s a better way to close that gap. 
 Though transactions are a part of the formal AMQP 0-9-1 specification, they have
an Achilles heel in that they’re huge drains on Rabbit performance. Not only can
using transactions drop your message throughput by a factor of 2–10x, but they also
make your producer app synchronous, which is one of the things you’re trying to get 
rid of with messaging. Knowing all of this, the guys at RabbitMQ decided to come up with a better way to ensure message delivery: publisher confirms.

2
 Similar to transactions,
you have to tell Rabbit to place the channel into confirm mode, and you can’t turn it
off without re-creating the channel. Once a channel is in confirm mode, every mes-
sage published on the channel will be assigned a unique  ID number (starting at 1).
Once the message has been delivered to all queues that have bindings matching the
message’s routing key, the channel will issue a publisher confirm to the producer app
(containing the message’s unique  ID). This lets the producer know the message has
been safely queued at all of its destinations. If the message and the queues are dura-
ble, the confirm is issued only after the queues have written the message to disk. The
major benefit of publisher confirms is that they’re asynchronous. Once a message has
been published, the producer app can go on to the next message while waiting for the
confirm. When the confirm for that message is finally received, a callback function in
the producer app will be fired so it can wake up and handle the confirmation
. If an
internal error occurs inside Rabbit that causes a message to be lost, Rabbit will send a
message nack (not acknowledged) that’s like a publisher confirm (it has the message’s
unique ID) but indicates the message was lost. Also, since there’s no concept of mes-
sage rollback (as with transactions), publisher confirms are much lighter weight and
have an almost negligible performance hit on the Rabbit broker. 
 Now you have the individual parts of RabbitMQ down, from consumers and pro-
ducers to durable messaging, but how do they all fit together? What does the lifecycle
of an actual message look like from beginning to end? The best way to answer that is
to look at the life of a message in code. 

版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/hanruikai/article/details/49024411

智能推荐

c# 调用c++ lib静态库_c#调用lib-程序员宅基地

文章浏览阅读2w次,点赞7次,收藏51次。四个步骤1.创建C++ Win32项目动态库dll 2.在Win32项目动态库中添加 外部依赖项 lib头文件和lib库3.导出C接口4.c#调用c++动态库开始你的表演...①创建一个空白的解决方案,在解决方案中添加 Visual C++ , Win32 项目空白解决方案的创建:添加Visual C++ , Win32 项目这......_c#调用lib

deepin/ubuntu安装苹方字体-程序员宅基地

文章浏览阅读4.6k次。苹方字体是苹果系统上的黑体,挺好看的。注重颜值的网站都会使用,例如知乎:font-family: -apple-system, BlinkMacSystemFont, Helvetica Neue, PingFang SC, Microsoft YaHei, Source Han Sans SC, Noto Sans CJK SC, W..._ubuntu pingfang

html表单常见操作汇总_html表单的处理程序有那些-程序员宅基地

文章浏览阅读159次。表单表单概述表单标签表单域按钮控件demo表单标签表单标签基本语法结构<form action="处理数据程序的url地址“ method=”get|post“ name="表单名称”></form><!--action,当提交表单时,向何处发送表单中的数据,地址可以是相对地址也可以是绝对地址--><!--method将表单中的数据传送给服务器处理,get方式直接显示在url地址中,数据可以被缓存,且长度有限制;而post方式数据隐藏传输,_html表单的处理程序有那些

PHP设置谷歌验证器(Google Authenticator)实现操作二步验证_php otp 验证器-程序员宅基地

文章浏览阅读1.2k次。使用说明:开启Google的登陆二步验证(即Google Authenticator服务)后用户登陆时需要输入额外由手机客户端生成的一次性密码。实现Google Authenticator功能需要服务器端和客户端的支持。服务器端负责密钥的生成、验证一次性密码是否正确。客户端记录密钥后生成一次性密码。下载谷歌验证类库文件放到项目合适位置(我这边放在项目Vender下面)https://github.com/PHPGangsta/GoogleAuthenticatorPHP代码示例://引入谷_php otp 验证器

【Python】matplotlib.plot画图横坐标混乱及间隔处理_matplotlib更改横轴间距-程序员宅基地

文章浏览阅读4.3k次,点赞5次,收藏11次。matplotlib.plot画图横坐标混乱及间隔处理_matplotlib更改横轴间距

docker — 容器存储_docker 保存容器-程序员宅基地

文章浏览阅读2.2k次。①Storage driver 处理各镜像层及容器层的处理细节,实现了多层数据的堆叠,为用户 提供了多层数据合并后的统一视图②所有 Storage driver 都使用可堆叠图像层和写时复制(CoW)策略③docker info 命令可查看当系统上的 storage driver主要用于测试目的,不建议用于生成环境。_docker 保存容器

随便推点

网络拓扑结构_网络拓扑csdn-程序员宅基地

文章浏览阅读834次,点赞27次,收藏13次。网络拓扑结构是指计算机网络中各组件(如计算机、服务器、打印机、路由器、交换机等设备)及其连接线路在物理布局或逻辑构型上的排列形式。这种布局不仅描述了设备间的实际物理连接方式,也决定了数据在网络中流动的路径和方式。不同的网络拓扑结构影响着网络的性能、可靠性、可扩展性及管理维护的难易程度。_网络拓扑csdn

JS重写Date函数,兼容IOS系统_date.prototype 将所有 ios-程序员宅基地

文章浏览阅读1.8k次,点赞5次,收藏8次。IOS系统Date的坑要创建一个指定时间的new Date对象时,通常的做法是:new Date("2020-09-21 11:11:00")这行代码在 PC 端和安卓端都是正常的,而在 iOS 端则会提示 Invalid Date 无效日期。在IOS年月日中间的横岗许换成斜杠,也就是new Date("2020/09/21 11:11:00")通常为了兼容IOS的这个坑,需要做一些额外的特殊处理,笔者在开发的时候经常会忘了兼容IOS系统。所以就想试着重写Date函数,一劳永逸,避免每次ne_date.prototype 将所有 ios

如何将EXCEL表导入plsql数据库中-程序员宅基地

文章浏览阅读5.3k次。方法一:用PLSQL Developer工具。 1 在PLSQL Developer的sql window里输入select * from test for update; 2 按F8执行 3 打开锁, 再按一下加号. 鼠标点到第一列的列头,使全列成选中状态,然后粘贴,最后commit提交即可。(前提..._excel导入pl/sql

Git常用命令速查手册-程序员宅基地

文章浏览阅读83次。Git常用命令速查手册1、初始化仓库git init2、将文件添加到仓库git add 文件名 # 将工作区的某个文件添加到暂存区 git add -u # 添加所有被tracked文件中被修改或删除的文件信息到暂存区,不处理untracked的文件git add -A # 添加所有被tracked文件中被修改或删除的文件信息到暂存区,包括untracked的文件...

分享119个ASP.NET源码总有一个是你想要的_千博二手车源码v2023 build 1120-程序员宅基地

文章浏览阅读202次。分享119个ASP.NET源码总有一个是你想要的_千博二手车源码v2023 build 1120

【C++缺省函数】 空类默认产生的6个类成员函数_空类默认产生哪些类成员函数-程序员宅基地

文章浏览阅读1.8k次。版权声明:转载请注明出处 http://blog.csdn.net/irean_lau。目录(?)[+]1、缺省构造函数。2、缺省拷贝构造函数。3、 缺省析构函数。4、缺省赋值运算符。5、缺省取址运算符。6、 缺省取址运算符 const。[cpp] view plain copy_空类默认产生哪些类成员函数

推荐文章

热门文章

相关标签