Kafka Producer机制优化-提高发送消息可靠性


名称解释:

Broker:负责消息的存储和转发,也可以叫消息中介节点
Topic:每种消息的分类叫做主题(Topic)。
Partition:每一个Topic被切分为多个Partitions。

背景

Producer构造Message对象时,传入key参数,当Producer发送Message,会根据key确定目标Partition,当Kafka集群中某个Partition所有存活的节点都失效或挂掉。会造成Producer尝试重新发送message.send.max.retries(默认值为3)次后抛出Exception,每次尝试都会休眠一定时间(默认值为100ms)。用户捕捉到异常其结果,停止发送会阻塞,继续发送消息会丢失。

解决思路

Kafka中默认发送消息方式不变,给用户提供一种可选择方式,增加一个消息发送失效转移的开关,当Producer发送到目标Partition的(所有副本本来存活的)节点都失效或挂掉,就转发到其他Partition上。
M:表示Partition的主分区。
S:表示Partition的从分区。下面图2所示,消息轮询发送到Partition的0-3上。
这里写图片描述
图1
上图本来一条消息是发送到Partition-0所在Broker的,当Partition-0所在Broker全部失效或挂掉后,如下图所示。
这里写图片描述
图2
消息(轮询均衡)发送到其他Partition上,待失效的Brokers恢复,发送消息恢复到图1状态。

Producer代码修改

代码修改如下:
ProducerConfig.scala类代码中增加一行:
val sendSwitchBrokerEnabled = props.getBoolean(“send.switch.broker.enable”, false)
DefaultEventHandler.scala类增加一行
val isSendSwitchBrokerEnabled = config.sendSwitchBrokerEnabled

DefaultEventHandler.scala类中增加如下方法:

private def getPartition(topic: String, key: Any, topicPartitionList: Seq[PartitionAndLeader], isSendSwitchBrokerEnabled: Boolean = false): Int = {
val numPartitions = topicPartitionList.size
if(numPartitions <= 0)
throw new UnknownTopicOrPartitionException("Topic " + topic + " doesn't exist")
var partition =
if (key == null) {
// If the key is null, we don't really need a partitioner
// So we look up in the send partition cache for the topic to decide the target partition
val id = sendPartitionPerTopicCache.get(topic)
id match {
case Some(partitionId) =>
// directly return the partitionId without checking availability of the leader,
// since we want to postpone the failure until the send operation anyways
partitionId
case None =>
val availablePartitions = topicPartitionList.filter(_.leaderBrokerIdOpt.isDefined)
if (availablePartitions.isEmpty)
throw new LeaderNotAvailableException("No leader for any partition in topic " + topic)
val index = Utils.abs(Random.nextInt) % availablePartitions.size
val partitionId = availablePartitions(index).partitionId
sendPartitionPerTopicCache.put(topic, partitionId)
partitionId
}
} else
partitioner.partition(key, numPartitions)

if(partition < 0 || partition >= numPartitions)
throw new UnknownTopicOrPartitionException("Invalid partition id: " + partition + " for topic " + topic +
"; Valid values are in the inclusive range of [0, " + (numPartitions-1) + "]")
trace("Assigning message of topic %s and key %s to a selected partition %d".format(topic, if (key == null) "[none]" else key.toString, partition))

if (isSendSwitchBrokerEnabled) {
if(partitionsLeaderInTopicsCache.isEmpty()) {
val availablePartitions = topicPartitionList.map { partitionAndLeader =>
if(partitionAndLeader.leaderBrokerIdOpt.isDefined) {
partitionsLeaderInTopicsCache.put(TopicAndPartition(topic, partitionAndLeader.partitionId),
partitionAndLeader.leaderBrokerIdOpt.get)
}
}
}

if(!partitionsLeaderInTopicsCache.containsKey(TopicAndPartition(topic, partition))) {
val availablePartitions = topicPartitionList.filter(_.leaderBrokerIdOpt.isDefined)
if (availablePartitions.isEmpty)
throw new LeaderNotAvailableException("No leader for any partition in topic " + topic)
val index = Utils.abs(Random.nextInt) % availablePartitions.size
partition = availablePartitions(index).partitionId
}
}
partition
}
智能推荐

注意!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系我们删除。



猜您在找
kafka消息的可靠性 Kafka的存储机制以及可靠性 Kafka 幂等,事物,消息可靠性 Kafka如何保证消息的可靠性传输 RabbitMQ 和 Kafka 的消息可靠性对比
智能推荐
 
© 2014-2019 ITdaan.com 粤ICP备14056181号  

赞助商广告