{"id":1118,"date":"2019-01-08T14:10:23","date_gmt":"2019-01-08T06:10:23","guid":{"rendered":"https:\/\/blog.jsjs.org\/?p=1118"},"modified":"2019-01-08T14:10:23","modified_gmt":"2019-01-08T06:10:23","slug":"kafka%e5%8f%b2%e4%b8%8a%e6%9c%80%e8%af%a6%e7%bb%86%e5%8e%9f%e7%90%86%e6%80%bb%e7%bb%93","status":"publish","type":"post","link":"https:\/\/blog.jsjs.org\/?p=1118","title":{"rendered":"Kafka\u53f2\u4e0a\u6700\u8be6\u7ec6\u539f\u7406\u603b\u7ed3"},"content":{"rendered":"<div>Kafka<\/div>\n<div>Kafka\u662f\u6700\u521d\u7531Linkedin\u516c\u53f8\u5f00\u53d1\uff0c\u662f\u4e00\u4e2a\u5206\u5e03\u5f0f\u3001\u652f\u6301\u5206\u533a\u7684\uff08partition\uff09\u3001\u591a\u526f\u672c\u7684\uff08replica\uff09\uff0c\u57fa\u4e8ezookeeper\u534f\u8c03\u7684\u5206\u5e03\u5f0f\u6d88\u606f\u7cfb\u7edf\uff0c\u5b83\u7684\u6700\u5927\u7684\u7279\u6027\u5c31\u662f\u53ef\u4ee5\u5b9e\u65f6\u7684\u5904\u7406\u5927\u91cf\u6570\u636e\u4ee5\u6ee1\u8db3\u5404\u79cd\u9700\u6c42\u573a\u666f\uff1a\u6bd4\u5982\u57fa\u4e8ehadoop\u7684\u6279\u5904\u7406\u7cfb\u7edf\u3001\u4f4e\u5ef6\u8fdf\u7684\u5b9e\u65f6\u7cfb\u7edf\u3001storm\/Spark\u6d41\u5f0f\u5904\u7406\u5f15\u64ce\uff0cweb\/nginx\u65e5\u5fd7\u3001\u8bbf\u95ee\u65e5\u5fd7\uff0c\u6d88\u606f\u670d\u52a1\u7b49\u7b49\uff0c\u7528scala\u8bed\u8a00\u7f16\u5199\uff0cLinkedin\u4e8e2010\u5e74\u8d21\u732e\u7ed9\u4e86Apache\u57fa\u91d1\u4f1a\u5e76\u6210\u4e3a\u9876\u7ea7\u5f00\u6e90 \u9879\u76ee\u3002<\/div>\n<div><\/div>\n<div>\n<div>1.\u524d\u8a00<\/div>\n<div>\n<div>\u6d88\u606f\u961f\u5217\u7684\u6027\u80fd\u597d\u574f\uff0c\u5176\u6587\u4ef6\u5b58\u50a8\u673a\u5236\u8bbe\u8ba1\u662f\u8861\u91cf\u4e00\u4e2a\u6d88\u606f\u961f\u5217\u670d\u52a1\u6280\u672f\u6c34\u5e73\u548c\u6700\u5173\u952e\u6307\u6807\u4e4b\u4e00\u3002\u4e0b\u9762\u5c06\u4eceKafka\u6587\u4ef6\u5b58\u50a8\u673a\u5236\u548c\u7269\u7406\u7ed3\u6784\u89d2\u5ea6\uff0c\u5206\u6790Kafka\u662f\u5982\u4f55\u5b9e\u73b0\u9ad8\u6548\u6587\u4ef6\u5b58\u50a8\uff0c\u53ca\u5b9e\u9645\u5e94\u7528\u6548\u679c\u3002<\/div>\n<div><\/div>\n<div>\u00a01.1 \u00a0Kafka\u7684\u7279\u6027:<\/div>\n<div>&#8211; \u9ad8\u541e\u5410\u91cf\u3001\u4f4e\u5ef6\u8fdf\uff1akafka\u6bcf\u79d2\u53ef\u4ee5\u5904\u7406\u51e0\u5341\u4e07\u6761\u6d88\u606f\uff0c\u5b83\u7684\u5ef6\u8fdf\u6700\u4f4e\u53ea\u6709\u51e0\u6beb\u79d2\uff0c\u6bcf\u4e2atopic\u53ef\u4ee5\u5206\u591a\u4e2apartition, consumer group \u5bf9partition\u8fdb\u884cconsume\u64cd\u4f5c\u3002<\/div>\n<div>&#8211; \u53ef\u6269\u5c55\u6027\uff1akafka\u96c6\u7fa4\u652f\u6301\u70ed\u6269\u5c55<\/div>\n<div>&#8211; \u6301\u4e45\u6027\u3001\u53ef\u9760\u6027\uff1a\u6d88\u606f\u88ab\u6301\u4e45\u5316\u5230\u672c\u5730\u78c1\u76d8\uff0c\u5e76\u4e14\u652f\u6301\u6570\u636e\u5907\u4efd\u9632\u6b62\u6570\u636e\u4e22\u5931<\/div>\n<div>&#8211; \u5bb9\u9519\u6027\uff1a\u5141\u8bb8\u96c6\u7fa4\u4e2d\u8282\u70b9\u5931\u8d25\uff08\u82e5\u526f\u672c\u6570\u91cf\u4e3an,\u5219\u5141\u8bb8n-1\u4e2a\u8282\u70b9\u5931\u8d25\uff09<\/div>\n<div>&#8211; \u9ad8\u5e76\u53d1\uff1a\u652f\u6301\u6570\u5343\u4e2a\u5ba2\u6237\u7aef\u540c\u65f6\u8bfb\u5199<\/div>\n<div><\/div>\n<div>1.2 \u00a0 Kafka\u7684\u4f7f\u7528\u573a\u666f\uff1a<\/div>\n<div>&#8211; \u65e5\u5fd7\u6536\u96c6\uff1a\u4e00\u4e2a\u516c\u53f8\u53ef\u4ee5\u7528Kafka\u53ef\u4ee5\u6536\u96c6\u5404\u79cd\u670d\u52a1\u7684log\uff0c\u901a\u8fc7kafka\u4ee5\u7edf\u4e00\u63a5\u53e3\u670d\u52a1\u7684\u65b9\u5f0f\u5f00\u653e\u7ed9\u5404\u79cdconsumer\uff0c\u4f8b\u5982hadoop\u3001Hbase\u3001Solr\u7b49\u3002<\/div>\n<div>&#8211; \u6d88\u606f\u7cfb\u7edf\uff1a\u89e3\u8026\u548c\u751f\u4ea7\u8005\u548c\u6d88\u8d39\u8005\u3001\u7f13\u5b58\u6d88\u606f\u7b49\u3002<\/div>\n<div>&#8211; \u7528\u6237\u6d3b\u52a8\u8ddf\u8e2a\uff1aKafka\u7ecf\u5e38\u88ab\u7528\u6765\u8bb0\u5f55web\u7528\u6237\u6216\u8005app\u7528\u6237\u7684\u5404\u79cd\u6d3b\u52a8\uff0c\u5982\u6d4f\u89c8\u7f51\u9875\u3001\u641c\u7d22\u3001\u70b9\u51fb\u7b49\u6d3b\u52a8\uff0c\u8fd9\u4e9b\u6d3b\u52a8\u4fe1\u606f\u88ab\u5404\u4e2a\u670d\u52a1\u5668\u53d1\u5e03\u5230kafka\u7684topic\u4e2d\uff0c\u7136\u540e\u8ba2\u9605\u8005\u901a\u8fc7\u8ba2\u9605\u8fd9\u4e9btopic\u6765\u505a\u5b9e\u65f6\u7684\u76d1\u63a7\u5206\u6790\uff0c\u6216\u8005\u88c5\u8f7d\u5230hadoop\u3001\u6570\u636e\u4ed3\u5e93\u4e2d\u505a\u79bb\u7ebf\u5206\u6790\u548c\u6316\u6398\u3002<\/div>\n<div>&#8211; \u8fd0\u8425\u6307\u6807\uff1aKafka\u4e5f\u7ecf\u5e38\u7528\u6765\u8bb0\u5f55\u8fd0\u8425\u76d1\u63a7\u6570\u636e\u3002\u5305\u62ec\u6536\u96c6\u5404\u79cd\u5206\u5e03\u5f0f\u5e94\u7528\u7684\u6570\u636e\uff0c\u751f\u4ea7\u5404\u79cd\u64cd\u4f5c\u7684\u96c6\u4e2d\u53cd\u9988\uff0c\u6bd4\u5982\u62a5\u8b66\u548c\u62a5\u544a\u3002<\/div>\n<div>&#8211; \u6d41\u5f0f\u5904\u7406\uff1a\u6bd4\u5982spark streaming\u548cstorm<\/div>\n<div>&#8211; \u4e8b\u4ef6\u6e90<\/div>\n<div><\/div>\n<div>1.3 \u00a0Kakfa\u7684\u8bbe\u8ba1\u601d\u60f3<\/div>\n<div>&#8211;\u00a0Kakfa Broker Leader\u7684\u9009\u4e3e\uff1aKakfa Broker\u96c6\u7fa4\u53d7Zookeeper\u7ba1\u7406\u3002\u6240\u6709\u7684Kafka Broker\u8282\u70b9\u4e00\u8d77\u53bbZookeeper\u4e0a\u6ce8\u518c\u4e00\u4e2a\u4e34\u65f6\u8282\u70b9\uff0c\u56e0\u4e3a\u53ea\u6709\u4e00\u4e2aKafka Broker\u4f1a\u6ce8\u518c\u6210\u529f\uff0c\u5176\u4ed6\u7684\u90fd\u4f1a\u5931\u8d25\uff0c\u6240\u4ee5\u8fd9\u4e2a\u6210\u529f\u5728Zookeeper\u4e0a\u6ce8\u518c\u4e34\u65f6\u8282\u70b9\u7684\u8fd9\u4e2aKafka Broker\u4f1a\u6210\u4e3aKafka Broker Controller\uff0c\u5176\u4ed6\u7684Kafka broker\u53ebKafka Broker follower\u3002\uff08\u8fd9\u4e2a\u8fc7\u7a0b\u53ebController\u5728ZooKeeper\u6ce8\u518cWatch\uff09\u3002\u8fd9\u4e2aController\u4f1a\u76d1\u542c\u5176\u4ed6\u7684Kafka Broker\u7684\u6240\u6709\u4fe1\u606f\uff0c\u5982\u679c\u8fd9\u4e2akafka broker controller\u5b95\u673a\u4e86\uff0c\u5728zookeeper\u4e0a\u9762\u7684\u90a3\u4e2a\u4e34\u65f6\u8282\u70b9\u5c31\u4f1a\u6d88\u5931\uff0c\u6b64\u65f6\u6240\u6709\u7684kafka broker\u53c8\u4f1a\u4e00\u8d77\u53bbZookeeper\u4e0a\u6ce8\u518c\u4e00\u4e2a\u4e34\u65f6\u8282\u70b9\uff0c\u56e0\u4e3a\u53ea\u6709\u4e00\u4e2aKafka Broker\u4f1a\u6ce8\u518c\u6210\u529f\uff0c\u5176\u4ed6\u7684\u90fd\u4f1a\u5931\u8d25\uff0c\u6240\u4ee5\u8fd9\u4e2a\u6210\u529f\u5728Zookeeper\u4e0a\u6ce8\u518c\u4e34\u65f6\u8282\u70b9\u7684\u8fd9\u4e2aKafka Broker\u4f1a\u6210\u4e3aKafka Broker Controller\uff0c\u5176\u4ed6\u7684Kafka broker\u53ebKafka Broker follower\u3002\u4f8b\u5982\uff1a\u4e00\u65e6\u6709\u4e00\u4e2abroker\u5b95\u673a\u4e86\uff0c\u8fd9\u4e2akafka broker controller\u4f1a\u8bfb\u53d6\u8be5\u5b95\u673abroker\u4e0a\u6240\u6709\u7684partition\u5728zookeeper\u4e0a\u7684\u72b6\u6001\uff0c\u5e76\u9009\u53d6ISR\u5217\u8868\u4e2d\u7684\u4e00\u4e2areplica\u4f5c\u4e3apartition leader\uff08\u5982\u679cISR\u5217\u8868\u4e2d\u7684replica\u5168\u6302\uff0c\u9009\u4e00\u4e2a\u5e78\u5b58\u7684replica\u4f5c\u4e3aleader; \u5982\u679c\u8be5partition\u7684\u6240\u6709\u7684replica\u90fd\u5b95\u673a\u4e86\uff0c\u5219\u5c06\u65b0\u7684leader\u8bbe\u7f6e\u4e3a-1\uff0c\u7b49\u5f85\u6062\u590d\uff0c\u7b49\u5f85ISR\u4e2d\u7684\u4efb\u4e00\u4e2aReplica\u201c\u6d3b\u201d\u8fc7\u6765\uff0c\u5e76\u4e14\u9009\u5b83\u4f5c\u4e3aLeader\uff1b\u6216\u9009\u62e9\u7b2c\u4e00\u4e2a\u201c\u6d3b\u201d\u8fc7\u6765\u7684Replica\uff08\u4e0d\u4e00\u5b9a\u662fISR\u4e2d\u7684\uff09\u4f5c\u4e3aLeader\uff09\uff0c\u8fd9\u4e2abroker\u5b95\u673a\u7684\u4e8b\u60c5\uff0ckafka controller\u4e5f\u4f1a\u901a\u77e5zookeeper\uff0czookeeper\u5c31\u4f1a\u901a\u77e5\u5176\u4ed6\u7684kafka broker\u3002<\/div>\n<div>\u8fd9\u91cc\u66fe\u7ecf\u53d1\u751f\u8fc7\u4e00\u4e2abug\uff0cTalkingData\u4f7f\u7528Kafka0.8.1\u7684\u65f6\u5019\uff0ckafka controller\u5728Zookeeper\u4e0a\u6ce8\u518c\u6210\u529f\u540e\uff0c\u5b83\u548cZookeeper\u901a\u4fe1\u7684timeout\u65f6\u95f4\u662f6s\uff0c\u4e5f\u5c31\u662f\u5982\u679ckafka controller\u5982\u679c\u67096s\u4e2d\u6ca1\u6709\u548cZookeeper\u505a\u5fc3\u8df3\uff0c\u90a3\u4e48Zookeeper\u5c31\u8ba4\u4e3a\u8fd9\u4e2akafka controller\u5df2\u7ecf\u6b7b\u4e86\uff0c\u5c31\u4f1a\u5728Zookeeper\u4e0a\u628a\u8fd9\u4e2a\u4e34\u65f6\u8282\u70b9\u5220\u6389\uff0c\u90a3\u4e48\u5176\u4ed6Kafka\u5c31\u4f1a\u8ba4\u4e3acontroller\u5df2\u7ecf\u6ca1\u4e86\uff0c\u5c31\u4f1a\u518d\u6b21\u62a2\u7740\u6ce8\u518c\u4e34\u65f6\u8282\u70b9\uff0c\u6ce8\u518c\u6210\u529f\u7684\u90a3\u4e2akafka broker\u6210\u4e3acontroller\uff0c\u7136\u540e\uff0c\u4e4b\u524d\u7684\u90a3\u4e2akafka controller\u5c31\u9700\u8981\u5404\u79cdshut down\u53bb\u5173\u95ed\u5404\u79cd\u8282\u70b9\u548c\u4e8b\u4ef6\u7684\u76d1\u542c\u3002\u4f46\u662f\u5f53kafka\u7684\u8bfb\u5199\u6d41\u91cf\u90fd\u975e\u5e38\u5de8\u5927\u7684\u65f6\u5019\uff0cTalkingData\u7684\u4e00\u4e2abug\u662f\uff0c\u7531\u4e8e\u7f51\u7edc\u7b49\u539f\u56e0\uff0ckafka controller\u548cZookeeper\u67096s\u4e2d\u6ca1\u6709\u901a\u4fe1\uff0c\u4e8e\u662f\u91cd\u65b0\u9009\u4e3e\u51fa\u4e86\u4e00\u4e2a\u65b0\u7684kafka controller\uff0c\u4f46\u662f\u539f\u6765\u7684controller\u5728shut down\u7684\u65f6\u5019\u603b\u662f\u4e0d\u6210\u529f\uff0c\u8fd9\u4e2a\u65f6\u5019producer\u8fdb\u6765\u7684message\u7531\u4e8eKafka\u96c6\u7fa4\u4e2d\u5b58\u5728\u4e24\u4e2akafka controller\u800c\u65e0\u6cd5\u843d\u5730\u3002\u5bfc\u81f4\u6570\u636e\u6de4\u79ef\u3002<\/div>\n<div>\u8fd9\u91cc\u66fe\u7ecf\u8fd8\u6709\u4e00\u4e2abug\uff0cTalkingData\u4f7f\u7528Kafka0.8.1\u7684\u65f6\u5019\uff0c\u5f53ack=0\u7684\u65f6\u5019\uff0c\u8868\u793aproducer\u53d1\u9001\u51fa\u53bbmessage\uff0c\u53ea\u8981\u5bf9\u5e94\u7684kafka broker topic partition leader\u63a5\u6536\u5230\u7684\u8fd9\u6761message\uff0cproducer\u5c31\u8fd4\u56de\u6210\u529f\uff0c\u4e0d\u7ba1partition leader \u662f\u5426\u771f\u7684\u6210\u529f\u628amessage\u771f\u6b63\u5b58\u5230kafka\u3002\u5f53ack=1\u7684\u65f6\u5019\uff0c\u8868\u793aproducer\u53d1\u9001\u51fa\u53bbmessage\uff0c\u540c\u6b65\u7684\u628amessage\u5b58\u5230\u5bf9\u5e94topic\u7684partition\u7684leader\u4e0a\uff0c\u7136\u540eproducer\u5c31\u8fd4\u56de\u6210\u529f\uff0cpartition leader\u5f02\u6b65\u7684\u628amessage\u540c\u6b65\u5230\u5176\u4ed6partition replica\u4e0a\u3002\u5f53ack=all\u6216-1\uff0c\u8868\u793aproducer\u53d1\u9001\u51fa\u53bbmessage\uff0c\u540c\u6b65\u7684\u628amessage\u5b58\u5230\u5bf9\u5e94topic\u7684partition\u7684leader\u548c\u5bf9\u5e94\u7684replica\u4e0a\u4e4b\u540e\uff0c\u624d\u8fd4\u56de\u6210\u529f\u3002\u4f46\u662f\u5982\u679c\u67d0\u4e2akafka controller\u00a0\u5207\u6362\u7684\u65f6\u5019\uff0c\u4f1a\u5bfc\u81f4partition leader\u7684\u5207\u6362\uff08\u8001\u7684\u00a0kafka controller\u4e0a\u9762\u7684partition leader\u4f1a\u9009\u4e3e\u5230\u5176\u4ed6\u7684kafka broker\u4e0a\uff09,\u4f46\u662f\u8fd9\u6837\u5c31\u4f1a\u5bfc\u81f4\u4e22\u6570\u636e\u3002<\/div>\n<div>&#8211;\u00a0\u00a0Consumergroup\uff1a\u5404\u4e2aconsumer\uff08consumer \u7ebf\u7a0b\uff09\u53ef\u4ee5\u7ec4\u6210\u4e00\u4e2a\u7ec4\uff08Consumer group\u00a0\uff09\uff0cpartition\u4e2d\u7684\u6bcf\u4e2amessage\u53ea\u80fd\u88ab\u7ec4\uff08Consumer group\u00a0\uff09\u4e2d\u7684\u4e00\u4e2aconsumer\uff08consumer \u7ebf\u7a0b\uff09\u6d88\u8d39\uff0c\u5982\u679c\u4e00\u4e2amessage\u53ef\u4ee5\u88ab\u591a\u4e2aconsumer\uff08consumer \u7ebf\u7a0b\uff09\u6d88\u8d39\u7684\u8bdd\uff0c\u90a3\u4e48\u8fd9\u4e9bconsumer\u5fc5\u987b\u5728\u4e0d\u540c\u7684\u7ec4\u3002Kafka\u4e0d\u652f\u6301\u4e00\u4e2apartition\u4e2d\u7684message\u7531\u4e24\u4e2a\u6216\u4e24\u4e2a\u4ee5\u4e0a\u7684\u540c\u4e00\u4e2aconsumer group\u4e0b\u7684consumer thread\u6765\u5904\u7406\uff0c\u9664\u975e\u518d\u542f\u52a8\u4e00\u4e2a\u65b0\u7684consumer group\u3002\u6240\u4ee5\u5982\u679c\u60f3\u540c\u65f6\u5bf9\u4e00\u4e2atopic\u505a\u6d88\u8d39\u7684\u8bdd\uff0c\u542f\u52a8\u591a\u4e2aconsumer group\u5c31\u53ef\u4ee5\u4e86\uff0c\u4f46\u662f\u8981\u6ce8\u610f\u7684\u662f\uff0c\u8fd9\u91cc\u7684\u591a\u4e2aconsumer\u7684\u6d88\u8d39\u90fd\u5fc5\u987b\u662f\u987a\u5e8f\u8bfb\u53d6partition\u91cc\u9762\u7684message\uff0c\u65b0\u542f\u52a8\u7684consumer\u9ed8\u8ba4\u4ecepartition\u961f\u5217\u6700\u5934\u7aef\u6700\u65b0\u7684\u5730\u65b9\u5f00\u59cb\u963b\u585e\u7684\u8bfbmessage\u3002\u5b83\u4e0d\u80fd\u50cfAMQ\u90a3\u6837\u53ef\u4ee5\u591a\u4e2aBET\u4f5c\u4e3aconsumer\u53bb\u4e92\u65a5\u7684\uff08for update\u60b2\u89c2\u9501\uff09\u5e76\u53d1\u5904\u7406message\uff0c\u8fd9\u662f\u56e0\u4e3a\u591a\u4e2aBET\u53bb\u6d88\u8d39\u4e00\u4e2aQueue\u4e2d\u7684\u6570\u636e\u7684\u65f6\u5019\uff0c\u7531\u4e8e\u8981\u4fdd\u8bc1\u4e0d\u80fd\u591a\u4e2a\u7ebf\u7a0b\u62ff\u540c\u4e00\u6761message\uff0c\u6240\u4ee5\u5c31\u9700\u8981\u884c\u7ea7\u522b\u60b2\u89c2\u6240\uff08for update\uff09,\u8fd9\u5c31\u5bfc\u81f4\u4e86consume\u7684\u6027\u80fd\u4e0b\u964d\uff0c\u541e\u5410\u91cf\u4e0d\u591f\u3002\u800ckafka\u4e3a\u4e86\u4fdd\u8bc1\u541e\u5410\u91cf\uff0c\u53ea\u5141\u8bb8\u540c\u4e00\u4e2aconsumer group\u4e0b\u7684\u4e00\u4e2aconsumer\u7ebf\u7a0b\u53bb\u8bbf\u95ee\u4e00\u4e2apartition\u3002\u5982\u679c\u89c9\u5f97\u6548\u7387\u4e0d\u9ad8\u7684\u65f6\u5019\uff0c\u53ef\u4ee5\u52a0partition\u7684\u6570\u91cf\u6765\u6a2a\u5411\u6269\u5c55\uff0c\u90a3\u4e48\u518d\u52a0\u65b0\u7684consumer thread\u53bb\u6d88\u8d39\u3002\u5982\u679c\u60f3\u591a\u4e2a\u4e0d\u540c\u7684\u4e1a\u52a1\u90fd\u9700\u8981\u8fd9\u4e2atopic\u7684\u6570\u636e\uff0c\u8d77\u591a\u4e2aconsumer group\u5c31\u597d\u4e86\uff0c\u5927\u5bb6\u90fd\u662f\u987a\u5e8f\u7684\u8bfb\u53d6message\uff0coffsite\u7684\u503c\u4e92\u4e0d\u5f71\u54cd\u3002\u8fd9\u6837\u6ca1\u6709\u9501\u7ade\u4e89\uff0c\u5145\u5206\u53d1\u6325\u4e86\u6a2a\u5411\u7684\u6269\u5c55\u6027\uff0c\u541e\u5410\u91cf\u6781\u9ad8\u3002\u8fd9\u4e5f\u5c31\u5f62\u6210\u4e86\u5206\u5e03\u5f0f\u6d88\u8d39\u7684\u6982\u5ff5\u3002<\/div>\n<div>\u00a0 \u00a0 \u5f53\u542f\u52a8\u4e00\u4e2aconsumer group\u53bb\u6d88\u8d39\u4e00\u4e2atopic\u7684\u65f6\u5019\uff0c\u65e0\u8bbatopic\u91cc\u9762\u6709\u591a\u4e2a\u5c11\u4e2apartition\uff0c\u65e0\u8bba\u6211\u4eecconsumer group\u91cc\u9762\u914d\u7f6e\u4e86\u591a\u5c11\u4e2aconsumer thread\uff0c\u8fd9\u4e2aconsumer group\u4e0b\u9762\u7684\u6240\u6709consumer thread\u4e00\u5b9a\u4f1a\u6d88\u8d39\u5168\u90e8\u7684partition\uff1b\u5373\u4fbf\u8fd9\u4e2aconsumer group\u4e0b\u53ea\u6709\u4e00\u4e2aconsumer thread\uff0c\u90a3\u4e48\u8fd9\u4e2aconsumer thread\u4e5f\u4f1a\u53bb\u6d88\u8d39\u6240\u6709\u7684partition\u3002\u56e0\u6b64\uff0c\u6700\u4f18\u7684\u8bbe\u8ba1\u5c31\u662f\uff0cconsumer group\u4e0b\u7684consumer thread\u7684\u6570\u91cf\u7b49\u4e8epartition\u6570\u91cf\uff0c\u8fd9\u6837\u6548\u7387\u662f\u6700\u9ad8\u7684\u3002<\/div>\n<div>\u00a0 \u00a0 \u540c\u4e00partition\u7684\u4e00\u6761message\u53ea\u80fd\u88ab\u540c\u4e00\u4e2aConsumer Group\u5185\u7684\u4e00\u4e2aConsumer\u6d88\u8d39\u3002\u4e0d\u80fd\u591f\u4e00\u4e2aconsumer group\u7684\u591a\u4e2aconsumer\u540c\u65f6\u6d88\u8d39\u4e00\u4e2apartition\u3002<\/div>\n<div>\u00a0 \u00a0 \u4e00\u4e2aconsumer group\u4e0b\uff0c\u65e0\u8bba\u6709\u591a\u5c11\u4e2aconsumer\uff0c\u8fd9\u4e2aconsumer group\u4e00\u5b9a\u56de\u53bb\u628a\u8fd9\u4e2atopic\u4e0b\u6240\u6709\u7684partition\u90fd\u6d88\u8d39\u4e86\u3002\u5f53consumer group\u91cc\u9762\u7684consumer\u6570\u91cf\u5c0f\u4e8e\u8fd9\u4e2atopic\u4e0b\u7684partition\u6570\u91cf\u7684\u65f6\u5019\uff0c\u5982\u4e0b\u56fegroupA,groupB\uff0c\u5c31\u4f1a\u51fa\u73b0\u4e00\u4e2aconusmer thread\u6d88\u8d39\u591a\u4e2apartition\u7684\u60c5\u51b5\uff0c\u603b\u4e4b\u662f\u8fd9\u4e2atopic\u4e0b\u7684partition\u90fd\u4f1a\u88ab\u6d88\u8d39\u3002\u5982\u679cconsumer group\u91cc\u9762\u7684consumer\u6570\u91cf\u7b49\u4e8e\u8fd9\u4e2atopic\u4e0b\u7684partition\u6570\u91cf\u7684\u65f6\u5019\uff0c\u5982\u4e0b\u56fegroupC\uff0c\u6b64\u65f6\u6548\u7387\u662f\u6700\u9ad8\u7684\uff0c\u6bcf\u4e2apartition\u90fd\u6709\u4e00\u4e2aconsumer thread\u53bb\u6d88\u8d39\u3002\u5f53consumer group\u91cc\u9762\u7684consumer\u6570\u91cf\u5927\u4e8e\u8fd9\u4e2atopic\u4e0b\u7684partition\u6570\u91cf\u7684\u65f6\u5019\uff0c\u5982\u4e0b\u56feGroupD\uff0c\u5c31\u4f1a\u6709\u4e00\u4e2aconsumer thread\u7a7a\u95f2\u3002\u56e0\u6b64\uff0c\u6211\u4eec\u5728\u8bbe\u5b9aconsumer group\u7684\u65f6\u5019\uff0c\u53ea\u9700\u8981\u6307\u660e\u91cc\u9762\u6709\u51e0\u4e2aconsumer\u6570\u91cf\u5373\u53ef\uff0c\u65e0\u9700\u6307\u5b9a\u5bf9\u5e94\u7684\u6d88\u8d39partition\u5e8f\u53f7\uff0cconsumer\u4f1a\u81ea\u52a8\u8fdb\u884crebalance\u3002<\/div>\n<div>\u00a0 \u00a0 \u591a\u4e2aConsumer Group\u4e0b\u7684consumer\u53ef\u4ee5\u6d88\u8d39\u540c\u4e00\u6761message\uff0c\u4f46\u662f\u8fd9\u79cd\u6d88\u8d39\u4e5f\u662f\u4ee5o\uff081\uff09\u7684\u65b9\u5f0f\u987a\u5e8f\u7684\u8bfb\u53d6message\u53bb\u6d88\u8d39,\uff0c\u6240\u4ee5\u4e00\u5b9a\u4f1a\u91cd\u590d\u6d88\u8d39\u8fd9\u6279message\u7684\uff0c\u4e0d\u80fd\u5411AMQ\u90a3\u6837\u591a\u4e2aBET\u4f5c\u4e3aconsumer\u6d88\u8d39\uff08\u5bf9message\u52a0\u9501\uff0c\u6d88\u8d39\u7684\u65f6\u5019\u4e0d\u80fd\u91cd\u590d\u6d88\u8d39message\uff09<\/div>\n<div>&#8211;\u00a0Consumer Rebalance\u7684\u89e6\u53d1\u6761\u4ef6\uff1a\uff081\uff09Consumer\u589e\u52a0\u6216\u5220\u9664\u4f1a\u89e6\u53d1 Consumer Group\u7684Rebalance\uff082\uff09Broker\u7684\u589e\u52a0\u6216\u8005\u51cf\u5c11\u90fd\u4f1a\u89e6\u53d1 Consumer Rebalance<\/div>\n<div>&#8211;\u00a0Consumer\uff1a\u00a0Consumer\u5904\u7406partition\u91cc\u9762\u7684message\u7684\u65f6\u5019\u662fo\uff081\uff09\u987a\u5e8f\u8bfb\u53d6\u7684\u3002\u6240\u4ee5\u5fc5\u987b\u7ef4\u62a4\u7740\u4e0a\u4e00\u6b21\u8bfb\u5230\u54ea\u91cc\u7684offsite\u4fe1\u606f\u3002high level API,offset\u5b58\u4e8eZookeeper\u4e2d\uff0clow level API\u7684offset\u7531\u81ea\u5df1\u7ef4\u62a4\u3002\u4e00\u822c\u6765\u8bf4\u90fd\u662f\u4f7f\u7528high level api\u7684\u3002Consumer\u7684delivery gurarantee\uff0c\u9ed8\u8ba4\u662f\u8bfb\u5b8cmessage\u5148commmit\u518d\u5904\u7406message\uff0cautocommit\u9ed8\u8ba4\u662ftrue\uff0c\u8fd9\u65f6\u5019\u5148commit\u5c31\u4f1a\u66f4\u65b0offsite+1\uff0c\u4e00\u65e6\u5904\u7406\u5931\u8d25\uff0coffsite\u5df2\u7ecf+1\uff0c\u8fd9\u4e2a\u65f6\u5019\u5c31\u4f1a\u4e22message\uff1b\u4e5f\u53ef\u4ee5\u914d\u7f6e\u6210\u8bfb\u5b8c\u6d88\u606f\u5904\u7406\u518dcommit\uff0c\u8fd9\u79cd\u60c5\u51b5\u4e0bconsumer\u7aef\u7684\u54cd\u5e94\u5c31\u4f1a\u6bd4\u8f83\u6162\u7684\uff0c\u9700\u8981\u7b49\u5904\u7406\u5b8c\u624d\u884c\u3002<\/div>\n<div>\u4e00\u822c\u60c5\u51b5\u4e0b\uff0c\u4e00\u5b9a\u662f\u4e00\u4e2aconsumer group\u5904\u7406\u4e00\u4e2atopic\u7684message\u3002Best Practice\u662f\u8fd9\u4e2aconsumer group\u91cc\u9762consumer\u7684\u6570\u91cf\u7b49\u4e8etopic\u91cc\u9762partition\u7684\u6570\u91cf\uff0c\u8fd9\u6837\u6548\u7387\u662f\u6700\u9ad8\u7684\uff0c\u4e00\u4e2aconsumer thread\u5904\u7406\u4e00\u4e2apartition\u3002\u5982\u679c\u8fd9\u4e2aconsumer group\u91cc\u9762consumer\u7684\u6570\u91cf\u5c0f\u4e8etopic\u91cc\u9762partition\u7684\u6570\u91cf\uff0c\u5c31\u4f1a\u6709consumer thread\u540c\u65f6\u5904\u7406\u591a\u4e2apartition\uff08\u8fd9\u4e2a\u662fkafka\u81ea\u52a8\u7684\u673a\u5236\uff0c\u6211\u4eec\u4e0d\u7528\u6307\u5b9a\uff09\uff0c\u4f46\u662f\u603b\u4e4b\u8fd9\u4e2atopic\u91cc\u9762\u7684\u6240\u6709partition\u90fd\u4f1a\u88ab\u5904\u7406\u5230\u7684\u3002\u3002\u5982\u679c\u8fd9\u4e2aconsumer group\u91cc\u9762consumer\u7684\u6570\u91cf\u5927\u4e8etopic\u91cc\u9762partition\u7684\u6570\u91cf\uff0c\u591a\u51fa\u7684consumer thread\u5c31\u4f1a\u95f2\u7740\u5565\u4e5f\u4e0d\u5e72\uff0c\u5269\u4e0b\u7684\u662f\u4e00\u4e2aconsumer thread\u5904\u7406\u4e00\u4e2apartition\uff0c\u8fd9\u5c31\u9020\u6210\u4e86\u8d44\u6e90\u7684\u6d6a\u8d39\uff0c\u56e0\u4e3a\u4e00\u4e2apartition\u4e0d\u53ef\u80fd\u88ab\u4e24\u4e2aconsumer thread\u53bb\u5904\u7406\u3002\u6240\u4ee5\u6211\u4eec\u7ebf\u4e0a\u7684\u5206\u5e03\u5f0f\u591a\u4e2aservice\u670d\u52a1\uff0c\u6bcf\u4e2aservice\u91cc\u9762\u7684kafka consumer\u6570\u91cf\u90fd\u5c0f\u4e8e\u5bf9\u5e94\u7684topic\u7684partition\u6570\u91cf\uff0c\u4f46\u662f\u6240\u6709\u670d\u52a1\u7684consumer\u6570\u91cf\u53ea\u548c\u7b49\u4e8epartition\u7684\u6570\u91cf\uff0c\u8fd9\u662f\u56e0\u4e3a\u5206\u5e03\u5f0fservice\u670d\u52a1\u7684\u6240\u6709consumer\u90fd\u6765\u81ea\u4e00\u4e2aconsumer group\uff0c\u5982\u679c\u6765\u81ea\u4e0d\u540c\u7684consumer group\u5c31\u4f1a\u5904\u7406\u91cd\u590d\u7684message\u4e86\uff08\u540c\u4e00\u4e2aconsumer group\u4e0b\u7684consumer\u4e0d\u80fd\u5904\u7406\u540c\u4e00\u4e2apartition\uff0c\u4e0d\u540c\u7684consumer group\u53ef\u4ee5\u5904\u7406\u540c\u4e00\u4e2atopic\uff0c\u90a3\u4e48\u90fd\u662f\u987a\u5e8f\u5904\u7406message\uff0c\u4e00\u5b9a\u4f1a\u5904\u7406\u91cd\u590d\u7684\u3002\u4e00\u822c\u8fd9\u79cd\u60c5\u51b5\u90fd\u662f\u4e24\u4e2a\u4e0d\u540c\u7684\u4e1a\u52a1\u903b\u8f91\uff0c\u624d\u4f1a\u542f\u52a8\u4e24\u4e2aconsumer group\u6765\u5904\u7406\u4e00\u4e2atopic\uff09\u3002<\/div>\n<div><\/div>\n<div>\u5982\u679cproducer\u7684\u6d41\u91cf\u589e\u5927\uff0c\u5f53\u524d\u7684topic\u7684parition\u6570\u91cf=consumer\u6570\u91cf\uff0c\u8fd9\u65f6\u5019\u7684\u5e94\u5bf9\u65b9\u5f0f\u5c31\u662f\u5f88\u60f3\u6269\u5c55\uff1a\u589e\u52a0topic\u4e0b\u7684partition\uff0c\u540c\u65f6\u589e\u52a0\u8fd9\u4e2aconsumer group\u4e0b\u7684consumer\u3002<\/div>\n<div>\u00a0\u00a0<img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107211358384?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<div>&#8211;\u00a0Delivery Mode :\u00a0Kafka producer \u53d1\u9001message\u4e0d\u7528\u7ef4\u62a4message\u7684offsite\u4fe1\u606f\uff0c\u56e0\u4e3a\u8fd9\u4e2a\u65f6\u5019\uff0coffsite\u5c31\u76f8\u5f53\u4e8e\u4e00\u4e2a\u81ea\u589eid\uff0cproducer\u5c31\u5c3d\u7ba1\u53d1\u9001message\u5c31\u597d\u4e86\u3002\u800c\u4e14Kafka\u4e0eAMQ\u4e0d\u540c\uff0cAMQ\u5927\u90fd\u7528\u5728\u5904\u7406\u4e1a\u52a1\u903b\u8f91\u4e0a\uff0c\u800cKafka\u5927\u90fd\u662f\u65e5\u5fd7\uff0c\u6240\u4ee5Kafka\u7684producer\u4e00\u822c\u90fd\u662f\u5927\u6279\u91cf\u7684batch\u53d1\u9001message\uff0c\u5411\u8fd9\u4e2atopic\u4e00\u6b21\u6027\u53d1\u9001\u4e00\u5927\u6279message\uff0cload balance\u5230\u4e00\u4e2apartition\u4e0a\uff0c\u4e00\u8d77\u63d2\u8fdb\u53bb\uff0coffsite\u4f5c\u4e3a\u81ea\u589eid\u81ea\u5df1\u589e\u52a0\u5c31\u597d\u3002\u4f46\u662fConsumer\u7aef\u662f\u9700\u8981\u7ef4\u62a4\u8fd9\u4e2apartition\u5f53\u524d\u6d88\u8d39\u5230\u54ea\u4e2amessage\u7684offsite\u4fe1\u606f\u7684\uff0c\u8fd9\u4e2aoffsite\u4fe1\u606f\uff0chigh level api\u662f\u7ef4\u62a4\u5728Zookeeper\u4e0a\uff0clow level api\u662f\u81ea\u5df1\u7684\u7a0b\u5e8f\u7ef4\u62a4\u3002\uff08Kafka\u7ba1\u7406\u754c\u9762\u4e0a\u53ea\u80fd\u663e\u793ahigh level api\u7684consumer\u90e8\u5206\uff0c\u56e0\u4e3alow level api\u7684partition offsite\u4fe1\u606f\u662f\u7a0b\u5e8f\u81ea\u5df1\u7ef4\u62a4\uff0ckafka\u662f\u4e0d\u77e5\u9053\u7684\uff0c\u65e0\u6cd5\u5728\u7ba1\u7406\u754c\u9762\u4e0a\u5c55\u793a \uff09\u5f53\u4f7f\u7528high level api\u7684\u65f6\u5019\uff0c\u5148\u62ffmessage\u5904\u7406\uff0c\u518d\u5b9a\u65f6\u81ea\u52a8commit offsite+1\uff08\u4e5f\u53ef\u4ee5\u6539\u6210\u624b\u52a8\uff09, \u5e76\u4e14kakfa\u5904\u7406message\u662f\u6ca1\u6709\u9501\u64cd\u4f5c\u7684\u3002\u56e0\u6b64\u5982\u679c\u5904\u7406message\u5931\u8d25\uff0c\u6b64\u65f6\u8fd8\u6ca1\u6709commit offsite+1\uff0c\u5f53consumer thread\u91cd\u542f\u540e\u4f1a\u91cd\u590d\u6d88\u8d39\u8fd9\u4e2amessage\u3002\u4f46\u662f\u4f5c\u4e3a\u9ad8\u541e\u5410\u91cf\u9ad8\u5e76\u53d1\u7684\u5b9e\u65f6\u5904\u7406\u7cfb\u7edf\uff0cat least once\u7684\u60c5\u51b5\u4e0b\uff0c\u81f3\u5c11\u4e00\u6b21\u4f1a\u88ab\u5904\u7406\u5230\uff0c\u662f\u53ef\u4ee5\u5bb9\u5fcd\u7684\u3002\u5982\u679c\u65e0\u6cd5\u5bb9\u5fcd\uff0c\u5c31\u5f97\u4f7f\u7528low level api\u6765\u81ea\u5df1\u7a0b\u5e8f\u7ef4\u62a4\u8fd9\u4e2aoffsite\u4fe1\u606f\uff0c\u90a3\u4e48\u60f3\u4ec0\u4e48\u65f6\u5019commit offsite+1\u5c31\u81ea\u5df1\u641e\u5b9a\u4e86\u3002<\/div>\n<div><\/div>\n<div>&#8211;\u00a0Topic &amp; Partition\uff1aTopic\u76f8\u5f53\u4e8e\u4f20\u7edf\u6d88\u606f\u7cfb\u7edfMQ\u4e2d\u7684\u4e00\u4e2a\u961f\u5217queue\uff0cproducer\u7aef\u53d1\u9001\u7684message\u5fc5\u987b\u6307\u5b9a\u662f\u53d1\u9001\u5230\u54ea\u4e2atopic\uff0c\u4f46\u662f\u4e0d\u9700\u8981\u6307\u5b9atopic\u4e0b\u7684\u54ea\u4e2apartition\uff0c\u56e0\u4e3akafka\u4f1a\u628a\u6536\u5230\u7684message\u8fdb\u884cload balance\uff0c\u5747\u5300\u7684\u5206\u5e03\u5728\u8fd9\u4e2atopic\u4e0b\u7684\u4e0d\u540c\u7684partition\u4e0a\uff08 hash(message) % [broker\u6570\u91cf] \u00a0\uff09\u3002\u7269\u7406\u4e0a\u5b58\u50a8\u4e0a\uff0c\u8fd9\u4e2atopic\u4f1a\u5206\u6210\u4e00\u4e2a\u6216\u591a\u4e2apartition\uff0c\u6bcf\u4e2apartiton\u76f8\u5f53\u4e8e\u662f\u4e00\u4e2a\u5b50queue\u3002\u5728\u7269\u7406\u7ed3\u6784\u4e0a\uff0c\u6bcf\u4e2apartition\u5bf9\u5e94\u4e00\u4e2a\u7269\u7406\u7684\u76ee\u5f55\uff08\u6587\u4ef6\u5939\uff09\uff0c\u6587\u4ef6\u5939\u547d\u540d\u662f[topicname]_[partition]_[\u5e8f\u53f7]\uff0c\u4e00\u4e2atopic\u53ef\u4ee5\u6709\u65e0\u6570\u591a\u7684partition\uff0c\u6839\u636e\u4e1a\u52a1\u9700\u6c42\u548c\u6570\u636e\u91cf\u6765\u8bbe\u7f6e\u3002\u5728kafka\u914d\u7f6e\u6587\u4ef6\u4e2d\u53ef\u968f\u65f6\u66f4\u9ad8num.partitions\u53c2\u6570\u6765\u914d\u7f6e\u66f4\u6539topic\u7684partition\u6570\u91cf\uff0c\u5728\u521b\u5efaTopic\u65f6\u901a\u8fc7\u53c2\u6570\u6307\u5b9aparittion\u6570\u91cf\u3002Topic\u521b\u5efa\u4e4b\u540e\u901a\u8fc7Kafka\u63d0\u4f9b\u7684\u5de5\u5177\u4e5f\u53ef\u4ee5\u4fee\u6539partiton\u6570\u91cf\u3002<\/div>\n<div>\u00a0 \u00a0\u4e00\u822c\u6765\u8bf4\uff0c\uff081\uff09\u4e00\u4e2aTopic\u7684Partition\u6570\u91cf\u5927\u4e8e\u7b49\u4e8eBroker\u7684\u6570\u91cf\uff0c\u53ef\u4ee5\u63d0\u9ad8\u541e\u5410\u7387\u3002\uff082\uff09\u540c\u4e00\u4e2aPartition\u7684Replica\u5c3d\u91cf\u5206\u6563\u5230\u4e0d\u540c\u7684\u673a\u5668\uff0c\u9ad8\u53ef\u7528\u3002<\/div>\n<div>\u00a0\u00a0\u5f53add a new partition\u7684\u65f6\u5019\uff0cpartition\u91cc\u9762\u7684message\u4e0d\u4f1a\u91cd\u65b0\u8fdb\u884c\u5206\u914d\uff0c\u539f\u6765\u7684partition\u91cc\u9762\u7684message\u6570\u636e\u4e0d\u4f1a\u53d8\uff0c\u65b0\u52a0\u7684\u8fd9\u4e2apartition\u521a\u5f00\u59cb\u662f\u7a7a\u7684\uff0c\u968f\u540e\u8fdb\u5165\u8fd9\u4e2atopic\u7684message\u5c31\u4f1a\u91cd\u65b0\u53c2\u4e0e\u6240\u6709partition\u7684load balance<\/div>\n<div>&#8211;\u00a0Partition Replica\uff1a\u6bcf\u4e2apartition\u53ef\u4ee5\u5728\u5176\u4ed6\u7684kafka broker\u8282\u70b9\u4e0a\u5b58\u526f\u672c\uff0c\u4ee5\u4fbf\u67d0\u4e2akafka broker\u8282\u70b9\u5b95\u673a\u4e0d\u4f1a\u5f71\u54cd\u8fd9\u4e2akafka\u96c6\u7fa4\u3002\u5b58replica\u526f\u672c\u7684\u65b9\u5f0f\u662f\u6309\u7167kafka broker\u7684\u987a\u5e8f\u5b58\u3002\u4f8b\u5982\u67095\u4e2akafka broker\u8282\u70b9\uff0c\u67d0\u4e2atopic\u67093\u4e2apartition\uff0c\u6bcf\u4e2apartition\u5b582\u4e2a\u526f\u672c\uff0c\u90a3\u4e48partition1\u5b58broker1,broker2\uff0cpartition2\u5b58broker2,broker3\u3002\u3002\u3002\u4ee5\u6b64\u7c7b\u63a8\uff08replica\u526f\u672c\u6570\u76ee\u4e0d\u80fd\u5927\u4e8ekafka broker\u8282\u70b9\u7684\u6570\u76ee\uff0c\u5426\u5219\u62a5\u9519\u3002\u8fd9\u91cc\u7684replica\u6570\u5176\u5b9e\u5c31\u662fpartition\u7684\u526f\u672c\u603b\u6570\uff0c\u5176\u4e2d\u5305\u62ec\u4e00\u4e2aleader\uff0c\u5176\u4ed6\u7684\u5c31\u662fcopy\u526f\u672c\uff09\u3002\u8fd9\u6837\u5982\u679c\u67d0\u4e2abroker\u5b95\u673a\uff0c\u5176\u5b9e\u6574\u4e2akafka\u5185\u6570\u636e\u4f9d\u7136\u662f\u5b8c\u6574\u7684\u3002\u4f46\u662f\uff0creplica\u526f\u672c\u6570\u8d8a\u9ad8\uff0c\u7cfb\u7edf\u867d\u7136\u8d8a\u7a33\u5b9a\uff0c\u4f46\u662f\u56de\u6765\u5e26\u8d44\u6e90\u548c\u6027\u80fd\u4e0a\u7684\u4e0b\u964d\uff1breplica\u526f\u672c\u5c11\u7684\u8bdd\uff0c\u4e5f\u4f1a\u9020\u6210\u7cfb\u7edf\u4e22\u6570\u636e\u7684\u98ce\u9669\u3002<\/div>\n<div>\u00a0 \uff081\uff09\u600e\u6837\u4f20\u9001\u6d88\u606f\uff1aproducer\u5148\u628amessage\u53d1\u9001\u5230partition leader\uff0c\u518d\u7531leader\u53d1\u9001\u7ed9\u5176\u4ed6partition follower\u3002\uff08\u5982\u679c\u8ba9producer\u53d1\u9001\u7ed9\u6bcf\u4e2areplica\u90a3\u5c31\u592a\u6162\u4e86\uff09<\/div>\n<div>\u00a0 \uff082\uff09\u5728\u5411Producer\u53d1\u9001ACK\u524d\u9700\u8981\u4fdd\u8bc1\u6709\u591a\u5c11\u4e2aReplica\u5df2\u7ecf\u6536\u5230\u8be5\u6d88\u606f\uff1a\u6839\u636eack\u914d\u7684\u4e2a\u6570\u800c\u5b9a<\/div>\n<div>\u00a0 \uff083\uff09\u600e\u6837\u5904\u7406\u67d0\u4e2aReplica\u4e0d\u5de5\u4f5c\u7684\u60c5\u51b5\uff1a\u5982\u679c\u8fd9\u4e2a\u90e8\u5de5\u4f5c\u7684partition replica\u4e0d\u5728ack\u5217\u8868\u4e2d\uff0c\u5c31\u662fproducer\u5728\u53d1\u9001\u6d88\u606f\u5230partition leader\u4e0a\uff0cpartition leader\u5411partition follower\u53d1\u9001message\u6ca1\u6709\u54cd\u5e94\u800c\u5df2\uff0c\u8fd9\u4e2a\u4e0d\u4f1a\u5f71\u54cd\u6574\u4e2a\u7cfb\u7edf\uff0c\u4e5f\u4e0d\u4f1a\u6709\u4ec0\u4e48\u95ee\u9898\u3002\u5982\u679c\u8fd9\u4e2a\u4e0d\u5de5\u4f5c\u7684partition replica\u5728ack\u5217\u8868\u4e2d\u7684\u8bdd\uff0cproducer\u53d1\u9001\u7684message\u7684\u65f6\u5019\u4f1a\u7b49\u5f85\u8fd9\u4e2a\u4e0d\u5de5\u4f5c\u7684partition replca\u5199message\u6210\u529f\uff0c\u4f46\u662f\u4f1a\u7b49\u5230time out\uff0c\u7136\u540e\u8fd4\u56de\u5931\u8d25\u56e0\u4e3a\u67d0\u4e2aack\u5217\u8868\u4e2d\u7684partition replica\u6ca1\u6709\u54cd\u5e94\uff0c\u6b64\u65f6kafka\u4f1a\u81ea\u52a8\u7684\u628a\u8fd9\u4e2a\u90e8\u5de5\u4f5c\u7684partition replica\u4eceack\u5217\u8868\u4e2d\u79fb\u9664\uff0c\u4ee5\u540e\u7684producer\u53d1\u9001message\u7684\u65f6\u5019\u5c31\u4e0d\u4f1a\u6709\u8fd9\u4e2aack\u5217\u8868\u4e0b\u7684\u8fd9\u4e2a\u90e8\u5de5\u4f5c\u7684partition replica\u4e86\u3002<\/div>\n<div>\u00a0 \uff084\uff09\u600e\u6837\u5904\u7406Failed Replica\u6062\u590d\u56de\u6765\u7684\u60c5\u51b5\uff1a\u5982\u679c\u8fd9\u4e2apartition replica\u4e4b\u524d\u4e0d\u5728ack\u5217\u8868\u4e2d\uff0c\u90a3\u4e48\u542f\u52a8\u540e\u91cd\u65b0\u53d7Zookeeper\u7ba1\u7406\u5373\u53ef\uff0c\u4e4b\u540eproducer\u53d1\u9001message\u7684\u65f6\u5019\uff0cpartition leader\u4f1a\u7ee7\u7eed\u53d1\u9001message\u5230\u8fd9\u4e2apartition follower\u4e0a\u3002\u5982\u679c\u8fd9\u4e2apartition replica\u4e4b\u524d\u5728ack\u5217\u8868\u4e2d\uff0c\u6b64\u65f6\u91cd\u542f\u540e\uff0c\u9700\u8981\u628a\u8fd9\u4e2apartition replica\u518d\u624b\u52a8\u52a0\u5230ack\u5217\u8868\u4e2d\u3002\uff08ack\u5217\u8868\u662f\u624b\u52a8\u6dfb\u52a0\u7684\uff0c\u51fa\u73b0\u67d0\u4e2a\u90e8\u5de5\u4f5c\u7684partition replica\u7684\u65f6\u5019\u81ea\u52a8\u4eceack\u5217\u8868\u4e2d\u79fb\u9664\u7684\uff09<\/div>\n<div>&#8211;\u00a0Partition leader\u4e0efollower\uff1apartition\u4e5f\u6709leader\u548cfollower\u4e4b\u5206\u3002leader\u662f\u4e3bpartition\uff0cproducer\u5199kafka\u7684\u65f6\u5019\u5148\u5199partition leader\uff0c\u518d\u7531partition leader push\u7ed9\u5176\u4ed6\u7684partition follower\u3002partition leader\u4e0efollower\u7684\u4fe1\u606f\u53d7Zookeeper\u63a7\u5236\uff0c\u4e00\u65e6partition leader\u6240\u5728\u7684broker\u8282\u70b9\u5b95\u673a\uff0czookeeper\u4f1a\u51b2\u5176\u4ed6\u7684broker\u7684partition follower\u4e0a\u9009\u62e9follower\u53d8\u4e3aparition leader\u3002<\/div>\n<div>&#8211;\u00a0Topic\u5206\u914dpartition\u548cpartition replica\u7684\u7b97\u6cd5\uff1a\uff081\uff09\u5c06Broker\uff08size=n\uff09\u548c\u5f85\u5206\u914d\u7684Partition\u6392\u5e8f\u3002\uff082\uff09\u5c06\u7b2ci\u4e2aPartition\u5206\u914d\u5230\u7b2c\uff08i%n\uff09\u4e2aBroker\u4e0a\u3002\uff083\uff09\u5c06\u7b2ci\u4e2aPartition\u7684\u7b2cj\u4e2aReplica\u5206\u914d\u5230\u7b2c\uff08(i + j) % n\uff09\u4e2aBroker\u4e0a<\/div>\n<div><\/div>\n<div>&#8211;\u00a0\u6d88\u606f\u6295\u9012\u53ef\u9760\u6027<\/div>\n<div>\u4e00\u4e2a\u6d88\u606f\u5982\u4f55\u7b97\u6295\u9012\u6210\u529f\uff0cKafka\u63d0\u4f9b\u4e86\u4e09\u79cd\u6a21\u5f0f\uff1a<\/div>\n<div>&#8211; \u7b2c\u4e00\u79cd\u662f\u5565\u90fd\u4e0d\u7ba1\uff0c\u53d1\u9001\u51fa\u53bb\u5c31\u5f53\u4f5c\u6210\u529f\uff0c\u8fd9\u79cd\u60c5\u51b5\u5f53\u7136\u4e0d\u80fd\u4fdd\u8bc1\u6d88\u606f\u6210\u529f\u6295\u9012\u5230broker\uff1b<\/div>\n<div>&#8211; \u7b2c\u4e8c\u79cd\u662fMaster-Slave\u6a21\u578b\uff0c\u53ea\u6709\u5f53Master\u548c\u6240\u6709Slave\u90fd\u63a5\u6536\u5230\u6d88\u606f\u65f6\uff0c\u624d\u7b97\u6295\u9012\u6210\u529f\uff0c\u8fd9\u79cd\u6a21\u578b\u63d0\u4f9b\u4e86\u6700\u9ad8\u7684\u6295\u9012\u53ef\u9760\u6027\uff0c\u4f46\u662f\u635f\u4f24\u4e86\u6027\u80fd\uff1b<\/div>\n<div>&#8211; \u7b2c\u4e09\u79cd\u6a21\u578b\uff0c\u5373\u53ea\u8981Master\u786e\u8ba4\u6536\u5230\u6d88\u606f\u5c31\u7b97\u6295\u9012\u6210\u529f\uff1b\u5b9e\u9645\u4f7f\u7528\u65f6\uff0c\u6839\u636e\u5e94\u7528\u7279\u6027\u9009\u62e9\uff0c\u7edd\u5927\u591a\u6570\u60c5\u51b5\u4e0b\u90fd\u4f1a\u4e2d\u548c\u53ef\u9760\u6027\u548c\u6027\u80fd\u9009\u62e9\u7b2c\u4e09\u79cd\u6a21\u578b<\/div>\n<div>\u00a0 \u6d88\u606f\u5728broker\u4e0a\u7684\u53ef\u9760\u6027\uff0c\u56e0\u4e3a\u6d88\u606f\u4f1a\u6301\u4e45\u5316\u5230\u78c1\u76d8\u4e0a\uff0c\u6240\u4ee5\u5982\u679c\u6b63\u5e38stop\u4e00\u4e2abroker\uff0c\u5176\u4e0a\u7684\u6570\u636e\u4e0d\u4f1a\u4e22\u5931\uff1b\u4f46\u662f\u5982\u679c\u4e0d\u6b63\u5e38stop\uff0c\u53ef\u80fd\u4f1a\u4f7f\u5b58\u5728\u9875\u9762\u7f13\u5b58\u6765\u4e0d\u53ca\u5199\u5165\u78c1\u76d8\u7684\u6d88\u606f\u4e22\u5931\uff0c\u8fd9\u53ef\u4ee5\u901a\u8fc7\u914d\u7f6eflush\u9875\u9762\u7f13\u5b58\u7684\u5468\u671f\u3001\u9608\u503c\u7f13\u89e3\uff0c\u4f46\u662f\u540c\u6837\u4f1a\u9891\u7e41\u7684\u5199\u78c1\u76d8\u4f1a\u5f71\u54cd\u6027\u80fd\uff0c\u53c8\u662f\u4e00\u4e2a\u9009\u62e9\u9898\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e\u3002<\/div>\n<div>\u00a0 \u6d88\u606f\u6d88\u8d39\u7684\u53ef\u9760\u6027\uff0cKafka\u63d0\u4f9b\u7684\u662f\u201cAt least once\u201d\u6a21\u578b\uff0c\u56e0\u4e3a\u6d88\u606f\u7684\u8bfb\u53d6\u8fdb\u5ea6\u7531offset\u63d0\u4f9b\uff0coffset\u53ef\u4ee5\u7531\u6d88\u8d39\u8005\u81ea\u5df1\u7ef4\u62a4\u4e5f\u53ef\u4ee5\u7ef4\u62a4\u5728zookeeper\u91cc\uff0c\u4f46\u662f\u5f53\u6d88\u606f\u6d88\u8d39\u540econsumer\u6302\u6389\uff0coffset\u6ca1\u6709\u5373\u65f6\u5199\u56de\uff0c\u5c31\u6709\u53ef\u80fd\u53d1\u751f\u91cd\u590d\u8bfb\u7684\u60c5\u51b5\uff0c\u8fd9\u79cd\u60c5\u51b5\u540c\u6837\u53ef\u4ee5\u901a\u8fc7\u8c03\u6574commit offset\u5468\u671f\u3001\u9608\u503c\u7f13\u89e3\uff0c\u751a\u81f3\u6d88\u8d39\u8005\u81ea\u5df1\u628a\u6d88\u8d39\u548ccommit offset\u505a\u6210\u4e00\u4e2a\u4e8b\u52a1\u89e3\u51b3\uff0c\u4f46\u662f\u5982\u679c\u4f60\u7684\u5e94\u7528\u4e0d\u5728\u4e4e\u91cd\u590d\u6d88\u8d39\uff0c\u90a3\u5c31\u5e72\u8106\u4e0d\u8981\u89e3\u51b3\uff0c\u4ee5\u6362\u53d6\u6700\u5927\u7684\u6027\u80fd\u3002<\/div>\n<div><\/div>\n<div>&#8211;\u00a0Partition ack\uff1a\u5f53ack=1\uff0c\u8868\u793aproducer\u5199partition leader\u6210\u529f\u540e\uff0cbroker\u5c31\u8fd4\u56de\u6210\u529f\uff0c\u65e0\u8bba\u5176\u4ed6\u7684partition follower\u662f\u5426\u5199\u6210\u529f\u3002\u5f53ack=2\uff0c\u8868\u793aproducer\u5199partition leader\u548c\u5176\u4ed6\u4e00\u4e2afollower\u6210\u529f\u7684\u65f6\u5019\uff0cbroker\u5c31\u8fd4\u56de\u6210\u529f\uff0c\u65e0\u8bba\u5176\u4ed6\u7684partition follower\u662f\u5426\u5199\u6210\u529f\u3002\u5f53ack=-1[parition\u7684\u6570\u91cf]\u7684\u65f6\u5019\uff0c\u8868\u793a\u53ea\u6709producer\u5168\u90e8\u5199\u6210\u529f\u7684\u65f6\u5019\uff0c\u624d\u7b97\u6210\u529f\uff0ckafka broker\u624d\u8fd4\u56de\u6210\u529f\u4fe1\u606f\u3002\u8fd9\u91cc\u9700\u8981\u6ce8\u610f\u7684\u662f\uff0c\u5982\u679cack=1\u7684\u65f6\u5019\uff0c\u4e00\u65e6\u6709\u4e2abroker\u5b95\u673a\u5bfc\u81f4partition\u7684follower\u548cleader\u5207\u6362\uff0c\u4f1a\u5bfc\u81f4\u4e22\u6570\u636e\u3002<\/div>\n<div>\u00a0\u00a0<img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107211558246?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<div><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107211616996?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<div>&#8211;\u00a0message\u72b6\u6001\uff1a\u5728Kafka\u4e2d\uff0c\u6d88\u606f\u7684\u72b6\u6001\u88ab\u4fdd\u5b58\u5728consumer\u4e2d\uff0cbroker\u4e0d\u4f1a\u5173\u5fc3\u54ea\u4e2a\u6d88\u606f\u88ab\u6d88\u8d39\u4e86\u88ab\u8c01\u6d88\u8d39\u4e86\uff0c\u53ea\u8bb0\u5f55\u4e00\u4e2aoffset\u503c\uff08\u6307\u5411partition\u4e2d\u4e0b\u4e00\u4e2a\u8981\u88ab\u6d88\u8d39\u7684\u6d88\u606f\u4f4d\u7f6e\uff09\uff0c\u8fd9\u5c31\u610f\u5473\u7740\u5982\u679cconsumer\u5904\u7406\u4e0d\u597d\u7684\u8bdd\uff0cbroker\u4e0a\u7684\u4e00\u4e2a\u6d88\u606f\u53ef\u80fd\u4f1a\u88ab\u6d88\u8d39\u591a\u6b21\u3002<\/div>\n<div>&#8211;\u00a0message\u6301\u4e45\u5316\uff1aKafka\u4e2d\u4f1a\u628a\u6d88\u606f\u6301\u4e45\u5316\u5230\u672c\u5730\u6587\u4ef6\u7cfb\u7edf\u4e2d\uff0c\u5e76\u4e14\u4fdd\u6301o(1)\u6781\u9ad8\u7684\u6548\u7387\u3002\u6211\u4eec\u4f17\u6240\u5468\u77e5IO\u8bfb\u53d6\u662f\u975e\u5e38\u8017\u8d44\u6e90\u7684\u6027\u80fd\u4e5f\u662f\u6700\u6162\u7684\uff0c\u8fd9\u5c31\u662f\u4e3a\u4e86\u6570\u636e\u5e93\u7684\u74f6\u9888\u7ecf\u5e38\u5728IO\u4e0a\uff0c\u9700\u8981\u6362SSD\u786c\u76d8\u7684\u539f\u56e0\u3002\u4f46\u662fKafka\u4f5c\u4e3a\u541e\u5410\u91cf\u6781\u9ad8\u7684MQ\uff0c\u5374\u53ef\u4ee5\u975e\u5e38\u9ad8\u6548\u7684message\u6301\u4e45\u5316\u5230\u6587\u4ef6\u3002\u8fd9\u662f\u56e0\u4e3aKafka\u662f\u987a\u5e8f\u5199\u5165o\uff081\uff09\u7684\u65f6\u95f4\u590d\u6742\u5ea6\uff0c\u901f\u5ea6\u975e\u5e38\u5feb\u3002\u4e5f\u662f\u9ad8\u541e\u5410\u91cf\u7684\u539f\u56e0\u3002\u7531\u4e8emessage\u7684\u5199\u5165\u6301\u4e45\u5316\u662f\u987a\u5e8f\u5199\u5165\u7684\uff0c\u56e0\u6b64message\u5728\u88ab\u6d88\u8d39\u7684\u65f6\u5019\u4e5f\u662f\u6309\u987a\u5e8f\u88ab\u6d88\u8d39\u7684\uff0c\u4fdd\u8bc1partition\u7684message\u662f\u987a\u5e8f\u6d88\u8d39\u7684\u3002\u4e00\u822c\u7684\u673a\u5668,\u5355\u673a\u6bcf\u79d2100k\u6761\u6570\u636e\u3002<\/div>\n<div>&#8211;\u00a0message\u6709\u6548\u671f\uff1aKafka\u4f1a\u957f\u4e45\u4fdd\u7559\u5176\u4e2d\u7684\u6d88\u606f\uff0c\u4ee5\u4fbfconsumer\u53ef\u4ee5\u591a\u6b21\u6d88\u8d39\uff0c\u5f53\u7136\u5176\u4e2d\u5f88\u591a\u7ec6\u8282\u662f\u53ef\u914d\u7f6e\u7684\u3002<\/div>\n<div>&#8211;\u00a0Produer :\u00a0Producer\u5411Topic\u53d1\u9001message\uff0c\u4e0d\u9700\u8981\u6307\u5b9apartition\uff0c\u76f4\u63a5\u53d1\u9001\u5c31\u597d\u4e86\u3002kafka\u901a\u8fc7partition ack\u6765\u63a7\u5236\u662f\u5426\u53d1\u9001\u6210\u529f\u5e76\u628a\u4fe1\u606f\u8fd4\u56de\u7ed9producer\uff0cproducer\u53ef\u4ee5\u6709\u4efb\u610f\u591a\u7684thread\uff0c\u8fd9\u4e9bkafka\u670d\u52a1\u5668\u7aef\u662f\u4e0dcare\u7684\u3002Producer\u7aef\u7684delivery guarantee\u9ed8\u8ba4\u662fAt least once\u7684\u3002\u4e5f\u53ef\u4ee5\u8bbe\u7f6eProducer\u5f02\u6b65\u53d1\u9001\u5b9e\u73b0At most once\u3002Producer\u53ef\u4ee5\u7528\u4e3b\u952e\u5e42\u7b49\u6027\u5b9e\u73b0Exactly once<\/div>\n<div>&#8211;\u00a0Kafka\u9ad8\u541e\u5410\u91cf\uff1a Kafka\u7684\u9ad8\u541e\u5410\u91cf\u4f53\u73b0\u5728\u8bfb\u5199\u4e0a\uff0c\u5206\u5e03\u5f0f\u5e76\u53d1\u7684\u8bfb\u548c\u5199\u90fd\u975e\u5e38\u5feb\uff0c\u5199\u7684\u6027\u80fd\u4f53\u73b0\u5728\u4ee5o(1)\u7684\u65f6\u95f4\u590d\u6742\u5ea6\u8fdb\u884c\u987a\u5e8f\u5199\u5165\u3002\u8bfb\u7684\u6027\u80fd\u4f53\u73b0\u5728\u4ee5o(1)\u7684\u65f6\u95f4\u590d\u6742\u5ea6\u8fdb\u884c\u987a\u5e8f\u8bfb\u53d6\uff0c\u00a0\u5bf9topic\u8fdb\u884cpartition\u5206\u533a\uff0cconsume group\u4e2d\u7684consume\u7ebf\u7a0b\u53ef\u4ee5\u4ee5\u5f88\u9ad8\u80fd\u6027\u80fd\u8fdb\u884c\u987a\u5e8f\u8bfb\u3002<\/div>\n<div>&#8211; Kafka delivery guarantee(message\u4f20\u9001\u4fdd\u8bc1)\uff1a\uff081\uff09At most once\u6d88\u606f\u53ef\u80fd\u4f1a\u4e22\uff0c\u7edd\u5bf9\u4e0d\u4f1a\u91cd\u590d\u4f20\u8f93\uff1b\uff082\uff09At least once \u6d88\u606f\u7edd\u5bf9\u4e0d\u4f1a\u4e22\uff0c\u4f46\u662f\u53ef\u80fd\u4f1a\u91cd\u590d\u4f20\u8f93\uff1b\uff083\uff09Exactly once\u6bcf\u6761\u4fe1\u606f\u80af\u5b9a\u4f1a\u88ab\u4f20\u8f93\u4e00\u6b21\u4e14\u4ec5\u4f20\u8f93\u4e00\u6b21\uff0c\u8fd9\u662f\u7528\u6237\u60f3\u8981\u7684\u3002<\/div>\n<div>&#8211;\u00a0\u6279\u91cf\u53d1\u9001\uff1aKafka\u652f\u6301\u4ee5\u6d88\u606f\u96c6\u5408\u4e3a\u5355\u4f4d\u8fdb\u884c\u6279\u91cf\u53d1\u9001\uff0c\u4ee5\u63d0\u9ad8push\u6548\u7387\u3002<\/div>\n<div>&#8211;\u00a0push-and-pull\u00a0: Kafka\u4e2d\u7684Producer\u548cconsumer\u91c7\u7528\u7684\u662fpush-and-pull\u6a21\u5f0f\uff0c\u5373Producer\u53ea\u7ba1\u5411broker push\u6d88\u606f\uff0cconsumer\u53ea\u7ba1\u4ecebroker pull\u6d88\u606f\uff0c\u4e24\u8005\u5bf9\u6d88\u606f\u7684\u751f\u4ea7\u548c\u6d88\u8d39\u662f\u5f02\u6b65\u7684\u3002<\/div>\n<div>&#8211;\u00a0Kafka\u96c6\u7fa4\u4e2dbroker\u4e4b\u95f4\u7684\u5173\u7cfb\uff1a\u4e0d\u662f\u4e3b\u4ece\u5173\u7cfb\uff0c\u5404\u4e2abroker\u5728\u96c6\u7fa4\u4e2d\u5730\u4f4d\u4e00\u6837\uff0c\u6211\u4eec\u53ef\u4ee5\u968f\u610f\u7684\u589e\u52a0\u6216\u5220\u9664\u4efb\u4f55\u4e00\u4e2abroker\u8282\u70b9\u3002<\/div>\n<div>&#8211;\u00a0\u8d1f\u8f7d\u5747\u8861\u65b9\u9762\uff1a Kafka\u63d0\u4f9b\u4e86\u4e00\u4e2a metadata API\u6765\u7ba1\u7406broker\u4e4b\u95f4\u7684\u8d1f\u8f7d\uff08\u5bf9Kafka0.8.x\u800c\u8a00\uff0c\u5bf9\u4e8e0.7.x\u4e3b\u8981\u9760zookeeper\u6765\u5b9e\u73b0\u8d1f\u8f7d\u5747\u8861\uff09\u3002<\/div>\n<div>&#8211;\u00a0\u540c\u6b65\u5f02\u6b65\uff1aProducer\u91c7\u7528\u5f02\u6b65push\u65b9\u5f0f\uff0c\u6781\u5927\u63d0\u9ad8Kafka\u7cfb\u7edf\u7684\u541e\u5410\u7387\uff08\u53ef\u4ee5\u901a\u8fc7\u53c2\u6570\u63a7\u5236\u662f\u91c7\u7528\u540c\u6b65\u8fd8\u662f\u5f02\u6b65\u65b9\u5f0f\uff09\u3002<\/div>\n<div>&#8211;\u00a0\u5206\u533a\u673a\u5236partition\uff1aKafka\u7684broker\u7aef\u652f\u6301\u6d88\u606f\u5206\u533apartition\uff0cProducer\u53ef\u4ee5\u51b3\u5b9a\u628a\u6d88\u606f\u53d1\u5230\u54ea\u4e2apartition\uff0c\u5728\u4e00\u4e2apartition\u00a0\u4e2dmessage\u7684\u987a\u5e8f\u5c31\u662fProducer\u53d1\u9001\u6d88\u606f\u7684\u987a\u5e8f\uff0c\u4e00\u4e2atopic\u4e2d\u53ef\u4ee5\u6709\u591a\u4e2apartition\uff0c\u5177\u4f53partition\u7684\u6570\u91cf\u662f\u53ef\u914d\u7f6e\u7684\u3002partition\u7684\u6982\u5ff5\u4f7f\u5f97kafka\u4f5c\u4e3aMQ\u53ef\u4ee5\u6a2a\u5411\u6269\u5c55\uff0c\u541e\u5410\u91cf\u5de8\u5927\u3002partition\u53ef\u4ee5\u8bbe\u7f6ereplica\u526f\u672c\uff0creplica\u526f\u672c\u5b58\u5728\u4e0d\u540c\u7684kafka broker\u8282\u70b9\u4e0a\uff0c\u7b2c\u4e00\u4e2apartition\u662fleader,\u5176\u4ed6\u7684\u662ffollower\uff0cmessage\u5148\u5199\u5230partition leader\u4e0a\uff0c\u518d\u7531partition leader push\u5230parition follower\u4e0a\u3002\u6240\u4ee5\u8bf4kafka\u53ef\u4ee5\u6c34\u5e73\u6269\u5c55\uff0c\u4e5f\u5c31\u662f\u6269\u5c55partition\u3002<\/div>\n<div>&#8211;\u00a0\u79bb\u7ebf\u6570\u636e\u88c5\u8f7d\uff1aKafka\u7531\u4e8e\u5bf9\u53ef\u62d3\u5c55\u7684\u6570\u636e\u6301\u4e45\u5316\u7684\u652f\u6301\uff0c\u5b83\u4e5f\u975e\u5e38\u9002\u5408\u5411Hadoop\u6216\u8005\u6570\u636e\u4ed3\u5e93\u4e2d\u8fdb\u884c\u6570\u636e\u88c5\u8f7d\u3002<\/div>\n<div>&#8211;\u00a0\u5b9e\u65f6\u6570\u636e\u4e0e\u79bb\u7ebf\u6570\u636e\uff1akafka\u65e2\u652f\u6301\u79bb\u7ebf\u6570\u636e\u4e5f\u652f\u6301\u5b9e\u65f6\u6570\u636e\uff0c\u56e0\u4e3akafka\u7684message\u6301\u4e45\u5316\u5230\u6587\u4ef6\uff0c\u5e76\u53ef\u4ee5\u8bbe\u7f6e\u6709\u6548\u671f\uff0c\u56e0\u6b64\u53ef\u4ee5\u628akafka\u4f5c\u4e3a\u4e00\u4e2a\u9ad8\u6548\u7684\u5b58\u50a8\u6765\u4f7f\u7528\uff0c\u53ef\u4ee5\u4f5c\u4e3a\u79bb\u7ebf\u6570\u636e\u4f9b\u540e\u9762\u7684\u5206\u6790\u3002\u5f53\u7136\u4f5c\u4e3a\u5206\u5e03\u5f0f\u5b9e\u65f6\u6d88\u606f\u7cfb\u7edf\uff0c\u5927\u591a\u6570\u60c5\u51b5\u4e0b\u8fd8\u662f\u7528\u4e8e\u5b9e\u65f6\u7684\u6570\u636e\u5904\u7406\u7684\uff0c\u4f46\u662f\u5f53cosumer\u6d88\u8d39\u80fd\u529b\u4e0b\u964d\u7684\u65f6\u5019\u53ef\u4ee5\u901a\u8fc7message\u7684\u6301\u4e45\u5316\u5728\u6de4\u79ef\u6570\u636e\u5728kafka\u3002<\/div>\n<div>&#8211;\u00a0\u63d2\u4ef6\u652f\u6301\uff1a\u73b0\u5728\u4e0d\u5c11\u6d3b\u8dc3\u7684\u793e\u533a\u5df2\u7ecf\u5f00\u53d1\u51fa\u4e0d\u5c11\u63d2\u4ef6\u6765\u62d3\u5c55Kafka\u7684\u529f\u80fd\uff0c\u5982\u7528\u6765\u914d\u5408Storm\u3001Hadoop\u3001flume\u76f8\u5173\u7684\u63d2\u4ef6\u3002<\/div>\n<div>&#8211;\u00a0\u89e3\u8026: \u00a0\u76f8\u5f53\u4e8e\u4e00\u4e2aMQ\uff0c\u4f7f\u5f97Producer\u548cConsumer\u4e4b\u95f4\u5f02\u6b65\u7684\u64cd\u4f5c\uff0c\u7cfb\u7edf\u4e4b\u95f4\u89e3\u8026<\/div>\n<div>&#8211;\u00a0\u5197\u4f59: \u00a0replica\u6709\u591a\u4e2a\u526f\u672c\uff0c\u4fdd\u8bc1\u4e00\u4e2abroker node\u5b95\u673a\u540e\u4e0d\u4f1a\u5f71\u54cd\u6574\u4e2a\u670d\u52a1<\/div>\n<div>&#8211;\u00a0\u6269\u5c55\u6027: \u00a0broker\u8282\u70b9\u53ef\u4ee5\u6c34\u5e73\u6269\u5c55\uff0cpartition\u4e5f\u53ef\u4ee5\u6c34\u5e73\u589e\u52a0\uff0cpartition replica\u4e5f\u53ef\u4ee5\u6c34\u5e73\u589e\u52a0<\/div>\n<div>&#8211;\u00a0\u5cf0\u503c: \u00a0\u5728\u8bbf\u95ee\u91cf\u5267\u589e\u7684\u60c5\u51b5\u4e0b\uff0ckafka\u6c34\u5e73\u6269\u5c55, \u5e94\u7528\u4ecd\u7136\u9700\u8981\u7ee7\u7eed\u53d1\u6325\u4f5c\u7528<\/div>\n<div>&#8211;\u00a0\u53ef\u6062\u590d\u6027: \u00a0\u7cfb\u7edf\u7684\u4e00\u90e8\u5206\u7ec4\u4ef6\u5931\u6548\u65f6\uff0c\u7531\u4e8e\u6709partition\u7684replica\u526f\u672c\uff0c\u4e0d\u4f1a\u5f71\u54cd\u5230\u6574\u4e2a\u7cfb\u7edf\u3002<\/div>\n<div>&#8211;\u00a0\u987a\u5e8f\u4fdd\u8bc1\u6027\uff1a\u7531\u4e8ekafka\u7684producer\u7684\u5199message\u4e0econsumer\u53bb\u8bfbmessage\u90fd\u662f\u987a\u5e8f\u7684\u8bfb\u5199\uff0c\u4fdd\u8bc1\u4e86\u9ad8\u6548\u7684\u6027\u80fd\u3002<\/div>\n<div>&#8211;\u00a0\u7f13\u51b2\uff1a\u7531\u4e8eproducer\u90a3\u9762\u53ef\u80fd\u4e1a\u52a1\u5f88\u7b80\u5355\uff0c\u800c\u540e\u7aefconsumer\u4e1a\u52a1\u4f1a\u5f88\u590d\u6742\u5e76\u6709\u6570\u636e\u5e93\u7684\u64cd\u4f5c\uff0c\u56e0\u6b64\u80af\u5b9a\u662fproducer\u4f1a\u6bd4consumer\u5904\u7406\u901f\u5ea6\u5feb\uff0c\u5982\u679c\u6ca1\u6709kafka\uff0cproducer\u76f4\u63a5\u8c03\u7528consumer\uff0c\u90a3\u4e48\u5c31\u4f1a\u9020\u6210\u6574\u4e2a\u7cfb\u7edf\u7684\u5904\u7406\u901f\u5ea6\u6162\uff0c\u52a0\u4e00\u5c42kafka\u4f5c\u4e3aMQ\uff0c\u53ef\u4ee5\u8d77\u5230\u7f13\u51b2\u7684\u4f5c\u7528\u3002<\/div>\n<div>&#8211;\u00a0\u5f02\u6b65\u901a\u4fe1\uff1a\u4f5c\u4e3aMQ\uff0cProducer\u4e0eConsumer\u5f02\u6b65\u901a\u4fe1<\/div>\n<\/div>\n<h2><a name=\"t0\"><\/a>2.Kafka\u6587\u4ef6\u5b58\u50a8\u673a\u5236<\/h2>\n<div>\n<div>2.1 Kafka\u90e8\u5206\u540d\u8bcd\u89e3\u91ca\u5982\u4e0b\uff1a<\/div>\n<div><\/div>\n<div>\u00a0\u00a0\u00a0\u00a0\u00a0Kafka\u4e2d\u53d1\u5e03\u8ba2\u9605\u7684\u5bf9\u8c61\u662ftopic\u3002\u6211\u4eec\u53ef\u4ee5\u4e3a\u6bcf\u7c7b\u6570\u636e\u521b\u5efa\u4e00\u4e2atopic\uff0c\u628a\u5411topic\u53d1\u5e03\u6d88\u606f\u7684\u5ba2\u6237\u7aef\u79f0\u4f5cproducer\uff0c\u4ecetopic\u8ba2\u9605\u6d88\u606f\u7684\u5ba2\u6237\u7aef\u79f0\u4f5cconsumer\u3002Producers\u548cconsumers\u53ef\u4ee5\u540c\u65f6\u4ece\u591a\u4e2atopic\u8bfb\u5199\u6570\u636e\u3002\u4e00\u4e2akafka\u96c6\u7fa4\u7531\u4e00\u4e2a\u6216\u591a\u4e2abroker\u670d\u52a1\u5668\u7ec4\u6210\uff0c\u5b83\u8d1f\u8d23\u6301\u4e45\u5316\u548c\u5907\u4efd\u5177\u4f53\u7684kafka\u6d88\u606f\u3002<\/div>\n<\/div>\n<ul>\n<li>Broker\uff1aKafka\u8282\u70b9\uff0c\u4e00\u4e2aKafka\u8282\u70b9\u5c31\u662f\u4e00\u4e2abroker\uff0c\u591a\u4e2abroker\u53ef\u4ee5\u7ec4\u6210\u4e00\u4e2aKafka\u96c6\u7fa4\u3002<\/li>\n<li>Topic\uff1a\u4e00\u7c7b\u6d88\u606f\uff0c\u6d88\u606f\u5b58\u653e\u7684\u76ee\u5f55\u5373\u4e3b\u9898\uff0c\u4f8b\u5982page view\u65e5\u5fd7\u3001click\u65e5\u5fd7\u7b49\u90fd\u53ef\u4ee5\u4ee5topic\u7684\u5f62\u5f0f\u5b58\u5728\uff0cKafka\u96c6\u7fa4\u80fd\u591f\u540c\u65f6\u8d1f\u8d23\u591a\u4e2atopic\u7684\u5206\u53d1\u3002<\/li>\n<li>Partition\uff1atopic\u7269\u7406\u4e0a\u7684\u5206\u7ec4\uff0c\u4e00\u4e2atopic\u53ef\u4ee5\u5206\u4e3a\u591a\u4e2apartition\uff0c\u6bcf\u4e2apartition\u662f\u4e00\u4e2a\u6709\u5e8f\u7684\u961f\u5217<\/li>\n<li>Segment\uff1apartition\u7269\u7406\u4e0a\u7531\u591a\u4e2asegment\u7ec4\u6210\uff0c\u6bcf\u4e2aSegment\u5b58\u7740message\u4fe1\u606f<\/li>\n<li>Producer\u00a0: \u751f\u4ea7message\u53d1\u9001\u5230topic<\/li>\n<li>Consumer\u00a0: \u8ba2\u9605topic\u6d88\u8d39message, consumer\u4f5c\u4e3a\u4e00\u4e2a\u7ebf\u7a0b\u6765\u6d88\u8d39<\/li>\n<li>Consumer Group\uff1a\u4e00\u4e2aConsumer Group\u5305\u542b\u591a\u4e2aconsumer, \u8fd9\u4e2a\u662f\u9884\u5148\u5728\u914d\u7f6e\u6587\u4ef6\u4e2d\u914d\u7f6e\u597d\u7684\u3002\u5404\u4e2aconsumer\uff08consumer \u7ebf\u7a0b\uff09\u53ef\u4ee5\u7ec4\u6210\u4e00\u4e2a\u7ec4\uff08Consumer group \uff09\uff0cpartition\u4e2d\u7684\u6bcf\u4e2amessage\u53ea\u80fd\u88ab\u7ec4\uff08Consumer group \uff09 \u4e2d\u7684\u4e00\u4e2aconsumer\uff08consumer \u7ebf\u7a0b \uff09\u6d88\u8d39\uff0c\u5982\u679c\u4e00\u4e2amessage\u53ef\u4ee5\u88ab\u591a\u4e2aconsumer\uff08consumer \u7ebf\u7a0b \uff09 \u6d88\u8d39\u7684\u8bdd\uff0c\u90a3\u4e48\u8fd9\u4e9bconsumer\u5fc5\u987b\u5728\u4e0d\u540c\u7684\u7ec4\u3002Kafka\u4e0d\u652f\u6301\u4e00\u4e2apartition\u4e2d\u7684message\u7531\u4e24\u4e2a\u6216\u4e24\u4e2a\u4ee5\u4e0a\u7684consumer thread\u6765\u5904\u7406\uff0c\u5373\u4fbf\u662f\u6765\u81ea\u4e0d\u540c\u7684consumer group\u7684\u4e5f\u4e0d\u884c\u3002\u5b83\u4e0d\u80fd\u50cfAMQ\u90a3\u6837\u53ef\u4ee5\u591a\u4e2aBET\u4f5c\u4e3aconsumer\u53bb\u5904\u7406message\uff0c\u8fd9\u662f\u56e0\u4e3a\u591a\u4e2aBET\u53bb\u6d88\u8d39\u4e00\u4e2aQueue\u4e2d\u7684\u6570\u636e\u7684\u65f6\u5019\uff0c\u7531\u4e8e\u8981\u4fdd\u8bc1\u4e0d\u80fd\u591a\u4e2a\u7ebf\u7a0b\u62ff\u540c\u4e00\u6761message\uff0c\u6240\u4ee5\u5c31\u9700\u8981\u884c\u7ea7\u522b\u60b2\u89c2\u6240\uff08for update\uff09,\u8fd9\u5c31\u5bfc\u81f4\u4e86consume\u7684\u6027\u80fd\u4e0b\u964d\uff0c\u541e\u5410\u91cf\u4e0d\u591f\u3002\u800ckafka\u4e3a\u4e86\u4fdd\u8bc1\u541e\u5410\u91cf\uff0c\u53ea\u5141\u8bb8\u4e00\u4e2aconsumer\u7ebf\u7a0b\u53bb\u8bbf\u95ee\u4e00\u4e2apartition\u3002\u5982\u679c\u89c9\u5f97\u6548\u7387\u4e0d\u9ad8\u7684\u65f6\u5019\uff0c\u53ef\u4ee5\u52a0partition\u7684\u6570\u91cf\u6765\u6a2a\u5411\u6269\u5c55\uff0c\u90a3\u4e48\u518d\u52a0\u65b0\u7684consumer thread\u53bb\u6d88\u8d39\u3002\u8fd9\u6837\u6ca1\u6709\u9501\u7ade\u4e89\uff0c\u5145\u5206\u53d1\u6325\u4e86\u6a2a\u5411\u7684\u6269\u5c55\u6027\uff0c\u541e\u5410\u91cf\u6781\u9ad8\u3002\u8fd9\u4e5f\u5c31\u5f62\u6210\u4e86\u5206\u5e03\u5f0f\u6d88\u8d39\u7684\u6982\u5ff5\u3002<\/li>\n<li>2.2 kafka\u4e00\u4e9b\u539f\u7406\u6982\u5ff5<\/li>\n<\/ul>\n<div>\n<div>1.\u6301\u4e45\u5316<\/div>\n<div>kafka\u4f7f\u7528\u6587\u4ef6\u5b58\u50a8\u6d88\u606f(append only log),\u8fd9\u5c31\u76f4\u63a5\u51b3\u5b9akafka\u5728\u6027\u80fd\u4e0a\u4e25\u91cd\u4f9d\u8d56\u6587\u4ef6\u7cfb\u7edf\u7684\u672c\u8eab\u7279\u6027.\u4e14\u65e0\u8bba\u4efb\u4f55OS\u4e0b,\u5bf9\u6587\u4ef6\u7cfb\u7edf\u672c\u8eab\u7684\u4f18\u5316\u662f\u975e\u5e38\u8270\u96be\u7684.\u6587\u4ef6\u7f13\u5b58\/\u76f4\u63a5\u5185\u5b58\u6620\u5c04\u7b49\u662f\u5e38\u7528\u7684\u624b\u6bb5.\u56e0\u4e3akafka\u662f\u5bf9\u65e5\u5fd7\u6587\u4ef6\u8fdb\u884cappend\u64cd\u4f5c,\u56e0\u6b64\u78c1\u76d8\u68c0\u7d22\u7684\u5f00\u652f\u662f\u8f83\u5c0f\u7684;\u540c\u65f6\u4e3a\u4e86\u51cf\u5c11\u78c1\u76d8\u5199\u5165\u7684\u6b21\u6570,broker\u4f1a\u5c06\u6d88\u606f\u6682\u65f6buffer\u8d77\u6765,\u5f53\u6d88\u606f\u7684\u4e2a\u6570(\u6216\u5c3a\u5bf8)\u8fbe\u5230\u4e00\u5b9a\u9600\u503c\u65f6,\u518dflush\u5230\u78c1\u76d8,\u8fd9\u6837\u51cf\u5c11\u4e86\u78c1\u76d8IO\u8c03\u7528\u7684\u6b21\u6570.\u5bf9\u4e8ekafka\u800c\u8a00,\u8f83\u9ad8\u6027\u80fd\u7684\u78c1\u76d8,\u5c06\u4f1a\u5e26\u6765\u66f4\u52a0\u76f4\u63a5\u7684\u6027\u80fd\u63d0\u5347.<\/div>\n<div><\/div>\n<div>2.\u6027\u80fd<\/div>\n<div>\u9664\u78c1\u76d8IO\u4e4b\u5916,\u6211\u4eec\u8fd8\u9700\u8981\u8003\u8651\u7f51\u7edcIO,\u8fd9\u76f4\u63a5\u5173\u7cfb\u5230kafka\u7684\u541e\u5410\u91cf\u95ee\u9898.kafka\u5e76\u6ca1\u6709\u63d0\u4f9b\u592a\u591a\u9ad8\u8d85\u7684\u6280\u5de7;\u5bf9\u4e8eproducer\u7aef,\u53ef\u4ee5\u5c06\u6d88\u606fbuffer\u8d77\u6765,\u5f53\u6d88\u606f\u7684\u6761\u6570\u8fbe\u5230\u4e00\u5b9a\u9600\u503c\u65f6,\u6279\u91cf\u53d1\u9001\u7ed9broker;\u5bf9\u4e8econsumer\u7aef\u4e5f\u662f\u4e00\u6837,\u6279\u91cffetch\u591a\u6761\u6d88\u606f.\u4e0d\u8fc7\u6d88\u606f\u91cf\u7684\u5927\u5c0f\u53ef\u4ee5\u901a\u8fc7\u914d\u7f6e\u6587\u4ef6\u6765\u6307\u5b9a.\u5bf9\u4e8ekafka broker\u7aef,\u4f3c\u4e4e\u6709\u4e2asendfile\u7cfb\u7edf\u8c03\u7528\u53ef\u4ee5\u6f5c\u5728\u7684\u63d0\u5347\u7f51\u7edcIO\u7684\u6027\u80fd:\u5c06\u6587\u4ef6\u7684\u6570\u636e\u6620\u5c04\u5230\u7cfb\u7edf\u5185\u5b58\u4e2d,socket\u76f4\u63a5\u8bfb\u53d6\u76f8\u5e94\u7684\u5185\u5b58\u533a\u57df\u5373\u53ef,\u800c\u65e0\u9700\u8fdb\u7a0b\u518d\u6b21copy\u548c\u4ea4\u6362(\u8fd9\u91cc\u6d89\u53ca\u5230&#8221;\u78c1\u76d8IO\u6570\u636e&#8221;\/&#8221;\u5185\u6838\u5185\u5b58&#8221;\/&#8221;\u8fdb\u7a0b\u5185\u5b58&#8221;\/&#8221;\u7f51\u7edc\u7f13\u51b2\u533a&#8221;,\u591a\u8005\u4e4b\u95f4\u7684\u6570\u636ecopy).<\/div>\n<div>\u5176\u5b9e\u5bf9\u4e8eproducer\/consumer\/broker\u4e09\u8005\u800c\u8a00,CPU\u7684\u5f00\u652f\u5e94\u8be5\u90fd\u4e0d\u5927,\u56e0\u6b64\u542f\u7528\u6d88\u606f\u538b\u7f29\u673a\u5236\u662f\u4e00\u4e2a\u826f\u597d\u7684\u7b56\u7565;\u538b\u7f29\u9700\u8981\u6d88\u8017\u5c11\u91cf\u7684CPU\u8d44\u6e90,\u4e0d\u8fc7\u5bf9\u4e8ekafka\u800c\u8a00,\u7f51\u7edcIO\u66f4\u5e94\u8be5\u9700\u8981\u8003\u8651.\u53ef\u4ee5\u5c06\u4efb\u4f55\u5728\u7f51\u7edc\u4e0a\u4f20\u8f93\u7684\u6d88\u606f\u90fd\u7ecf\u8fc7\u538b\u7f29.kafka\u652f\u6301gzip\/snappy\u7b49\u591a\u79cd\u538b\u7f29\u65b9\u5f0f.<\/div>\n<div><\/div>\n<div>3.\u8d1f\u8f7d\u5747\u8861<\/div>\n<div>kafka\u96c6\u7fa4\u4e2d\u7684\u4efb\u4f55\u4e00\u4e2abroker,\u90fd\u53ef\u4ee5\u5411producer\u63d0\u4f9bmetadata\u4fe1\u606f,\u8fd9\u4e9bmetadata\u4e2d\u5305\u542b&#8221;\u96c6\u7fa4\u4e2d\u5b58\u6d3b\u7684servers\u5217\u8868&#8221;\/&#8221;partitions leader\u5217\u8868&#8221;\u7b49\u4fe1\u606f(\u8bf7\u53c2\u770bzookeeper\u4e2d\u7684\u8282\u70b9\u4fe1\u606f). \u5f53producer\u83b7\u53d6\u5230metadata\u4fe1\u606f\u4e4b\u540e, producer\u5c06\u4f1a\u548cTopic\u4e0b\u6240\u6709partition leader\u4fdd\u6301socket\u8fde\u63a5;\u6d88\u606f\u7531producer\u76f4\u63a5\u901a\u8fc7socket\u53d1\u9001\u5230broker,\u4e2d\u95f4\u4e0d\u4f1a\u7ecf\u8fc7\u4efb\u4f55&#8221;\u8def\u7531\u5c42&#8221;.<\/div>\n<div>\u5f02\u6b65\u53d1\u9001\uff0c\u5c06\u591a\u6761\u6d88\u606f\u6682\u4e14\u5728\u5ba2\u6237\u7aefbuffer\u8d77\u6765,\u5e76\u5c06\u4ed6\u4eec\u6279\u91cf\u53d1\u9001\u5230broker;\u5c0f\u6570\u636eIO\u592a\u591a,\u4f1a\u62d6\u6162\u6574\u4f53\u7684\u7f51\u7edc\u5ef6\u8fdf,\u6279\u91cf\u5ef6\u8fdf\u53d1\u9001\u4e8b\u5b9e\u4e0a\u63d0\u5347\u4e86\u7f51\u7edc\u6548\u7387;\u4e0d\u8fc7\u8fd9\u4e5f\u6709\u4e00\u5b9a\u7684\u9690\u60a3,\u6bd4\u5982\u5f53producer\u5931\u6548\u65f6,\u90a3\u4e9b\u5c1a\u672a\u53d1\u9001\u7684\u6d88\u606f\u5c06\u4f1a\u4e22\u5931\u3002<\/div>\n<div><\/div>\n<div>4.Topic\u6a21\u578b<\/div>\n<div>\u5176\u4ed6JMS\u5b9e\u73b0,\u6d88\u606f\u6d88\u8d39\u7684\u4f4d\u7f6e\u662f\u6709prodiver\u4fdd\u7559,\u4ee5\u4fbf\u907f\u514d\u91cd\u590d\u53d1\u9001\u6d88\u606f\u6216\u8005\u5c06\u6ca1\u6709\u6d88\u8d39\u6210\u529f\u7684\u6d88\u606f\u91cd\u53d1\u7b49,\u540c\u65f6\u8fd8\u8981\u63a7\u5236\u6d88\u606f\u7684\u72b6\u6001.\u8fd9\u5c31\u8981\u6c42JMS broker\u9700\u8981\u592a\u591a\u989d\u5916\u7684\u5de5\u4f5c.\u5728kafka\u4e2d,partition\u4e2d\u7684\u6d88\u606f\u53ea\u6709\u4e00\u4e2aconsumer\u5728\u6d88\u8d39,\u4e14\u4e0d\u5b58\u5728\u6d88\u606f\u72b6\u6001\u7684\u63a7\u5236,\u4e5f\u6ca1\u6709\u590d\u6742\u7684\u6d88\u606f\u786e\u8ba4\u673a\u5236,\u53ef\u89c1kafka broker\u7aef\u662f\u76f8\u5f53\u8f7b\u91cf\u7ea7\u7684.\u5f53\u6d88\u606f\u88abconsumer\u63a5\u6536\u4e4b\u540e,consumer\u53ef\u4ee5\u5728\u672c\u5730\u4fdd\u5b58\u6700\u540e\u6d88\u606f\u7684offset,\u5e76\u95f4\u6b47\u6027\u7684\u5411zookeeper\u6ce8\u518coffset.\u7531\u6b64\u53ef\u89c1,consumer\u5ba2\u6237\u7aef\u4e5f\u5f88\u8f7b\u91cf\u7ea7\u3002<\/div>\n<div>kafka\u4e2dconsumer\u8d1f\u8d23\u7ef4\u62a4\u6d88\u606f\u7684\u6d88\u8d39\u8bb0\u5f55,\u800cbroker\u5219\u4e0d\u5173\u5fc3\u8fd9\u4e9b,\u8fd9\u79cd\u8bbe\u8ba1\u4e0d\u4ec5\u63d0\u9ad8\u4e86consumer\u7aef\u7684\u7075\u6d3b\u6027,\u4e5f\u9002\u5ea6\u7684\u51cf\u8f7b\u4e86broker\u7aef\u8bbe\u8ba1\u7684\u590d\u6742\u5ea6;\u8fd9\u662f\u548c\u4f17\u591aJMS prodiver\u7684\u533a\u522b.\u6b64\u5916,kafka\u4e2d\u6d88\u606fACK\u7684\u8bbe\u8ba1\u4e5f\u548cJMS\u6709\u5f88\u5927\u4e0d\u540c,kafka\u4e2d\u7684\u6d88\u606f\u662f\u6279\u91cf(\u901a\u5e38\u4ee5\u6d88\u606f\u7684\u6761\u6570\u6216\u8005chunk\u7684\u5c3a\u5bf8\u4e3a\u5355\u4f4d)\u53d1\u9001\u7ed9consumer,\u5f53\u6d88\u606f\u6d88\u8d39\u6210\u529f\u540e,\u5411zookeeper\u63d0\u4ea4\u6d88\u606f\u7684offset,\u800c\u4e0d\u4f1a\u5411broker\u4ea4\u4ed8ACK.\u6216\u8bb8\u4f60\u5df2\u7ecf\u610f\u8bc6\u5230,\u8fd9\u79cd&#8221;\u5bbd\u677e&#8221;\u7684\u8bbe\u8ba1,\u5c06\u4f1a\u6709&#8221;\u4e22\u5931&#8221;\u6d88\u606f\/&#8221;\u6d88\u606f\u91cd\u53d1&#8221;\u7684\u5371\u9669.<\/div>\n<div><\/div>\n<div>5.\u6d88\u606f\u4f20\u8f93\u4e00\u81f4<\/div>\n<div>Kafka\u63d0\u4f9b3\u79cd\u6d88\u606f\u4f20\u8f93\u4e00\u81f4\u6027\u8bed\u4e49\uff1a\u6700\u591a1\u6b21\uff0c\u6700\u5c111\u6b21\uff0c\u6070\u597d1\u6b21\u3002<\/div>\n<div>\u6700\u5c111\u6b21\uff1a\u53ef\u80fd\u4f1a\u91cd\u4f20\u6570\u636e\uff0c\u6709\u53ef\u80fd\u51fa\u73b0\u6570\u636e\u88ab\u91cd\u590d\u5904\u7406\u7684\u60c5\u51b5;<\/div>\n<div>\u6700\u591a1\u6b21\uff1a\u53ef\u80fd\u4f1a\u51fa\u73b0\u6570\u636e\u4e22\u5931\u60c5\u51b5;<\/div>\n<div>\u6070\u597d1\u6b21\uff1a\u5e76\u4e0d\u662f\u6307\u771f\u6b63\u53ea\u4f20\u8f931\u6b21\uff0c\u53ea\u4e0d\u8fc7\u6709\u4e00\u4e2a\u673a\u5236\u3002\u786e\u4fdd\u4e0d\u4f1a\u51fa\u73b0\u201c\u6570\u636e\u88ab\u91cd\u590d\u5904\u7406\u201d\u548c\u201c\u6570\u636e\u4e22\u5931\u201d\u7684\u60c5\u51b5\u3002<\/div>\n<div><\/div>\n<div>at most once: \u6d88\u8d39\u8005fetch\u6d88\u606f,\u7136\u540e\u4fdd\u5b58offset,\u7136\u540e\u5904\u7406\u6d88\u606f;\u5f53client\u4fdd\u5b58offset\u4e4b\u540e,\u4f46\u662f\u5728\u6d88\u606f\u5904\u7406\u8fc7\u7a0b\u4e2dconsumer\u8fdb\u7a0b\u5931\u6548(crash),\u5bfc\u81f4\u90e8\u5206\u6d88\u606f\u672a\u80fd\u7ee7\u7eed\u5904\u7406.\u90a3\u4e48\u6b64\u540e\u53ef\u80fd\u5176\u4ed6consumer\u4f1a\u63a5\u7ba1,\u4f46\u662f\u56e0\u4e3aoffset\u5df2\u7ecf\u63d0\u524d\u4fdd\u5b58,\u90a3\u4e48\u65b0\u7684consumer\u5c06\u4e0d\u80fdfetch\u5230offset\u4e4b\u524d\u7684\u6d88\u606f(\u5c3d\u7ba1\u5b83\u4eec\u5c1a\u6ca1\u6709\u88ab\u5904\u7406),\u8fd9\u5c31\u662f&#8221;at most once&#8221;.<\/div>\n<div>at least once: \u6d88\u8d39\u8005fetch\u6d88\u606f,\u7136\u540e\u5904\u7406\u6d88\u606f,\u7136\u540e\u4fdd\u5b58offset.\u5982\u679c\u6d88\u606f\u5904\u7406\u6210\u529f\u4e4b\u540e,\u4f46\u662f\u5728\u4fdd\u5b58offset\u9636\u6bb5zookeeper\u5f02\u5e38\u6216\u8005consumer\u5931\u6548,\u5bfc\u81f4\u4fdd\u5b58offset\u64cd\u4f5c\u672a\u80fd\u6267\u884c\u6210\u529f,\u8fd9\u5c31\u5bfc\u81f4\u63a5\u4e0b\u6765\u518d\u6b21fetch\u65f6\u53ef\u80fd\u83b7\u5f97\u4e0a\u6b21\u5df2\u7ecf\u5904\u7406\u8fc7\u7684\u6d88\u606f,\u8fd9\u5c31\u662f&#8221;at least once&#8221;.<\/div>\n<div>&#8220;Kafka Cluster&#8221;\u5230\u6d88\u8d39\u8005\u7684\u573a\u666f\u4e2d\u53ef\u4ee5\u91c7\u53d6\u4ee5\u4e0b\u65b9\u6848\u6765\u5f97\u5230\u201c\u6070\u597d1\u6b21\u201d\u7684\u4e00\u81f4\u6027\u8bed\u4e49\uff1a<\/div>\n<div>\u6700\u5c111\u6b21\uff0b\u6d88\u8d39\u8005\u7684\u8f93\u51fa\u4e2d\u989d\u5916\u589e\u52a0\u5df2\u5904\u7406\u6d88\u606f\u6700\u5927\u7f16\u53f7\uff1a\u7531\u4e8e\u5df2\u5904\u7406\u6d88\u606f\u6700\u5927\u7f16\u53f7\u7684\u5b58\u5728\uff0c\u4e0d\u4f1a\u51fa\u73b0\u91cd\u590d\u5904\u7406\u6d88\u606f\u7684\u60c5\u51b5\u3002<\/div>\n<div><\/div>\n<div>6.\u526f\u672c<\/div>\n<div>kafka\u4e2d,replication\u7b56\u7565\u662f\u57fa\u4e8epartition,\u800c\u4e0d\u662ftopic;kafka\u5c06\u6bcf\u4e2apartition\u6570\u636e\u590d\u5236\u5230\u591a\u4e2aserver\u4e0a,\u4efb\u4f55\u4e00\u4e2apartition\u6709\u4e00\u4e2aleader\u548c\u591a\u4e2afollower(\u53ef\u4ee5\u6ca1\u6709);\u5907\u4efd\u7684\u4e2a\u6570\u53ef\u4ee5\u901a\u8fc7broker\u914d\u7f6e\u6587\u4ef6\u6765\u8bbe\u5b9a\u3002leader\u5904\u7406\u6240\u6709\u7684read-write\u8bf7\u6c42,follower\u9700\u8981\u548cleader\u4fdd\u6301\u540c\u6b65.Follower\u5c31\u50cf\u4e00\u4e2a&#8221;consumer&#8221;,\u6d88\u8d39\u6d88\u606f\u5e76\u4fdd\u5b58\u5728\u672c\u5730\u65e5\u5fd7\u4e2d;leader\u8d1f\u8d23\u8ddf\u8e2a\u6240\u6709\u7684follower\u72b6\u6001,\u5982\u679cfollower&#8221;\u843d\u540e&#8221;\u592a\u591a\u6216\u8005\u5931\u6548,leader\u5c06\u4f1a\u628a\u5b83\u4ecereplicas\u540c\u6b65\u5217\u8868\u4e2d\u5220\u9664.\u5f53\u6240\u6709\u7684follower\u90fd\u5c06\u4e00\u6761\u6d88\u606f\u4fdd\u5b58\u6210\u529f,\u6b64\u6d88\u606f\u624d\u88ab\u8ba4\u4e3a\u662f&#8221;committed&#8221;,\u90a3\u4e48\u6b64\u65f6consumer\u624d\u80fd\u6d88\u8d39\u5b83,\u8fd9\u79cd\u540c\u6b65\u7b56\u7565,\u5c31\u8981\u6c42follower\u548cleader\u4e4b\u95f4\u5fc5\u987b\u5177\u6709\u826f\u597d\u7684\u7f51\u7edc\u73af\u5883.\u5373\u4f7f\u53ea\u6709\u4e00\u4e2areplicas\u5b9e\u4f8b\u5b58\u6d3b,\u4ecd\u7136\u53ef\u4ee5\u4fdd\u8bc1\u6d88\u606f\u7684\u6b63\u5e38\u53d1\u9001\u548c\u63a5\u6536,\u53ea\u8981zookeeper\u96c6\u7fa4\u5b58\u6d3b\u5373\u53ef.<\/div>\n<div>\u9009\u62e9follower\u65f6\u9700\u8981\u517c\u987e\u4e00\u4e2a\u95ee\u9898,\u5c31\u662f\u65b0leader server\u4e0a\u6240\u5df2\u7ecf\u627f\u8f7d\u7684partition leader\u7684\u4e2a\u6570,\u5982\u679c\u4e00\u4e2aserver\u4e0a\u6709\u8fc7\u591a\u7684partition leader,\u610f\u5473\u7740\u6b64server\u5c06\u627f\u53d7\u7740\u66f4\u591a\u7684IO\u538b\u529b.\u5728\u9009\u4e3e\u65b0leader,\u9700\u8981\u8003\u8651\u5230&#8221;\u8d1f\u8f7d\u5747\u8861&#8221;,partition leader\u8f83\u5c11\u7684broker\u5c06\u4f1a\u66f4\u6709\u53ef\u80fd\u6210\u4e3a\u65b0\u7684leader.<\/div>\n<div><\/div>\n<div>7.log<\/div>\n<div>\u6bcf\u4e2alog entry\u683c\u5f0f\u4e3a&#8221;4\u4e2a\u5b57\u8282\u7684\u6570\u5b57N\u8868\u793a\u6d88\u606f\u7684\u957f\u5ea6&#8221; + &#8220;N\u4e2a\u5b57\u8282\u7684\u6d88\u606f\u5185\u5bb9&#8221;;\u6bcf\u4e2a\u65e5\u5fd7\u90fd\u6709\u4e00\u4e2aoffset\u6765\u552f\u4e00\u7684\u6807\u8bb0\u4e00\u6761\u6d88\u606f,offset\u7684\u503c\u4e3a8\u4e2a\u5b57\u8282\u7684\u6570\u5b57,\u8868\u793a\u6b64\u6d88\u606f\u5728\u6b64partition\u4e2d\u6240\u5904\u7684\u8d77\u59cb\u4f4d\u7f6e..\u6bcf\u4e2apartition\u5728\u7269\u7406\u5b58\u50a8\u5c42\u9762,\u6709\u591a\u4e2alog file\u7ec4\u6210(\u79f0\u4e3asegment).segment file\u7684\u547d\u540d\u4e3a&#8221;\u6700\u5c0foffset&#8221;.kafka.\u4f8b\u5982&#8221;00000000000.kafka&#8221;;\u5176\u4e2d&#8221;\u6700\u5c0foffset&#8221;\u8868\u793a\u6b64segment\u4e2d\u8d77\u59cb\u6d88\u606f\u7684offset.<\/div>\n<div>\u83b7\u53d6\u6d88\u606f\u65f6,\u9700\u8981\u6307\u5b9aoffset\u548c\u6700\u5927chunk\u5c3a\u5bf8,offset\u7528\u6765\u8868\u793a\u6d88\u606f\u7684\u8d77\u59cb\u4f4d\u7f6e,chunk size\u7528\u6765\u8868\u793a\u6700\u5927\u83b7\u53d6\u6d88\u606f\u7684\u603b\u957f\u5ea6(\u95f4\u63a5\u7684\u8868\u793a\u6d88\u606f\u7684\u6761\u6570).\u6839\u636eoffset,\u53ef\u4ee5\u627e\u5230\u6b64\u6d88\u606f\u6240\u5728segment\u6587\u4ef6,\u7136\u540e\u6839\u636esegment\u7684\u6700\u5c0foffset\u53d6\u5dee\u503c,\u5f97\u5230\u5b83\u5728file\u4e2d\u7684\u76f8\u5bf9\u4f4d\u7f6e,\u76f4\u63a5\u8bfb\u53d6\u8f93\u51fa\u5373\u53ef.<\/div>\n<div><\/div>\n<div><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107211730692?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<div>8.\u5206\u5e03\u5f0f<\/div>\n<div>kafka\u4f7f\u7528zookeeper\u6765\u5b58\u50a8\u4e00\u4e9bmeta\u4fe1\u606f,\u5e76\u4f7f\u7528\u4e86zookeeper watch\u673a\u5236\u6765\u53d1\u73b0meta\u4fe1\u606f\u7684\u53d8\u66f4\u5e76\u4f5c\u51fa\u76f8\u5e94\u7684\u52a8\u4f5c(\u6bd4\u5982consumer\u5931\u6548,\u89e6\u53d1\u8d1f\u8f7d\u5747\u8861\u7b49)<\/div>\n<div>Broker node registry: \u5f53\u4e00\u4e2akafka broker\u542f\u52a8\u540e,\u9996\u5148\u4f1a\u5411zookeeper\u6ce8\u518c\u81ea\u5df1\u7684\u8282\u70b9\u4fe1\u606f(\u4e34\u65f6znode),\u540c\u65f6\u5f53broker\u548czookeeper\u65ad\u5f00\u8fde\u63a5\u65f6,\u6b64znode\u4e5f\u4f1a\u88ab\u5220\u9664.<\/div>\n<div>Broker Topic Registry: \u5f53\u4e00\u4e2abroker\u542f\u52a8\u65f6,\u4f1a\u5411zookeeper\u6ce8\u518c\u81ea\u5df1\u6301\u6709\u7684topic\u548cpartitions\u4fe1\u606f,\u4ecd\u7136\u662f\u4e00\u4e2a\u4e34\u65f6znode.<\/div>\n<div>Consumer and Consumer group: \u6bcf\u4e2aconsumer\u5ba2\u6237\u7aef\u88ab\u521b\u5efa\u65f6,\u4f1a\u5411zookeeper\u6ce8\u518c\u81ea\u5df1\u7684\u4fe1\u606f;\u6b64\u4f5c\u7528\u4e3b\u8981\u662f\u4e3a\u4e86&#8221;\u8d1f\u8f7d\u5747\u8861&#8221;.\u4e00\u4e2agroup\u4e2d\u7684\u591a\u4e2aconsumer\u53ef\u4ee5\u4ea4\u9519\u7684\u6d88\u8d39\u4e00\u4e2atopic\u7684\u6240\u6709partitions;\u7b80\u800c\u8a00\u4e4b,\u4fdd\u8bc1\u6b64topic\u7684\u6240\u6709partitions\u90fd\u80fd\u88ab\u6b64group\u6240\u6d88\u8d39,\u4e14\u6d88\u8d39\u65f6\u4e3a\u4e86\u6027\u80fd\u8003\u8651,\u8ba9partition\u76f8\u5bf9\u5747\u8861\u7684\u5206\u6563\u5230\u6bcf\u4e2aconsumer\u4e0a.<\/div>\n<div>Consumer id Registry: \u6bcf\u4e2aconsumer\u90fd\u6709\u4e00\u4e2a\u552f\u4e00\u7684ID(host:uuid,\u53ef\u4ee5\u901a\u8fc7\u914d\u7f6e\u6587\u4ef6\u6307\u5b9a,\u4e5f\u53ef\u4ee5\u7531\u7cfb\u7edf\u751f\u6210),\u6b64id\u7528\u6765\u6807\u8bb0\u6d88\u8d39\u8005\u4fe1\u606f.<\/div>\n<div>Consumer offset Tracking: \u7528\u6765\u8ddf\u8e2a\u6bcf\u4e2aconsumer\u76ee\u524d\u6240\u6d88\u8d39\u7684partition\u4e2d\u6700\u5927\u7684offset.\u6b64znode\u4e3a\u6301\u4e45\u8282\u70b9,\u53ef\u4ee5\u770b\u51faoffset\u8ddfgroup_id\u6709\u5173,\u4ee5\u8868\u660e\u5f53group\u4e2d\u4e00\u4e2a\u6d88\u8d39\u8005\u5931\u6548,\u5176\u4ed6consumer\u53ef\u4ee5\u7ee7\u7eed\u6d88\u8d39.<\/div>\n<div>Partition Owner registry: \u7528\u6765\u6807\u8bb0partition\u6b63\u5728\u88ab\u54ea\u4e2aconsumer\u6d88\u8d39.\u4e34\u65f6znode\u3002\u6b64\u8282\u70b9\u8868\u8fbe\u4e86&#8221;\u4e00\u4e2apartition&#8221;\u53ea\u80fd\u88abgroup\u4e0b\u4e00\u4e2aconsumer\u6d88\u8d39,\u540c\u65f6\u5f53group\u4e0b\u67d0\u4e2aconsumer\u5931\u6548,\u90a3\u4e48\u5c06\u4f1a\u89e6\u53d1\u8d1f\u8f7d\u5747\u8861(\u5373:\u8ba9partitions\u5728\u591a\u4e2aconsumer\u95f4\u5747\u8861\u6d88\u8d39,\u63a5\u7ba1\u90a3\u4e9b&#8221;\u6e38\u79bb&#8221;\u7684partitions)<\/div>\n<div>\u5f53consumer\u542f\u52a8\u65f6,\u6240\u89e6\u53d1\u7684\u64cd\u4f5c:<\/div>\n<div>A) \u9996\u5148\u8fdb\u884c&#8221;Consumer id Registry&#8221;;<\/div>\n<div>B) \u7136\u540e\u5728&#8221;Consumer id Registry&#8221;\u8282\u70b9\u4e0b\u6ce8\u518c\u4e00\u4e2awatch\u7528\u6765\u76d1\u542c\u5f53\u524dgroup\u4e2d\u5176\u4ed6consumer\u7684&#8221;leave&#8221;\u548c&#8221;join&#8221;;\u53ea\u8981\u6b64znode path\u4e0b\u8282\u70b9\u5217\u8868\u53d8\u66f4,\u90fd\u4f1a\u89e6\u53d1\u6b64group\u4e0bconsumer\u7684\u8d1f\u8f7d\u5747\u8861.(\u6bd4\u5982\u4e00\u4e2aconsumer\u5931\u6548,\u90a3\u4e48\u5176\u4ed6consumer\u63a5\u7ba1partitions).<\/div>\n<div>C) \u5728&#8221;Broker id registry&#8221;\u8282\u70b9\u4e0b,\u6ce8\u518c\u4e00\u4e2awatch\u7528\u6765\u76d1\u542cbroker\u7684\u5b58\u6d3b\u60c5\u51b5;\u5982\u679cbroker\u5217\u8868\u53d8\u66f4,\u5c06\u4f1a\u89e6\u53d1\u6240\u6709\u7684groups\u4e0b\u7684consumer\u91cd\u65b0balance.<\/div>\n<div><\/div>\n<div>\u603b\u7ed3:<\/div>\n<div>1) Producer\u7aef\u4f7f\u7528zookeeper\u7528\u6765&#8221;\u53d1\u73b0&#8221;broker\u5217\u8868,\u4ee5\u53ca\u548cTopic\u4e0b\u6bcf\u4e2apartition leader\u5efa\u7acbsocket\u8fde\u63a5\u5e76\u53d1\u9001\u6d88\u606f.<\/div>\n<div>2) Broker\u7aef\u4f7f\u7528zookeeper\u7528\u6765\u6ce8\u518cbroker\u4fe1\u606f,\u5df2\u7ecf\u76d1\u6d4bpartition leader\u5b58\u6d3b\u6027.<\/div>\n<div>3) Consumer\u7aef\u4f7f\u7528zookeeper\u7528\u6765\u6ce8\u518cconsumer\u4fe1\u606f,\u5176\u4e2d\u5305\u62ecconsumer\u6d88\u8d39\u7684partition\u5217\u8868\u7b49,\u540c\u65f6\u4e5f\u7528\u6765\u53d1\u73b0broker\u5217\u8868,\u5e76\u548cpartition leader\u5efa\u7acbsocket\u8fde\u63a5,\u5e76\u83b7\u53d6\u6d88\u606f\u3002<\/div>\n<div><\/div>\n<div>9.Leader\u7684\u9009\u62e9<\/div>\n<div>Kafka\u7684\u6838\u5fc3\u662f\u65e5\u5fd7\u6587\u4ef6\uff0c\u65e5\u5fd7\u6587\u4ef6\u5728\u96c6\u7fa4\u4e2d\u7684\u540c\u6b65\u662f\u5206\u5e03\u5f0f\u6570\u636e\u7cfb\u7edf\u6700\u57fa\u7840\u7684\u8981\u7d20\u3002<\/div>\n<div>\u5982\u679cleaders\u6c38\u8fdc\u4e0d\u4f1adown\u7684\u8bdd\u6211\u4eec\u5c31\u4e0d\u9700\u8981followers\u4e86\uff01\u4e00\u65e6leader down\u6389\u4e86\uff0c\u9700\u8981\u5728followers\u4e2d\u9009\u62e9\u4e00\u4e2a\u65b0\u7684leader.\u4f46\u662ffollowers\u672c\u8eab\u6709\u53ef\u80fd\u5ef6\u65f6\u592a\u4e45\u6216\u8005crash\uff0c\u6240\u4ee5\u5fc5\u987b\u9009\u62e9\u9ad8\u8d28\u91cf\u7684follower\u4f5c\u4e3aleader.\u5fc5\u987b\u4fdd\u8bc1\uff0c\u4e00\u65e6\u4e00\u4e2a\u6d88\u606f\u88ab\u63d0\u4ea4\u4e86\uff0c\u4f46\u662fleader down\u6389\u4e86\uff0c\u65b0\u9009\u51fa\u7684leader\u5fc5\u987b\u53ef\u4ee5\u63d0\u4f9b\u8fd9\u6761\u6d88\u606f\u3002\u5927\u90e8\u5206\u7684\u5206\u5e03\u5f0f\u7cfb\u7edf\u91c7\u7528\u4e86\u591a\u6570\u6295\u7968\u6cd5\u5219\u9009\u62e9\u65b0\u7684leader,\u5bf9\u4e8e\u591a\u6570\u6295\u7968\u6cd5\u5219\uff0c\u5c31\u662f\u6839\u636e\u6240\u6709\u526f\u672c\u8282\u70b9\u7684\u72b6\u51b5\u52a8\u6001\u7684\u9009\u62e9\u6700\u9002\u5408\u7684\u4f5c\u4e3aleader.Kafka\u5e76\u4e0d\u662f\u4f7f\u7528\u8fd9\u79cd\u65b9\u6cd5\u3002<\/div>\n<div>Kafka\u52a8\u6001\u7ef4\u62a4\u4e86\u4e00\u4e2a\u540c\u6b65\u72b6\u6001\u7684\u526f\u672c\u7684\u96c6\u5408\uff08a set of in-sync replicas\uff09\uff0c\u7b80\u79f0ISR\uff0c\u5728\u8fd9\u4e2a\u96c6\u5408\u4e2d\u7684\u8282\u70b9\u90fd\u662f\u548cleader\u4fdd\u6301\u9ad8\u5ea6\u4e00\u81f4\u7684\uff0c\u4efb\u4f55\u4e00\u6761\u6d88\u606f\u5fc5\u987b\u88ab\u8fd9\u4e2a\u96c6\u5408\u4e2d\u7684\u6bcf\u4e2a\u8282\u70b9\u8bfb\u53d6\u5e76\u8ffd\u52a0\u5230\u65e5\u5fd7\u4e2d\u4e86\uff0c\u624d\u56de\u901a\u77e5\u5916\u90e8\u8fd9\u4e2a\u6d88\u606f\u5df2\u7ecf\u88ab\u63d0\u4ea4\u4e86\u3002\u56e0\u6b64\u8fd9\u4e2a\u96c6\u5408\u4e2d\u7684\u4efb\u4f55\u4e00\u4e2a\u8282\u70b9\u968f\u65f6\u90fd\u53ef\u4ee5\u88ab\u9009\u4e3aleader.ISR\u5728ZooKeeper\u4e2d\u7ef4\u62a4\u3002ISR\u4e2d\u6709f+1\u4e2a\u8282\u70b9\uff0c\u5c31\u53ef\u4ee5\u5141\u8bb8\u5728f\u4e2a\u8282\u70b9down\u6389\u7684\u60c5\u51b5\u4e0b\u4e0d\u4f1a\u4e22\u5931\u6d88\u606f\u5e76\u6b63\u5e38\u63d0\u4f9b\u670d\u3002ISR\u7684\u6210\u5458\u662f\u52a8\u6001\u7684\uff0c\u5982\u679c\u4e00\u4e2a\u8282\u70b9\u88ab\u6dd8\u6c70\u4e86\uff0c\u5f53\u5b83\u91cd\u65b0\u8fbe\u5230\u201c\u540c\u6b65\u4e2d\u201d\u7684\u72b6\u6001\u65f6\uff0c\u4ed6\u53ef\u4ee5\u91cd\u65b0\u52a0\u5165ISR.\u8fd9\u79cdleader\u7684\u9009\u62e9\u65b9\u5f0f\u662f\u975e\u5e38\u5feb\u901f\u7684\uff0c\u9002\u5408kafka\u7684\u5e94\u7528\u573a\u666f\u3002<\/div>\n<div>\u4e00\u4e2a\u90aa\u6076\u7684\u60f3\u6cd5\uff1a\u5982\u679c\u6240\u6709\u8282\u70b9\u90fddown\u6389\u4e86\u600e\u4e48\u529e\uff1fKafka\u5bf9\u4e8e\u6570\u636e\u4e0d\u4f1a\u4e22\u5931\u7684\u4fdd\u8bc1\uff0c\u662f\u57fa\u4e8e\u81f3\u5c11\u4e00\u4e2a\u8282\u70b9\u662f\u5b58\u6d3b\u7684\uff0c\u4e00\u65e6\u6240\u6709\u8282\u70b9\u90fddown\u4e86\uff0c\u8fd9\u4e2a\u5c31\u4e0d\u80fd\u4fdd\u8bc1\u4e86\u3002<\/div>\n<div>\u5b9e\u9645\u5e94\u7528\u4e2d\uff0c\u5f53\u6240\u6709\u7684\u526f\u672c\u90fddown\u6389\u65f6\uff0c\u5fc5\u987b\u53ca\u65f6\u4f5c\u51fa\u53cd\u5e94\u3002\u53ef\u4ee5\u6709\u4ee5\u4e0b\u4e24\u79cd\u9009\u62e9:<\/div>\n<div>1. \u7b49\u5f85ISR\u4e2d\u7684\u4efb\u4f55\u4e00\u4e2a\u8282\u70b9\u6062\u590d\u5e76\u62c5\u4efbleader\u3002<\/div>\n<div>2. \u9009\u62e9\u6240\u6709\u8282\u70b9\u4e2d\uff08\u4e0d\u53ea\u662fISR\uff09\u7b2c\u4e00\u4e2a\u6062\u590d\u7684\u8282\u70b9\u4f5c\u4e3aleader.<\/div>\n<div>\u8fd9\u662f\u4e00\u4e2a\u5728\u53ef\u7528\u6027\u548c\u8fde\u7eed\u6027\u4e4b\u95f4\u7684\u6743\u8861\u3002\u5982\u679c\u7b49\u5f85ISR\u4e2d\u7684\u8282\u70b9\u6062\u590d\uff0c\u4e00\u65e6ISR\u4e2d\u7684\u8282\u70b9\u8d77\u4e0d\u8d77\u6765\u6216\u8005\u6570\u636e\u90fd\u662f\u4e86\uff0c\u90a3\u96c6\u7fa4\u5c31\u6c38\u8fdc\u6062\u590d\u4e0d\u4e86\u4e86\u3002\u5982\u679c\u7b49\u5f85ISR\u610f\u5916\u7684\u8282\u70b9\u6062\u590d\uff0c\u8fd9\u4e2a\u8282\u70b9\u7684\u6570\u636e\u5c31\u4f1a\u88ab\u4f5c\u4e3a\u7ebf\u4e0a\u6570\u636e\uff0c\u6709\u53ef\u80fd\u548c\u771f\u5b9e\u7684\u6570\u636e\u6709\u6240\u51fa\u5165\uff0c\u56e0\u4e3a\u6709\u4e9b\u6570\u636e\u5b83\u53ef\u80fd\u8fd8\u6ca1\u540c\u6b65\u5230\u3002Kafka\u76ee\u524d\u9009\u62e9\u4e86\u7b2c\u4e8c\u79cd\u7b56\u7565\uff0c\u5728\u672a\u6765\u7684\u7248\u672c\u4e2d\u5c06\u4f7f\u8fd9\u4e2a\u7b56\u7565\u7684\u9009\u62e9\u53ef\u914d\u7f6e\uff0c\u53ef\u4ee5\u6839\u636e\u573a\u666f\u7075\u6d3b\u7684\u9009\u62e9\u3002<\/div>\n<div>\u8fd9\u79cd\u7a98\u5883\u4e0d\u53eaKafka\u4f1a\u9047\u5230\uff0c\u51e0\u4e4e\u6240\u6709\u7684\u5206\u5e03\u5f0f\u6570\u636e\u7cfb\u7edf\u90fd\u4f1a\u9047\u5230\u3002<\/div>\n<div><\/div>\n<div>10.\u526f\u672c\u7ba1\u7406<\/div>\n<div>\u4ee5\u4e0a\u4ec5\u4ec5\u4ee5\u4e00\u4e2atopic\u4e00\u4e2a\u5206\u533a\u4e3a\u4f8b\u5b50\u8fdb\u884c\u4e86\u8ba8\u8bba\uff0c\u4f46\u5b9e\u9645\u4e0a\u4e00\u4e2aKafka\u5c06\u4f1a\u7ba1\u7406\u6210\u5343\u4e0a\u4e07\u7684topic\u5206\u533a.Kafka\u5c3d\u91cf\u7684\u4f7f\u6240\u6709\u5206\u533a\u5747\u5300\u7684\u5206\u5e03\u5230\u96c6\u7fa4\u6240\u6709\u7684\u8282\u70b9\u4e0a\u800c\u4e0d\u662f\u96c6\u4e2d\u5728\u67d0\u4e9b\u8282\u70b9\u4e0a\uff0c\u53e6\u5916\u4e3b\u4ece\u5173\u7cfb\u4e5f\u5c3d\u91cf\u5747\u8861\u8fd9\u6837\u6bcf\u4e2a\u51e0\u70b9\u90fd\u4f1a\u62c5\u4efb\u4e00\u5b9a\u6bd4\u4f8b\u7684\u5206\u533a\u7684leader.<\/div>\n<div>\u4f18\u5316leader\u7684\u9009\u62e9\u8fc7\u7a0b\u4e5f\u662f\u5f88\u91cd\u8981\u7684\uff0c\u5b83\u51b3\u5b9a\u4e86\u7cfb\u7edf\u53d1\u751f\u6545\u969c\u65f6\u7684\u7a7a\u7a97\u671f\u6709\u591a\u4e45\u3002Kafka\u9009\u62e9\u4e00\u4e2a\u8282\u70b9\u4f5c\u4e3a\u201ccontroller\u201d,\u5f53\u53d1\u73b0\u6709\u8282\u70b9down\u6389\u7684\u65f6\u5019\u5b83\u8d1f\u8d23\u5728\u6e38\u6cf3\u5206\u533a\u7684\u6240\u6709\u8282\u70b9\u4e2d\u9009\u62e9\u65b0\u7684leader,\u8fd9\u4f7f\u5f97Kafka\u53ef\u4ee5\u6279\u91cf\u7684\u9ad8\u6548\u7684\u7ba1\u7406\u6240\u6709\u5206\u533a\u8282\u70b9\u7684\u4e3b\u4ece\u5173\u7cfb\u3002\u5982\u679ccontroller down\u6389\u4e86\uff0c\u6d3b\u7740\u7684\u8282\u70b9\u4e2d\u7684\u4e00\u4e2a\u4f1a\u5907\u5207\u6362\u4e3a\u65b0\u7684controller.<\/div>\n<div><\/div>\n<div>11.Leader\u4e0e\u526f\u672c\u540c\u6b65<\/div>\n<div>\u5bf9\u4e8e\u67d0\u4e2a\u5206\u533a\u6765\u8bf4\uff0c\u4fdd\u5b58\u6b63\u5206\u533a\u7684&#8221;broker&#8221;\u4e3a\u8be5\u5206\u533a\u7684&#8221;leader&#8221;\uff0c\u4fdd\u5b58\u5907\u4efd\u5206\u533a\u7684&#8221;broker&#8221;\u4e3a\u8be5\u5206\u533a\u7684&#8221;follower&#8221;\u3002\u5907\u4efd\u5206\u533a\u4f1a\u5b8c\u5168\u590d\u5236\u6b63\u5206\u533a\u7684\u6d88\u606f\uff0c\u5305\u62ec\u6d88\u606f\u7684\u7f16\u53f7\u7b49\u9644\u52a0\u5c5e\u6027\u503c\u3002\u4e3a\u4e86\u4fdd\u6301\u6b63\u5206\u533a\u548c\u5907\u4efd\u5206\u533a\u7684\u5185\u5bb9\u4e00\u81f4\uff0cKafka\u91c7\u53d6\u7684\u65b9\u6848\u662f\u5728\u4fdd\u5b58\u5907\u4efd\u5206\u533a\u7684&#8221;broker&#8221;\u4e0a\u5f00\u542f\u4e00\u4e2a\u6d88\u8d39\u8005\u8fdb\u7a0b\u8fdb\u884c\u6d88\u8d39\uff0c\u4ece\u800c\u4f7f\u5f97\u6b63\u5206\u533a\u7684\u5185\u5bb9\u4e0e\u5907\u4efd\u5206\u533a\u7684\u5185\u5bb9\u4fdd\u6301\u4e00\u81f4\u3002\u4e00\u822c\u60c5\u51b5\u4e0b\uff0c\u4e00\u4e2a\u5206\u533a\u6709\u4e00\u4e2a\u201c\u6b63\u5206\u533a\u201d\u548c\u96f6\u5230\u591a\u4e2a\u201c\u5907\u4efd\u5206\u533a\u201d\u3002\u53ef\u4ee5\u914d\u7f6e\u201c\u6b63\u5206\u533a+\u5907\u4efd\u5206\u533a\u201d\u7684\u603b\u6570\u91cf\uff0c\u5173\u4e8e\u8fd9\u4e2a\u914d\u7f6e\uff0c\u4e0d\u540c\u4e3b\u9898\u53ef\u4ee5\u6709\u4e0d\u540c\u7684\u914d\u7f6e\u503c\u3002\u6ce8\u610f\uff0c\u751f\u4ea7\u8005\uff0c\u6d88\u8d39\u8005\u53ea\u4e0e\u4fdd\u5b58\u6b63\u5206\u533a\u7684&#8221;leader&#8221;\u8fdb\u884c\u901a\u4fe1\u3002<\/div>\n<div><\/div>\n<div>Kafka\u5141\u8bb8topic\u7684\u5206\u533a\u62e5\u6709\u82e5\u5e72\u526f\u672c\uff0c\u8fd9\u4e2a\u6570\u91cf\u662f\u53ef\u4ee5\u914d\u7f6e\u7684\uff0c\u4f60\u53ef\u4ee5\u4e3a\u6bcf\u4e2atopic\u914d\u7f6e\u526f\u672c\u7684\u6570\u91cf\u3002Kafka\u4f1a\u81ea\u52a8\u5728\u6bcf\u4e2a\u526f\u672c\u4e0a\u5907\u4efd\u6570\u636e\uff0c\u6240\u4ee5\u5f53\u4e00\u4e2a\u8282\u70b9down\u6389\u65f6\u6570\u636e\u4f9d\u7136\u662f\u53ef\u7528\u7684\u3002<\/div>\n<div>Kafka\u7684\u526f\u672c\u529f\u80fd\u4e0d\u662f\u5fc5\u987b\u7684\uff0c\u4f60\u53ef\u4ee5\u914d\u7f6e\u53ea\u6709\u4e00\u4e2a\u526f\u672c\uff0c\u8fd9\u6837\u5176\u5b9e\u5c31\u76f8\u5f53\u4e8e\u53ea\u6709\u4e00\u4efd\u6570\u636e\u3002<\/div>\n<div>\u521b\u5efa\u526f\u672c\u7684\u5355\u4f4d\u662ftopic\u7684\u5206\u533a\uff0c\u6bcf\u4e2a\u5206\u533a\u90fd\u6709\u4e00\u4e2aleader\u548c\u96f6\u6216\u591a\u4e2afollowers.\u6240\u6709\u7684\u8bfb\u5199\u64cd\u4f5c\u90fd\u7531leader\u5904\u7406\uff0c\u4e00\u822c\u5206\u533a\u7684\u6570\u91cf\u90fd\u6bd4broker\u7684\u6570\u91cf\u591a\u7684\u591a\uff0c\u5404\u5206\u533a\u7684leader\u5747\u5300\u7684\u5206\u5e03\u5728brokers\u4e2d\u3002\u6240\u6709\u7684followers\u90fd\u590d\u5236leader\u7684\u65e5\u5fd7\uff0c\u65e5\u5fd7\u4e2d\u7684\u6d88\u606f\u548c\u987a\u5e8f\u90fd\u548cleader\u4e2d\u7684\u4e00\u81f4\u3002followers\u5411\u666e\u901a\u7684consumer\u90a3\u6837\u4eceleader\u90a3\u91cc\u62c9\u53d6\u6d88\u606f\u5e76\u4fdd\u5b58\u5728\u81ea\u5df1\u7684\u65e5\u5fd7\u6587\u4ef6\u4e2d\u3002<\/div>\n<div>\u8bb8\u591a\u5206\u5e03\u5f0f\u7684\u6d88\u606f\u7cfb\u7edf\u81ea\u52a8\u7684\u5904\u7406\u5931\u8d25\u7684\u8bf7\u6c42\uff0c\u5b83\u4eec\u5bf9\u4e00\u4e2a\u8282\u70b9\u662f\u5426\u7740\uff08alive\uff09\u201d\u6709\u7740\u6e05\u6670\u7684\u5b9a\u4e49\u3002Kafka\u5224\u65ad\u4e00\u4e2a\u8282\u70b9\u662f\u5426\u6d3b\u7740\u6709\u4e24\u4e2a\u6761\u4ef6\uff1a<\/div>\n<div>1. \u8282\u70b9\u5fc5\u987b\u53ef\u4ee5\u7ef4\u62a4\u548cZooKeeper\u7684\u8fde\u63a5\uff0cZookeeper\u901a\u8fc7\u5fc3\u8df3\u673a\u5236\u68c0\u67e5\u6bcf\u4e2a\u8282\u70b9\u7684\u8fde\u63a5\u3002<\/div>\n<div>2. \u5982\u679c\u8282\u70b9\u662f\u4e2afollower,\u4ed6\u5fc5\u987b\u80fd\u53ca\u65f6\u7684\u540c\u6b65leader\u7684\u5199\u64cd\u4f5c\uff0c\u5ef6\u65f6\u4e0d\u80fd\u592a\u4e45\u3002<\/div>\n<div>\u7b26\u5408\u4ee5\u4e0a\u6761\u4ef6\u7684\u8282\u70b9\u51c6\u786e\u7684\u8bf4\u5e94\u8be5\u662f\u201c\u540c\u6b65\u4e2d\u7684\uff08in sync\uff09\u201d\uff0c\u800c\u4e0d\u662f\u6a21\u7cca\u7684\u8bf4\u662f\u201c\u6d3b\u7740\u7684\u201d\u6216\u662f\u201c\u5931\u8d25\u7684\u201d\u3002Leader\u4f1a\u8ffd\u8e2a\u6240\u6709\u201c\u540c\u6b65\u4e2d\u201d\u7684\u8282\u70b9\uff0c\u4e00\u65e6\u4e00\u4e2adown\u6389\u4e86\uff0c\u6216\u662f\u5361\u4f4f\u4e86\uff0c\u6216\u662f\u5ef6\u65f6\u592a\u4e45\uff0cleader\u5c31\u4f1a\u628a\u5b83\u79fb\u9664\u3002\u81f3\u4e8e\u5ef6\u65f6\u591a\u4e45\u7b97\u662f\u201c\u592a\u4e45\u201d\uff0c\u662f\u7531\u53c2\u6570replica.lag.max.messages\u51b3\u5b9a\u7684\uff0c\u600e\u6837\u7b97\u662f\u5361\u4f4f\u4e86\uff0c\u600e\u662f\u7531\u53c2\u6570replica.lag.time.max.ms\u51b3\u5b9a\u7684\u3002<\/div>\n<div>\u53ea\u6709\u5f53\u6d88\u606f\u88ab\u6240\u6709\u7684\u526f\u672c\u52a0\u5165\u5230\u65e5\u5fd7\u4e2d\u65f6\uff0c\u624d\u7b97\u662f\u201ccommitted\u201d\uff0c\u53ea\u6709committed\u7684\u6d88\u606f\u624d\u4f1a\u53d1\u9001\u7ed9consumer\uff0c\u8fd9\u6837\u5c31\u4e0d\u7528\u62c5\u5fc3\u4e00\u65e6leader down\u6389\u4e86\u6d88\u606f\u4f1a\u4e22\u5931\u3002Producer\u4e5f\u53ef\u4ee5\u9009\u62e9\u662f\u5426\u7b49\u5f85\u6d88\u606f\u88ab\u63d0\u4ea4\u7684\u901a\u77e5\uff0c\u8fd9\u4e2a\u662f\u7531\u53c2\u6570acks\u51b3\u5b9a\u7684\u3002<\/div>\n<div>Kafka\u4fdd\u8bc1\u53ea\u8981\u6709\u4e00\u4e2a\u201c\u540c\u6b65\u4e2d\u201d\u7684\u8282\u70b9\uff0c\u201ccommitted\u201d\u7684\u6d88\u606f\u5c31\u4e0d\u4f1a\u4e22\u5931\u3002<\/div>\n<div><\/div>\n<div><\/div>\n<\/div>\n<ul>\n<li>2.3 \u00a0kafka\u62d3\u6251\u7ed3\u6784\n<p><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107211822608?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/li>\n<\/ul>\n<div>\n<div>\n<div>\u00a0 \u00a0 \u00a0 \u00a0\u4e00\u4e2a\u5178\u578b\u7684Kafka\u96c6\u7fa4\u4e2d\u5305\u542b\u82e5\u5e72Producer\uff08\u53ef\u4ee5\u662fweb\u524d\u7aefFET\uff0c\u6216\u8005\u662f\u670d\u52a1\u5668\u65e5\u5fd7\u7b49\uff09\uff0c\u82e5\u5e72broker\uff08Kafka\u652f\u6301\u6c34\u5e73\u6269\u5c55\uff0c\u4e00\u822cbroker\u6570\u91cf\u8d8a\u591a\uff0c\u96c6\u7fa4\u541e\u5410\u7387\u8d8a\u9ad8\uff09\uff0c\u82e5\u5e72ConsumerGroup\uff0c\u4ee5\u53ca\u4e00\u4e2aZookeeper\u96c6\u7fa4\u3002Kafka\u901a\u8fc7Zookeeper\u7ba1\u7406Kafka\u96c6\u7fa4\u914d\u7f6e\uff1a\u9009\u4e3eKafka broker\u7684leader\uff0c\u4ee5\u53ca\u5728Consumer Group\u53d1\u751f\u53d8\u5316\u65f6\u8fdb\u884crebalance\uff0c\u56e0\u4e3aconsumer\u6d88\u8d39kafka topic\u7684partition\u7684offsite\u4fe1\u606f\u662f\u5b58\u5728Zookeeper\u7684\u3002Producer\u4f7f\u7528push\u6a21\u5f0f\u5c06\u6d88\u606f\u53d1\u5e03\u5230broker\uff0cConsumer\u4f7f\u7528pull\u6a21\u5f0f\u4ecebroker\u8ba2\u9605\u5e76\u6d88\u8d39\u6d88\u606f\u3002<\/div>\n<div><\/div>\n<\/div>\n<\/div>\n<p>\u5206\u6790\u8fc7\u7a0b\u5206\u4e3a\u4ee5\u4e0b4\u4e2a\u6b65\u9aa4\uff1a<\/p>\n<ul>\n<li>topic\u4e2dpartition\u5b58\u50a8\u5206\u5e03<\/li>\n<li>partiton\u4e2d\u6587\u4ef6\u5b58\u50a8\u65b9\u5f0f (partition\u5728linux\u670d\u52a1\u5668\u4e0a\u5c31\u662f\u4e00\u4e2a\u76ee\u5f55\uff08\u6587\u4ef6\u5939\uff09)<\/li>\n<li>partiton\u4e2dsegment\u6587\u4ef6\u5b58\u50a8\u7ed3\u6784<\/li>\n<li>\u5728partition\u4e2d\u5982\u4f55\u901a\u8fc7offset\u67e5\u627emessage<\/li>\n<\/ul>\n<p>\u901a\u8fc7\u4e0a\u8ff04\u8fc7\u7a0b\u8be6\u7ec6\u5206\u6790\uff0c\u6211\u4eec\u5c31\u53ef\u4ee5\u6e05\u695a\u8ba4\u8bc6\u5230kafka\u6587\u4ef6\u5b58\u50a8\u673a\u5236\u7684\u5965\u79d8\u3002<\/p>\n<\/div>\n<div><\/div>\n<div>2.3 topic\u4e2dpartition\u5b58\u50a8\u5206\u5e03<\/div>\n<div>\n<p>\u5047\u8bbe\u5b9e\u9a8c\u73af\u5883\u4e2dKafka\u96c6\u7fa4\u53ea\u6709\u4e00\u4e2abroker\uff0cxxx\/message-folder\u4e3a\u6570\u636e\u6587\u4ef6\u5b58\u50a8\u6839\u76ee\u5f55\uff0c\u5728Kafka broker\u4e2dserver.properties\u6587\u4ef6\u914d\u7f6e(\u53c2\u6570log.dirs=xxx\/message-folder)\uff0c\u4f8b\u5982\u521b\u5efa2\u4e2atopic\u540d \u79f0\u5206\u522b\u4e3areport_push\u3001launch_info, partitions\u6570\u91cf\u90fd\u4e3apartitions=4<\/p>\n<p>\u5b58\u50a8\u8def\u5f84\u548c\u76ee\u5f55\u89c4\u5219\u4e3a\uff1a<\/p>\n<p>xxx\/message-folder<\/p>\n<\/div>\n<div>\u00a0 |&#8211;report_push-0<br \/>\n|&#8211;report_push-1<br \/>\n|&#8211;report_push-2<br \/>\n|&#8211;report_push-3<br \/>\n|&#8211;launch_info-0<br \/>\n|&#8211;launch_info-1<br \/>\n|&#8211;launch_info-2<br \/>\n|&#8211;launch_info-3<\/div>\n<div><\/div>\n<div>\n<div>\u5728Kafka\u6587\u4ef6\u5b58\u50a8\u4e2d\uff0c\u540c\u4e00\u4e2atopic\u4e0b\u6709\u591a\u4e2a\u4e0d\u540cpartition\uff0c\u6bcf\u4e2apartition\u4e3a\u4e00\u4e2a\u76ee\u5f55\uff0cpartiton\u547d\u540d\u89c4\u5219\u4e3atopic\u540d\u79f0+\u6709\u5e8f\u5e8f\u53f7\uff0c\u7b2c\u4e00\u4e2apartiton\u5e8f\u53f7\u4ece0\u5f00\u59cb\uff0c\u5e8f\u53f7\u6700\u5927\u503c\u4e3apartitions\u6570\u91cf\u51cf1\u3002<\/div>\n<\/div>\n<div>\u6d88\u606f\u53d1\u9001\u65f6\u90fd\u88ab\u53d1\u9001\u5230\u4e00\u4e2atopic\uff0c\u5176\u672c\u8d28\u5c31\u662f\u4e00\u4e2a\u76ee\u5f55\uff0c\u800ctopic\u7531\u662f\u7531\u4e00\u4e9bPartition\u7ec4\u6210,\u5176\u7ec4\u7ec7\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\uff1a<\/div>\n<div><\/div>\n<div>\u6211\u4eec\u53ef\u4ee5\u770b\u5230\uff0cPartition\u662f\u4e00\u4e2aQueue\u7684\u7ed3\u6784\uff0c\u6bcf\u4e2aPartition\u4e2d\u7684\u6d88\u606f\u90fd\u662f\u6709\u5e8f\u7684\uff0c\u751f\u4ea7\u7684\u6d88\u606f\u88ab\u4e0d\u65ad\u8ffd\u52a0\u5230Partition\u4e0a\uff0c\u5176\u4e2d\u7684\u6bcf\u4e00\u4e2a\u6d88\u606f\u90fd\u88ab\u8d4b\u4e88\u4e86\u4e00\u4e2a\u552f\u4e00\u7684offset\u503c\u3002<\/div>\n<div><\/div>\n<div>Kafka\u96c6\u7fa4\u4f1a\u4fdd\u5b58\u6240\u6709\u7684\u6d88\u606f\uff0c\u4e0d\u7ba1\u6d88\u606f\u6709\u6ca1\u6709\u88ab\u6d88\u8d39\uff1b\u6211\u4eec\u53ef\u4ee5\u8bbe\u5b9a\u6d88\u606f\u7684\u8fc7\u671f\u65f6\u95f4\uff0c\u53ea\u6709\u8fc7\u671f\u7684\u6570\u636e\u624d\u4f1a\u88ab\u81ea\u52a8\u6e05\u9664\u4ee5\u91ca\u653e\u78c1\u76d8\u7a7a\u95f4\u3002\u6bd4\u5982\u6211\u4eec\u8bbe\u7f6e\u6d88\u606f\u8fc7\u671f\u65f6\u95f4\u4e3a2\u5929\uff0c\u90a3\u4e48\u8fd92\u5929\u5185\u7684\u6240\u6709\u6d88\u606f\u90fd\u4f1a\u88ab\u4fdd\u5b58\u5230\u96c6\u7fa4\u4e2d\uff0c\u6570\u636e\u53ea\u6709\u8d85\u8fc7\u4e86\u4e24\u5929\u624d\u4f1a\u88ab\u6e05\u9664\u3002<\/div>\n<div><\/div>\n<div>Kafka\u53ea\u7ef4\u62a4\u5728Partition\u4e2d\u7684offset\u503c\uff0c\u56e0\u4e3a\u8fd9\u4e2aoffsite\u6807\u8bc6\u7740\u8fd9\u4e2apartition\u7684message\u6d88\u8d39\u5230\u54ea\u6761\u4e86\u3002Consumer\u6bcf\u6d88\u8d39\u4e00\u4e2a\u6d88\u606f\uff0coffset\u5c31\u4f1a\u52a01\u3002\u5176\u5b9e\u6d88\u606f\u7684\u72b6\u6001\u5b8c\u5168\u662f\u7531Consumer\u63a7\u5236\u7684\uff0cConsumer\u53ef\u4ee5\u8ddf\u8e2a\u548c\u91cd\u8bbe\u8fd9\u4e2aoffset\u503c\uff0c\u8fd9\u6837\u7684\u8bddConsumer\u5c31\u53ef\u4ee5\u8bfb\u53d6\u4efb\u610f\u4f4d\u7f6e\u7684\u6d88\u606f\u3002<\/div>\n<div><\/div>\n<div>\u628a\u6d88\u606f\u65e5\u5fd7\u4ee5Partition\u7684\u5f62\u5f0f\u5b58\u653e\u6709\u591a\u91cd\u8003\u8651\uff0c\u7b2c\u4e00\uff0c\u65b9\u4fbf\u5728\u96c6\u7fa4\u4e2d\u6269\u5c55\uff0c\u6bcf\u4e2aPartition\u53ef\u4ee5\u901a\u8fc7\u8c03\u6574\u4ee5\u9002\u5e94\u5b83\u6240\u5728\u7684\u673a\u5668\uff0c\u800c\u4e00\u4e2atopic\u53c8\u53ef\u4ee5\u6709\u591a\u4e2aPartition\u7ec4\u6210\uff0c\u56e0\u6b64\u6574\u4e2a\u96c6\u7fa4\u5c31\u53ef\u4ee5\u9002\u5e94\u4efb\u610f\u5927\u5c0f\u7684\u6570\u636e\u4e86\uff1b\u7b2c\u4e8c\u5c31\u662f\u53ef\u4ee5\u63d0\u9ad8\u5e76\u53d1\uff0c\u56e0\u4e3a\u53ef\u4ee5\u4ee5Partition\u4e3a\u5355\u4f4d\u8bfb\u5199\u4e86\u3002<\/div>\n<div><\/div>\n<div>\u901a\u8fc7\u4e0a\u9762\u4ecb\u7ecd\u7684\u6211\u4eec\u53ef\u4ee5\u77e5\u9053\uff0ckafka\u4e2d\u7684\u6570\u636e\u662f\u6301\u4e45\u5316\u7684\u5e76\u4e14\u80fd\u591f\u5bb9\u9519\u7684\u3002Kafka\u5141\u8bb8\u7528\u6237\u4e3a\u6bcf\u4e2atopic\u8bbe\u7f6e\u526f\u672c\u6570\u91cf\uff0c\u526f\u672c\u6570\u91cf\u51b3\u5b9a\u4e86\u6709\u51e0\u4e2abroker\u6765\u5b58\u653e\u5199\u5165\u7684\u6570\u636e\u3002\u5982\u679c\u4f60\u7684\u526f\u672c\u6570\u91cf\u8bbe\u7f6e\u4e3a3\uff0c\u90a3\u4e48\u4e00\u4efd\u6570\u636e\u5c31\u4f1a\u88ab\u5b58\u653e\u57283\u53f0\u4e0d\u540c\u7684\u673a\u5668\u4e0a\uff0c\u90a3\u4e48\u5c31\u5141\u8bb8\u67092\u4e2a\u673a\u5668\u5931\u8d25\u3002\u4e00\u822c\u63a8\u8350\u526f\u672c\u6570\u91cf\u81f3\u5c11\u4e3a2\uff0c\u8fd9\u6837\u5c31\u53ef\u4ee5\u4fdd\u8bc1\u589e\u51cf\u3001\u91cd\u542f\u673a\u5668\u65f6\u4e0d\u4f1a\u5f71\u54cd\u5230\u6570\u636e\u6d88\u8d39\u3002\u5982\u679c\u5bf9\u6570\u636e\u6301\u4e45\u5316\u6709\u66f4\u9ad8\u7684\u8981\u6c42\uff0c\u53ef\u4ee5\u628a\u526f\u672c\u6570\u91cf\u8bbe\u7f6e\u4e3a3\u6216\u8005\u66f4\u591a\u3002<\/div>\n<div><\/div>\n<div>Kafka\u4e2d\u7684topic\u662f\u4ee5partition\u7684\u5f62\u5f0f\u5b58\u653e\u7684\uff0c\u6bcf\u4e00\u4e2atopic\u90fd\u53ef\u4ee5\u8bbe\u7f6e\u5b83\u7684partition\u6570\u91cf\uff0cPartition\u7684\u6570\u91cf\u51b3\u5b9a\u4e86\u7ec4\u6210topic\u7684message\u7684\u6570\u91cf\u3002Producer\u5728\u751f\u4ea7\u6570\u636e\u65f6\uff0c\u4f1a\u6309\u7167\u4e00\u5b9a\u89c4\u5219\uff08\u8fd9\u4e2a\u89c4\u5219\u662f\u53ef\u4ee5\u81ea\u5b9a\u4e49\u7684\uff09\u628a\u6d88\u606f\u53d1\u5e03\u5230topic\u7684\u5404\u4e2apartition\u4e2d\u3002\u4e0a\u9762\u5c06\u7684\u526f\u672c\u90fd\u662f\u4ee5partition\u4e3a\u5355\u4f4d\u7684\uff0c\u4e0d\u8fc7\u53ea\u6709\u4e00\u4e2apartition\u7684\u526f\u672c\u4f1a\u88ab\u9009\u4e3e\u6210leader\u4f5c\u4e3a\u8bfb\u5199\u7528\u3002<\/div>\n<div><\/div>\n<div>\u5173\u4e8e\u5982\u4f55\u8bbe\u7f6epartition\u503c\u9700\u8981\u8003\u8651\u7684\u56e0\u7d20\u3002\u4e00\u4e2apartition\u53ea\u80fd\u88ab\u4e00\u4e2a\u6d88\u8d39\u8005\u6d88\u8d39\uff08\u4e00\u4e2a\u6d88\u8d39\u8005\u53ef\u4ee5\u540c\u65f6\u6d88\u8d39\u591a\u4e2apartition\uff09\uff0c\u56e0\u6b64\uff0c\u5982\u679c\u8bbe\u7f6e\u7684partition\u7684\u6570\u91cf\u5c0f\u4e8econsumer\u7684\u6570\u91cf\uff0c\u5c31\u4f1a\u6709\u6d88\u8d39\u8005\u6d88\u8d39\u4e0d\u5230\u6570\u636e\u3002\u6240\u4ee5\uff0c\u63a8\u8350partition\u7684\u6570\u91cf\u4e00\u5b9a\u8981\u5927\u4e8e\u540c\u65f6\u8fd0\u884c\u7684consumer\u7684\u6570\u91cf\u3002\u53e6\u5916\u4e00\u65b9\u9762\uff0c\u5efa\u8baepartition\u7684\u6570\u91cf\u5927\u4e8e\u96c6\u7fa4broker\u7684\u6570\u91cf\uff0c\u8fd9\u6837leader partition\u5c31\u53ef\u4ee5\u5747\u5300\u7684\u5206\u5e03\u5728\u5404\u4e2abroker\u4e2d\uff0c\u6700\u7ec8\u4f7f\u5f97\u96c6\u7fa4\u8d1f\u8f7d\u5747\u8861\u3002\u5728Cloudera,\u6bcf\u4e2atopic\u90fd\u6709\u4e0a\u767e\u4e2apartition\u3002\u9700\u8981\u6ce8\u610f\u7684\u662f\uff0ckafka\u9700\u8981\u4e3a\u6bcf\u4e2apartition\u5206\u914d\u4e00\u4e9b\u5185\u5b58\u6765\u7f13\u5b58\u6d88\u606f\u6570\u636e\uff0c\u5982\u679cpartition\u6570\u91cf\u8d8a\u5927\uff0c\u5c31\u8981\u4e3akafka\u5206\u914d\u66f4\u5927\u7684heap space\u3002<\/div>\n<div>\n<div>2.4 partiton\u4e2d\u6587\u4ef6\u5b58\u50a8\u65b9\u5f0f<\/div>\n<div><\/div>\n<ul>\n<li>\u6bcf\u4e2apartion(\u76ee\u5f55)\u76f8\u5f53\u4e8e\u4e00\u4e2a\u5de8\u578b\u6587\u4ef6\u88ab\u5e73\u5747\u5206\u914d\u5230\u591a\u4e2a\u5927\u5c0f\u76f8\u7b49segment(\u6bb5)\u6570\u636e\u6587\u4ef6\u4e2d\u3002\u4f46\u6bcf\u4e2a\u6bb5segment file\u6d88\u606f\u6570\u91cf\u4e0d\u4e00\u5b9a\u76f8\u7b49\uff0c\u8fd9\u79cd\u7279\u6027\u65b9\u4fbfold segment file\u5feb\u901f\u88ab\u5220\u9664\u3002<\/li>\n<li>\u6bcf\u4e2apartiton\u53ea\u9700\u8981\u652f\u6301\u987a\u5e8f\u8bfb\u5199\u5c31\u884c\u4e86\uff0csegment\u6587\u4ef6\u751f\u547d\u5468\u671f\u7531\u670d\u52a1\u7aef\u914d\u7f6e\u53c2\u6570\u51b3\u5b9a\u3002<\/li>\n<\/ul>\n<p>\u8fd9\u6837\u505a\u7684\u597d\u5904\u5c31\u662f\u80fd\u5feb\u901f\u5220\u9664\u65e0\u7528\u6587\u4ef6\uff0c\u6709\u6548\u63d0\u9ad8\u78c1\u76d8\u5229\u7528\u7387\u3002<\/p>\n<\/div>\n<div><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107212036116?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<div>\n<div>2.5 partiton\u4e2dsegment\u6587\u4ef6\u5b58\u50a8\u7ed3\u6784<\/div>\n<div>\n<div>producer\u53d1message\u5230\u67d0\u4e2atopic\uff0cmessage\u4f1a\u88ab\u5747\u5300\u7684\u5206\u5e03\u5230\u591a\u4e2apartition\u4e0a\uff08\u968f\u673a\u6216\u6839\u636e\u7528\u6237\u6307\u5b9a\u7684\u56de\u8c03\u51fd\u6570\u8fdb\u884c\u5206\u5e03\uff09\uff0ckafka broker\u6536\u5230message\u5f80\u5bf9\u5e94partition\u7684\u6700\u540e\u4e00\u4e2asegment\u4e0a\u6dfb\u52a0\u8be5\u6d88\u606f\uff0c\u5f53\u67d0\u4e2asegment\u4e0a\u7684\u6d88\u606f\u6761\u6570\u8fbe\u5230\u914d\u7f6e\u503c\u6216\u6d88\u606f\u53d1\u5e03\u65f6\u95f4\u8d85\u8fc7\u9608\u503c\u65f6\uff0csegment\u4e0a\u7684\u6d88\u606f\u4f1a\u88abflush\u5230\u78c1\u76d8\uff0c\u53ea\u6709flush\u5230\u78c1\u76d8\u4e0a\u7684\u6d88\u606fconsumer\u624d\u80fd\u6d88\u8d39\uff0csegment\u8fbe\u5230\u4e00\u5b9a\u7684\u5927\u5c0f\u540e\u5c06\u4e0d\u4f1a\u518d\u5f80\u8be5segment\u5199\u6570\u636e\uff0cbroker\u4f1a\u521b\u5efa\u65b0\u7684segment\u3002<\/div>\n<div><\/div>\n<div>\u6bcf\u4e2apart\u5728\u5185\u5b58\u4e2d\u5bf9\u5e94\u4e00\u4e2aindex\uff0c\u8bb0\u5f55\u6bcf\u4e2asegment\u4e2d\u7684\u7b2c\u4e00\u6761\u6d88\u606f\u504f\u79fb\u3002<\/div>\n<\/div>\n<ul>\n<li>segment file\u7ec4\u6210\uff1a\u75312\u5927\u90e8\u5206\u7ec4\u6210\uff0c\u5206\u522b\u4e3aindex file\u548cdata file\uff0c\u6b642\u4e2a\u6587\u4ef6\u4e00\u4e00\u5bf9\u5e94\uff0c\u6210\u5bf9\u51fa\u73b0\uff0c\u540e\u7f00&#8221;.index&#8221;\u548c\u201c.log\u201d\u5206\u522b\u8868\u793a\u4e3asegment\u7d22\u5f15\u6587\u4ef6\u3001\u6570\u636e\u6587\u4ef6.<\/li>\n<li>segment\u6587\u4ef6\u547d\u540d\u89c4\u5219\uff1apartion\u5168\u5c40\u7684\u7b2c\u4e00\u4e2asegment\u4ece0\u5f00\u59cb\uff0c\u540e\u7eed\u6bcf\u4e2asegment\u6587\u4ef6\u540d\u4e3a\u4e0a\u4e00\u4e2a\u5168\u5c40partion\u7684\u6700\u5927offset(\u504f\u79fbmessage\u6570)\u3002\u6570\u503c\u6700\u5927\u4e3a64\u4f4dlong\u5927\u5c0f\uff0c19\u4f4d\u6570\u5b57\u5b57\u7b26\u957f\u5ea6\uff0c\u6ca1\u6709\u6570\u5b57\u75280\u586b\u5145\u3002<\/li>\n<\/ul>\n<div>\n<div><\/div>\n<div>\u6bcf\u4e2asegment\u4e2d\u5b58\u50a8\u5f88\u591a\u6761\u6d88\u606f\uff0c\u6d88\u606fid\u7531\u5176\u903b\u8f91\u4f4d\u7f6e\u51b3\u5b9a\uff0c\u5373\u4ece\u6d88\u606fid\u53ef\u76f4\u63a5\u5b9a\u4f4d\u5230\u6d88\u606f\u7684\u5b58\u50a8\u4f4d\u7f6e\uff0c\u907f\u514did\u5230\u4f4d\u7f6e\u7684\u989d\u5916\u6620\u5c04\u3002<\/div>\n<\/div>\n<div>\n<p>\u4e0b\u9762\u6587\u4ef6\u5217\u8868\u662f\u7b14\u8005\u5728Kafka broker\u4e0a\u505a\u7684\u4e00\u4e2a\u5b9e\u9a8c\uff0c\u521b\u5efa\u4e00\u4e2atopicXXX\u5305\u542b1 partition\uff0c\u8bbe\u7f6e\u6bcf\u4e2asegment\u5927\u5c0f\u4e3a500MB,\u5e76\u542f\u52a8producer\u5411Kafka broker\u5199\u5165\u5927\u91cf\u6570\u636e,\u5982\u4e0b\u56fe2\u6240\u793asegment\u6587\u4ef6\u5217\u8868\u5f62\u8c61\u8bf4\u660e\u4e86\u4e0a\u8ff02\u4e2a\u89c4\u5219\uff1a<\/p>\n<\/div>\n<div><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107212205785?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<div>\n<p>\u4ee5\u4e0a\u8ff0\u56fe2\u4e2d\u4e00\u5bf9segment file\u6587\u4ef6\u4e3a\u4f8b\uff0c\u8bf4\u660esegment\u4e2dindex&lt;\u2014-&gt;data file\u5bf9\u5e94\u5173\u7cfb\u7269\u7406\u7ed3\u6784\u5982\u4e0b\uff1a<\/p>\n<\/div>\n<div><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107212224538?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<p>\u4e0a\u8ff0\u56fe3\u4e2d\u7d22\u5f15\u6587\u4ef6\u5b58\u50a8\u5927\u91cf\u5143\u6570\u636e\uff0c\u6570\u636e\u6587\u4ef6\u5b58\u50a8\u5927\u91cf\u6d88\u606f\uff0c\u7d22\u5f15\u6587\u4ef6\u4e2d\u5143\u6570\u636e\u6307\u5411\u5bf9\u5e94\u6570\u636e\u6587\u4ef6\u4e2dmessage\u7684\u7269\u7406\u504f\u79fb\u5730\u5740\u3002\u5176\u4e2d\u4ee5\u7d22\u5f15\u6587\u4ef6\u4e2d \u5143\u6570\u636e3,497\u4e3a\u4f8b\uff0c\u4f9d\u6b21\u5728\u6570\u636e\u6587\u4ef6\u4e2d\u8868\u793a\u7b2c3\u4e2amessage(\u5728\u5168\u5c40partiton\u8868\u793a\u7b2c368772\u4e2amessage)\u3001\u4ee5\u53ca\u8be5\u6d88\u606f\u7684\u7269\u7406\u504f\u79fb \u5730\u5740\u4e3a497\u3002<\/p>\n<div>\n<p>\u4ece\u4e0a\u8ff0\u56fe3\u4e86\u89e3\u5230segment data file\u7531\u8bb8\u591amessage\u7ec4\u6210\uff0c\u4e0b\u9762\u8be6\u7ec6\u8bf4\u660emessage\u7269\u7406\u7ed3\u6784\u5982\u4e0b\uff1a<\/p>\n<\/div>\n<div><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107212325100?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<h3><a name=\"t1\"><\/a>\u53c2\u6570\u8bf4\u660e\uff1a<\/h3>\n<div class=\"table-box\">\n<table>\n<tbody>\n<tr>\n<th>\u5173\u952e\u5b57<\/th>\n<th>\u89e3\u91ca\u8bf4\u660e<\/th>\n<\/tr>\n<\/tbody>\n<tbody>\n<tr>\n<td>8 byte offset<\/td>\n<td>\u5728parition(\u5206\u533a)\u5185\u7684\u6bcf\u6761\u6d88\u606f\u90fd\u6709\u4e00\u4e2a\u6709\u5e8f\u7684id\u53f7\uff0c\u8fd9\u4e2aid\u53f7\u88ab\u79f0\u4e3a\u504f\u79fb(offset),\u5b83\u53ef\u4ee5\u552f\u4e00\u786e\u5b9a\u6bcf\u6761\u6d88\u606f\u5728parition(\u5206\u533a)\u5185\u7684\u4f4d\u7f6e\u3002\u5373offset\u8868\u793apartiion\u7684\u7b2c\u591a\u5c11message<\/td>\n<\/tr>\n<tr>\n<td>4 byte message size<\/td>\n<td>message\u5927\u5c0f<\/td>\n<\/tr>\n<tr>\n<td>4 byte CRC32<\/td>\n<td>\u7528crc32\u6821\u9a8cmessage<\/td>\n<\/tr>\n<tr>\n<td>1 byte \u201cmagic&#8221;<\/td>\n<td>\u8868\u793a\u672c\u6b21\u53d1\u5e03Kafka\u670d\u52a1\u7a0b\u5e8f\u534f\u8bae\u7248\u672c\u53f7<\/td>\n<\/tr>\n<tr>\n<td>1 byte \u201cattributes&#8221;<\/td>\n<td>\u8868\u793a\u4e3a\u72ec\u7acb\u7248\u672c\u3001\u6216\u6807\u8bc6\u538b\u7f29\u7c7b\u578b\u3001\u6216\u7f16\u7801\u7c7b\u578b\u3002<\/td>\n<\/tr>\n<tr>\n<td>4 byte key length<\/td>\n<td>\u8868\u793akey\u7684\u957f\u5ea6,\u5f53key\u4e3a-1\u65f6\uff0cK byte key\u5b57\u6bb5\u4e0d\u586b<\/td>\n<\/tr>\n<tr>\n<td>K byte key<\/td>\n<td>\u53ef\u9009<\/td>\n<\/tr>\n<tr>\n<td>value bytes payload<\/td>\n<td>\u8868\u793a\u5b9e\u9645\u6d88\u606f\u6570\u636e\u3002<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<div><\/div>\n<div>\n<div>2.6 \u5728partition\u4e2d\u5982\u4f55\u901a\u8fc7offset\u67e5\u627emessage<\/div>\n<p>\u4f8b\u5982\u8bfb\u53d6offset=368776\u7684message\uff0c\u9700\u8981\u901a\u8fc7\u4e0b\u97622\u4e2a\u6b65\u9aa4\u67e5\u627e\u3002<\/p>\n<ul>\n<li>\u7b2c\u4e00\u6b65\u67e5\u627esegment file\n<p>\u4e0a\u8ff0\u56fe2\u4e3a\u4f8b\uff0c\u5176\u4e2d00000000000000000000.index\u8868\u793a\u6700\u5f00\u59cb\u7684\u6587\u4ef6\uff0c\u8d77\u59cb\u504f\u79fb\u91cf(offset)\u4e3a0.\u7b2c\u4e8c\u4e2a\u6587\u4ef6 00000000000000368769.index\u7684\u6d88\u606f\u91cf\u8d77\u59cb\u504f\u79fb\u91cf\u4e3a368770 = 368769 + 1.\u540c\u6837\uff0c\u7b2c\u4e09\u4e2a\u6587\u4ef600000000000000737337.index\u7684\u8d77\u59cb\u504f\u79fb\u91cf\u4e3a737338=737337 + 1\uff0c\u5176\u4ed6\u540e\u7eed\u6587\u4ef6\u4f9d\u6b21\u7c7b\u63a8\uff0c\u4ee5\u8d77\u59cb\u504f\u79fb\u91cf\u547d\u540d\u5e76\u6392\u5e8f\u8fd9\u4e9b\u6587\u4ef6\uff0c\u53ea\u8981\u6839\u636eoffset **\u4e8c\u5206\u67e5\u627e**\u6587\u4ef6\u5217\u8868\uff0c\u5c31\u53ef\u4ee5\u5feb\u901f\u5b9a\u4f4d\u5230\u5177\u4f53\u6587\u4ef6\u3002<\/p>\n<p>\u5f53offset=368776\u65f6\u5b9a\u4f4d\u523000000000000000368769.index|log<\/li>\n<li>\u7b2c\u4e8c\u6b65\u901a\u8fc7segment file\u67e5\u627emessage\u901a\u8fc7\u7b2c\u4e00\u6b65\u5b9a\u4f4d\u5230segment file\uff0c\u5f53offset=368776\u65f6\uff0c\u4f9d\u6b21\u5b9a\u4f4d\u523000000000000000368769.index\u7684\u5143\u6570\u636e\u7269\u7406\u4f4d\u7f6e\u548c 00000000000000368769.log\u7684\u7269\u7406\u504f\u79fb\u5730\u5740\uff0c\u7136\u540e\u518d\u901a\u8fc700000000000000368769.log\u987a\u5e8f\u67e5\u627e\u76f4\u5230 offset=368776\u4e3a\u6b62\u3002<\/li>\n<\/ul>\n<div>\n<div>segment index file\u91c7\u53d6\u7a00\u758f\u7d22\u5f15\u5b58\u50a8\u65b9\u5f0f\uff0c\u5b83\u51cf\u5c11\u7d22\u5f15\u6587\u4ef6\u5927\u5c0f\uff0c\u901a\u8fc7mmap\u53ef\u4ee5\u76f4\u63a5\u5185\u5b58\u64cd\u4f5c\uff0c\u7a00\u758f\u7d22\u5f15\u4e3a\u6570\u636e\u6587\u4ef6\u7684\u6bcf\u4e2a\u5bf9\u5e94message\u8bbe\u7f6e\u4e00\u4e2a\u5143\u6570\u636e\u6307\u9488,\u5b83 \u6bd4\u7a20\u5bc6\u7d22\u5f15\u8282\u7701\u4e86\u66f4\u591a\u7684\u5b58\u50a8\u7a7a\u95f4\uff0c\u4f46\u67e5\u627e\u8d77\u6765\u9700\u8981\u6d88\u8017\u66f4\u591a\u7684\u65f6\u95f4\u3002<\/div>\n<div><\/div>\n<div>kafka\u4f1a\u8bb0\u5f55offset\u5230zk\u4e2d\u3002\u4f46\u662f\uff0czk client api\u5bf9zk\u7684\u9891\u7e41\u5199\u5165\u662f\u4e00\u4e2a\u4f4e\u6548\u7684\u64cd\u4f5c\u30020.8.2 kafka\u5f15\u5165\u4e86native offset storage\uff0c\u5c06offset\u7ba1\u7406\u4ecezk\u79fb\u51fa\uff0c\u5e76\u4e14\u53ef\u4ee5\u505a\u5230\u6c34\u5e73\u6269\u5c55\u3002\u5176\u539f\u7406\u5c31\u662f\u5229\u7528\u4e86kafka\u7684compacted topic\uff0coffset\u4ee5consumer group,topic\u4e0epartion\u7684\u7ec4\u5408\u4f5c\u4e3akey\u76f4\u63a5\u63d0\u4ea4\u5230compacted topic\u4e2d\u3002\u540c\u65f6Kafka\u53c8\u5728\u5185\u5b58\u4e2d\u7ef4\u62a4\u4e86\u7684\u4e09\u5143\u7ec4\u6765\u7ef4\u62a4\u6700\u65b0\u7684offset\u4fe1\u606f\uff0cconsumer\u6765\u53d6\u6700\u65b0offset\u4fe1\u606f\u7684\u65f6\u5019\u76f4\u63a5\u5185\u5b58\u91cc\u62ff\u5373\u53ef\u3002\u5f53\u7136\uff0ckafka\u5141\u8bb8\u4f60\u5feb\u901f\u7684checkpoint\u6700\u65b0\u7684offset\u4fe1\u606f\u5230\u78c1\u76d8\u4e0a\u3002<\/div>\n<\/div>\n<\/div>\n<div><\/div>\n<div>3.Partition Replication\u539f\u5219<\/div>\n<div>\n<p>Kafka\u9ad8\u6548\u6587\u4ef6\u5b58\u50a8\u8bbe\u8ba1\u7279\u70b9<\/p>\n<ul>\n<li>Kafka\u628atopic\u4e2d\u4e00\u4e2aparition\u5927\u6587\u4ef6\u5206\u6210\u591a\u4e2a\u5c0f\u6587\u4ef6\u6bb5\uff0c\u901a\u8fc7\u591a\u4e2a\u5c0f\u6587\u4ef6\u6bb5\uff0c\u5c31\u5bb9\u6613\u5b9a\u671f\u6e05\u9664\u6216\u5220\u9664\u5df2\u7ecf\u6d88\u8d39\u5b8c\u6587\u4ef6\uff0c\u51cf\u5c11\u78c1\u76d8\u5360\u7528\u3002<\/li>\n<li>\u901a\u8fc7\u7d22\u5f15\u4fe1\u606f\u53ef\u4ee5\u5feb\u901f\u5b9a\u4f4dmessage\u548c\u786e\u5b9aresponse\u7684\u6700\u5927\u5927\u5c0f\u3002<\/li>\n<li>\u901a\u8fc7index\u5143\u6570\u636e\u5168\u90e8\u6620\u5c04\u5230memory\uff0c\u53ef\u4ee5\u907f\u514dsegment file\u7684IO\u78c1\u76d8\u64cd\u4f5c\u3002<\/li>\n<li>\u901a\u8fc7\u7d22\u5f15\u6587\u4ef6\u7a00\u758f\u5b58\u50a8\uff0c\u53ef\u4ee5\u5927\u5e45\u964d\u4f4eindex\u6587\u4ef6\u5143\u6570\u636e\u5360\u7528\u7a7a\u95f4\u5927\u5c0f\u3002<\/li>\n<\/ul>\n<div>\n<div><\/div>\n<div><\/div>\n<div>\n<p>1. Kafka\u96c6\u7fa4partition\u00a0replication\u9ed8\u8ba4\u81ea\u52a8\u5206\u914d\u5206\u6790<\/p>\n<p>\u4e0b\u9762\u4ee5\u4e00\u4e2aKafka\u96c6\u7fa4\u4e2d4\u4e2aBroker\u4e3e\u4f8b\uff0c\u521b\u5efa1\u4e2atopic\u5305\u542b4\u4e2aPartition\uff0c2 Replication\uff1b\u6570\u636eProducer\u6d41\u52a8\u5982\u56fe\u6240\u793a\uff1a<\/p>\n<p>(1)<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107212450867?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/p>\n<p>(2)\u5f53\u96c6\u7fa4\u4e2d\u65b0\u589e2\u8282\u70b9\uff0cPartition\u589e\u52a0\u52306\u4e2a\u65f6\u5206\u5e03\u60c5\u51b5\u5982\u4e0b\uff1a<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107212535915?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/p>\n<p>\u526f\u672c\u5206\u914d\u903b\u8f91\u89c4\u5219\u5982\u4e0b\uff1a<\/p>\n<ul>\n<li>\u5728Kafka\u96c6\u7fa4\u4e2d\uff0c\u6bcf\u4e2aBroker\u90fd\u6709\u5747\u7b49\u5206\u914dPartition\u7684Leader\u673a\u4f1a\u3002<\/li>\n<li>\u4e0a\u8ff0\u56feBroker Partition\u4e2d\uff0c\u7bad\u5934\u6307\u5411\u4e3a\u526f\u672c\uff0c\u4ee5Partition-0\u4e3a\u4f8b:broker1\u4e2dparition-0\u4e3aLeader\uff0cBroker2\u4e2dPartition-0\u4e3a\u526f\u672c\u3002<\/li>\n<li>\u4e0a\u8ff0\u56fe\u79cd\u6bcf\u4e2aBroker(\u6309\u7167BrokerId\u6709\u5e8f)\u4f9d\u6b21\u5206\u914d\u4e3bPartition,\u4e0b\u4e00\u4e2aBroker\u4e3a\u526f\u672c\uff0c\u5982\u6b64\u5faa\u73af\u8fed\u4ee3\u5206\u914d\uff0c\u591a\u526f\u672c\u90fd\u9075\u5faa\u6b64\u89c4\u5219\u3002<\/li>\n<\/ul>\n<div><\/div>\n<div>\u526f\u672c\u5206\u914d\u7b97\u6cd5\u5982\u4e0b\uff1a<\/div>\n<div>\n<ul>\n<li>\u5c06\u6240\u6709N Broker\u548c\u5f85\u5206\u914d\u7684i\u4e2aPartition\u6392\u5e8f.<\/li>\n<li>\u5c06\u7b2ci\u4e2aPartition\u5206\u914d\u5230\u7b2c(i mod n)\u4e2aBroker\u4e0a.<\/li>\n<li>\u5c06\u7b2ci\u4e2aPartition\u7684\u7b2cj\u4e2a\u526f\u672c\u5206\u914d\u5230\u7b2c((i + j) mod n)\u4e2aBroker\u4e0a.<\/li>\n<\/ul>\n<div><\/div>\n<div>4.Kafka Broker\u4e00\u4e9b\u7279\u6027<\/div>\n<div>\n<div>4.1 \u65e0\u72b6\u6001\u7684Kafka Broker :<\/div>\n<div>1. Broker\u6ca1\u6709\u526f\u672c\u673a\u5236\uff0c\u4e00\u65e6broker\u5b95\u673a\uff0c\u8be5broker\u7684\u6d88\u606f\u5c06\u90fd\u4e0d\u53ef\u7528\u3002<\/div>\n<div>2. Broker\u4e0d\u4fdd\u5b58\u8ba2\u9605\u8005\u7684\u72b6\u6001\uff0c\u7531\u8ba2\u9605\u8005\u81ea\u5df1\u4fdd\u5b58\u3002<\/div>\n<div>3. \u65e0\u72b6\u6001\u5bfc\u81f4\u6d88\u606f\u7684\u5220\u9664\u6210\u4e3a\u96be\u9898\uff08\u53ef\u80fd\u5220\u9664\u7684\u6d88\u606f\u6b63\u5728\u88ab\u8ba2\u9605\uff09\uff0ckafka\u91c7\u7528\u57fa\u4e8e\u65f6\u95f4\u7684SLA(\u670d\u52a1\u6c34\u5e73\u4fdd\u8bc1)\uff0c\u6d88\u606f\u4fdd\u5b58\u4e00\u5b9a\u65f6\u95f4\uff08\u901a\u5e38\u4e3a7\u5929\uff09\u540e\u4f1a\u88ab\u5220\u9664\u3002<\/div>\n<div>4. \u6d88\u606f\u8ba2\u9605\u8005\u53ef\u4ee5rewind back\u5230\u4efb\u610f\u4f4d\u7f6e\u91cd\u65b0\u8fdb\u884c\u6d88\u8d39\uff0c\u5f53\u8ba2\u9605\u8005\u6545\u969c\u65f6\uff0c\u53ef\u4ee5\u9009\u62e9\u6700\u5c0f\u7684offset\u8fdb\u884c\u91cd\u65b0\u8bfb\u53d6\u6d88\u8d39\u6d88\u606f\u3002<\/div>\n<div><\/div>\n<div>4.2 message\u7684\u4ea4\u4ed8\u4e0e\u751f\u547d\u5468\u671f \uff1a<\/div>\n<div>1. \u4e0d\u662f\u4e25\u683c\u7684JMS\uff0c \u56e0\u6b64kafka\u5bf9\u6d88\u606f\u7684\u91cd\u590d\u3001\u4e22\u5931\u3001\u9519\u8bef\u4ee5\u53ca\u987a\u5e8f\u578b\u6ca1\u6709\u4e25\u683c\u7684\u8981\u6c42\u3002\uff08\u8fd9\u662f\u4e0eAMQ\u6700\u5927\u7684\u533a\u522b\uff09<\/div>\n<div>2. kafka\u63d0\u4f9bat-least-once delivery,\u5373\u5f53consumer\u5b95\u673a\u540e\uff0c\u6709\u4e9b\u6d88\u606f\u53ef\u80fd\u4f1a\u88ab\u91cd\u590ddelivery\u3002<\/div>\n<div>3. \u56e0\u6bcf\u4e2apartition\u53ea\u4f1a\u88abconsumer group\u5185\u7684\u4e00\u4e2aconsumer\u6d88\u8d39\uff0c\u6545kafka\u4fdd\u8bc1\u6bcf\u4e2apartition\u5185\u7684\u6d88\u606f\u4f1a\u88ab\u987a\u5e8f\u7684\u8ba2\u9605\u3002<\/div>\n<div>4. Kafka\u4e3a\u6bcf\u6761\u6d88\u606f\u4e3a\u6bcf\u6761\u6d88\u606f\u8ba1\u7b97CRC\u6821\u9a8c\uff0c\u7528\u4e8e\u9519\u8bef\u68c0\u6d4b\uff0ccrc\u6821\u9a8c\u4e0d\u901a\u8fc7\u7684\u6d88\u606f\u4f1a\u76f4\u63a5\u88ab\u4e22\u5f03\u6389\u3002<\/div>\n<div><\/div>\n<div>4.3 \u538b\u7f29<\/div>\n<div><\/div>\n<div>Kafka\u652f\u6301\u4ee5\u96c6\u5408\uff08batch\uff09\u4e3a\u5355\u4f4d\u53d1\u9001\u6d88\u606f\uff0c\u5728\u6b64\u57fa\u7840\u4e0a\uff0cKafka\u8fd8\u652f\u6301\u5bf9\u6d88\u606f\u96c6\u5408\u8fdb\u884c\u538b\u7f29\uff0cProducer\u7aef\u53ef\u4ee5\u901a\u8fc7GZIP\u6216Snappy\u683c\u5f0f\u5bf9\u6d88\u606f\u96c6\u5408\u8fdb\u884c\u538b\u7f29\u3002Producer\u7aef\u8fdb\u884c\u538b\u7f29\u4e4b\u540e\uff0c\u5728Consumer\u7aef\u9700\u8fdb\u884c\u89e3\u538b\u3002\u538b\u7f29\u7684\u597d\u5904\u5c31\u662f\u51cf\u5c11\u4f20\u8f93\u7684\u6570\u636e\u91cf\uff0c\u51cf\u8f7b\u5bf9\u7f51\u7edc\u4f20\u8f93\u7684\u538b\u529b\uff0c\u5728\u5bf9\u5927\u6570\u636e\u5904\u7406\u4e0a\uff0c\u74f6\u9888\u5f80\u5f80\u4f53\u73b0\u5728\u7f51\u7edc\u4e0a\u800c\u4e0d\u662fCPU\u3002<\/div>\n<div><\/div>\n<div>\u90a3\u4e48\u5982\u4f55\u533a\u5206\u6d88\u606f\u662f\u538b\u7f29\u7684\u8fd8\u662f\u672a\u538b\u7f29\u7684\u5462\uff0cKafka\u5728\u6d88\u606f\u5934\u90e8\u6dfb\u52a0\u4e86\u4e00\u4e2a\u63cf\u8ff0\u538b\u7f29\u5c5e\u6027\u5b57\u8282\uff0c\u8fd9\u4e2a\u5b57\u8282\u7684\u540e\u4e24\u4f4d\u8868\u793a\u6d88\u606f\u7684\u538b\u7f29\u91c7\u7528\u7684\u7f16\u7801\uff0c\u5982\u679c\u540e\u4e24\u4f4d\u4e3a0\uff0c\u5219\u8868\u793a\u6d88\u606f\u672a\u88ab\u538b\u7f29\u3002<\/div>\n<div><\/div>\n<div>4.4 \u6d88\u606f\u53ef\u9760\u6027<\/div>\n<div><\/div>\n<div>\u5728\u6d88\u606f\u7cfb\u7edf\u4e2d\uff0c\u4fdd\u8bc1\u6d88\u606f\u5728\u751f\u4ea7\u548c\u6d88\u8d39\u8fc7\u7a0b\u4e2d\u7684\u53ef\u9760\u6027\u662f\u5341\u5206\u91cd\u8981\u7684\uff0c\u5728\u5b9e\u9645\u6d88\u606f\u4f20\u9012\u8fc7\u7a0b\u4e2d\uff0c\u53ef\u80fd\u4f1a\u51fa\u73b0\u5982\u4e0b\u4e09\u4e2d\u60c5\u51b5\uff1a<\/div>\n<div><\/div>\n<div>&#8211; \u4e00\u4e2a\u6d88\u606f\u53d1\u9001\u5931\u8d25<\/div>\n<div><\/div>\n<div>&#8211; \u4e00\u4e2a\u6d88\u606f\u88ab\u53d1\u9001\u591a\u6b21<\/div>\n<div><\/div>\n<div>&#8211; \u6700\u7406\u60f3\u7684\u60c5\u51b5\uff1aexactly-once ,\u4e00\u4e2a\u6d88\u606f\u53d1\u9001\u6210\u529f\u4e14\u4ec5\u53d1\u9001\u4e86\u4e00\u6b21<\/div>\n<div><\/div>\n<div>\u6709\u8bb8\u591a\u7cfb\u7edf\u58f0\u79f0\u5b83\u4eec\u5b9e\u73b0\u4e86exactly-once\uff0c\u4f46\u662f\u5b83\u4eec\u5176\u5b9e\u5ffd\u7565\u4e86\u751f\u4ea7\u8005\u6216\u6d88\u8d39\u8005\u5728\u751f\u4ea7\u548c\u6d88\u8d39\u8fc7\u7a0b\u4e2d\u6709\u53ef\u80fd\u5931\u8d25\u7684\u60c5\u51b5\u3002\u6bd4\u5982\u867d\u7136\u4e00\u4e2aProducer\u6210\u529f\u53d1\u9001\u4e00\u4e2a\u6d88\u606f\uff0c\u4f46\u662f\u6d88\u606f\u5728\u53d1\u9001\u9014\u4e2d\u4e22\u5931\uff0c\u6216\u8005\u6210\u529f\u53d1\u9001\u5230broker\uff0c\u4e5f\u88abconsumer\u6210\u529f\u53d6\u8d70\uff0c\u4f46\u662f\u8fd9\u4e2aconsumer\u5728\u5904\u7406\u53d6\u8fc7\u6765\u7684\u6d88\u606f\u65f6\u5931\u8d25\u4e86\u3002<\/div>\n<div><\/div>\n<div>\u4eceProducer\u7aef\u770b\uff1aKafka\u662f\u8fd9\u4e48\u5904\u7406\u7684\uff0c\u5f53\u4e00\u4e2a\u6d88\u606f\u88ab\u53d1\u9001\u540e\uff0cProducer\u4f1a\u7b49\u5f85broker\u6210\u529f\u63a5\u6536\u5230\u6d88\u606f\u7684\u53cd\u9988\uff08\u53ef\u901a\u8fc7\u53c2\u6570\u63a7\u5236\u7b49\u5f85\u65f6\u95f4\uff09\uff0c\u5982\u679c\u6d88\u606f\u5728\u9014\u4e2d\u4e22\u5931\u6216\u662f\u5176\u4e2d\u4e00\u4e2abroker\u6302\u6389\uff0cProducer\u4f1a\u91cd\u65b0\u53d1\u9001\uff08\u6211\u4eec\u77e5\u9053Kafka\u6709\u5907\u4efd\u673a\u5236\uff0c\u53ef\u4ee5\u901a\u8fc7\u53c2\u6570\u63a7\u5236\u662f\u5426\u7b49\u5f85\u6240\u6709\u5907\u4efd\u8282\u70b9\u90fd\u6536\u5230\u6d88\u606f\uff09\u3002<\/div>\n<div><\/div>\n<div>\u4eceConsumer\u7aef\u770b\uff1a\u524d\u9762\u8bb2\u5230\u8fc7partition\uff0cbroker\u7aef\u8bb0\u5f55\u4e86partition\u4e2d\u7684\u4e00\u4e2aoffset\u503c\uff0c\u8fd9\u4e2a\u503c\u6307\u5411Consumer\u4e0b\u4e00\u4e2a\u5373\u5c06\u6d88\u8d39message\u3002\u5f53Consumer\u6536\u5230\u4e86\u6d88\u606f\uff0c\u4f46\u5374\u5728\u5904\u7406\u8fc7\u7a0b\u4e2d\u6302\u6389\uff0c\u6b64\u65f6Consumer\u53ef\u4ee5\u901a\u8fc7\u8fd9\u4e2aoffset\u503c\u91cd\u65b0\u627e\u5230\u4e0a\u4e00\u4e2a\u6d88\u606f\u518d\u8fdb\u884c\u5904\u7406\u3002Consumer\u8fd8\u6709\u6743\u9650\u63a7\u5236\u8fd9\u4e2aoffset\u503c\uff0c\u5bf9\u6301\u4e45\u5316\u5230broker\u7aef\u7684\u6d88\u606f\u505a\u4efb\u610f\u5904\u7406\u3002<\/div>\n<div><\/div>\n<div>4.5 \u5907\u4efd\u673a\u5236<\/div>\n<div><\/div>\n<div>\u5907\u4efd\u673a\u5236\u662fKafka0.8\u7248\u672c\u7684\u65b0\u7279\u6027\uff0c\u5907\u4efd\u673a\u5236\u7684\u51fa\u73b0\u5927\u5927\u63d0\u9ad8\u4e86Kafka\u96c6\u7fa4\u7684\u53ef\u9760\u6027\u3001\u7a33\u5b9a\u6027\u3002\u6709\u4e86\u5907\u4efd\u673a\u5236\u540e\uff0cKafka\u5141\u8bb8\u96c6\u7fa4\u4e2d\u7684\u8282\u70b9\u6302\u6389\u540e\u800c\u4e0d\u5f71\u54cd\u6574\u4e2a\u96c6\u7fa4\u5de5\u4f5c\u3002\u4e00\u4e2a\u5907\u4efd\u6570\u91cf\u4e3an\u7684\u96c6\u7fa4\u5141\u8bb8n-1\u4e2a\u8282\u70b9\u5931\u8d25\u3002\u5728\u6240\u6709\u5907\u4efd\u8282\u70b9\u4e2d\uff0c\u6709\u4e00\u4e2a\u8282\u70b9\u4f5c\u4e3alead\u8282\u70b9\uff0c\u8fd9\u4e2a\u8282\u70b9\u4fdd\u5b58\u4e86\u5176\u5b83\u5907\u4efd\u8282\u70b9\u5217\u8868\uff0c\u5e76\u7ef4\u6301\u5404\u4e2a\u5907\u4efd\u95f4\u7684\u72b6\u4f53\u540c\u6b65\u3002\u4e0b\u9762\u8fd9\u5e45\u56fe\u89e3\u91ca\u4e86Kafka\u7684\u5907\u4efd\u673a\u5236:<\/div>\n<div><\/div>\n<div><\/div>\n<div><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107212704151?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<div>4.6 Kafka\u9ad8\u6548\u6027\u76f8\u5173\u8bbe\u8ba1<\/div>\n<div><\/div>\n<div>4.6.1 \u6d88\u606f\u7684\u6301\u4e45\u5316<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div>Kafka\u9ad8\u5ea6\u4f9d\u8d56\u6587\u4ef6\u7cfb\u7edf\u6765\u5b58\u50a8\u548c\u7f13\u5b58\u6d88\u606f(AMQ\u7684nessage\u662f\u6301\u4e45\u5316\u5230mysql\u6570\u636e\u5e93\u4e2d\u7684)\uff0c\u56e0\u4e3a\u4e00\u822c\u7684\u4eba\u8ba4\u4e3a\u78c1\u76d8\u662f\u7f13\u6162\u7684\uff0c\u8fd9\u5bfc\u81f4\u4eba\u4eec\u5bf9\u6301\u4e45\u5316\u7ed3\u6784\u5177\u6709\u7ade\u4e89\u6027\u6301\u6000\u7591\u6001\u5ea6\u3002\u5176\u5b9e\uff0c\u78c1\u76d8\u7684\u5feb\u6216\u8005\u6162\uff0c\u8fd9\u51b3\u5b9a\u4e8e\u6211\u4eec\u5982\u4f55\u4f7f\u7528\u78c1\u76d8\u3002\u56e0\u4e3a\u78c1\u76d8\u7ebf\u6027\u5199\u7684\u901f\u5ea6\u8fdc\u8fdc\u5927\u4e8e\u968f\u673a\u5199\u3002\u7ebf\u6027\u8bfb\u5199\u5728\u5927\u591a\u6570\u5e94\u7528\u573a\u666f\u4e0b\u662f\u53ef\u4ee5\u9884\u6d4b\u7684\u3002<\/div>\n<div>\n<div>\n<div>4.6.2 \u5e38\u6570\u65f6\u95f4\u6027\u80fd\u4fdd\u8bc1<\/div>\n<div>\u6bcf\u4e2aTopic\u7684Partition\u7684\u662f\u4e00\u4e2a\u5927\u6587\u4ef6\u5939\uff0c\u91cc\u9762\u6709\u65e0\u6570\u4e2a\u5c0f\u6587\u4ef6\u5939segment\uff0c\u4f46partition\u662f\u4e00\u4e2a\u961f\u5217\uff0c\u961f\u5217\u4e2d\u7684\u5143\u7d20\u662fsegment,\u6d88\u8d39\u7684\u65f6\u5019\u5148\u4ece\u7b2c0\u4e2asegment\u5f00\u59cb\u6d88\u8d39\uff0c\u65b0\u6765message\u5b58\u5728\u6700\u540e\u4e00\u4e2a\u6d88\u606f\u961f\u5217\u4e2d\u3002\u5bf9\u4e8esegment\u4e5f\u662f\u5bf9\u961f\u5217\uff0c\u961f\u5217\u5143\u7d20\u662fmessage,\u6709\u5bf9\u5e94\u7684offsite\u6807\u8bc6\u662f\u54ea\u4e2amessage\u3002\u6d88\u8d39\u7684\u65f6\u5019\u5148\u4ece\u8fd9\u4e2asegment\u7684\u7b2c\u4e00\u4e2amessage\u5f00\u59cb\u6d88\u8d39\uff0c\u65b0\u6765\u7684message\u5b58\u5728segment\u7684\u6700\u540e\u3002<\/div>\n<div><\/div>\n<div>\u6d88\u606f\u7cfb\u7edf\u7684\u6301\u4e45\u5316\u961f\u5217\u53ef\u4ee5\u6784\u5efa\u5728\u5bf9\u4e00\u4e2a\u6587\u4ef6\u7684\u8bfb\u548c\u8ffd\u52a0\u4e0a\uff0c\u5c31\u50cf\u4e00\u822c\u60c5\u51b5\u4e0b\u7684\u65e5\u5fd7\u89e3\u51b3\u65b9\u6848\u3002\u5b83\u6709\u4e00\u4e2a\u4f18\u70b9\uff0c\u6240\u6709\u7684\u64cd\u4f5c\u90fd\u662f\u5e38\u6570\u65f6\u95f4\uff0c\u5e76\u4e14\u8bfb\u5199\u4e4b\u95f4\u4e0d\u4f1a\u76f8\u4e92\u963b\u585e\u3002\u8fd9\u79cd\u8bbe\u8ba1\u5177\u6709\u6781\u5927\u7684\u6027\u80fd\u4f18\u52bf\uff1a\u6700\u7ec8\u7cfb\u7edf\u6027\u80fd\u548c\u6570\u636e\u5927\u5c0f\u5b8c\u5168\u65e0\u5173\uff0c\u670d\u52a1\u5668\u53ef\u4ee5\u5145\u5206\u5229\u7528\u5ec9\u4ef7\u7684\u786c\u76d8\u6765\u63d0\u4f9b\u9ad8\u6548\u7684\u6d88\u606f\u670d\u52a1\u3002<\/div>\n<div><\/div>\n<div>\u4e8b\u5b9e\u4e0a\u8fd8\u6709\u4e00\u70b9\uff0c\u78c1\u76d8\u7a7a\u95f4\u7684\u65e0\u9650\u589e\u5927\u800c\u4e0d\u5f71\u54cd\u6027\u80fd\u8fd9\u70b9\uff0c\u610f\u5473\u7740\u6211\u4eec\u53ef\u4ee5\u63d0\u4f9b\u4e00\u822c\u6d88\u606f\u7cfb\u7edf\u65e0\u6cd5\u63d0\u4f9b\u7684\u7279\u6027\u3002\u6bd4\u5982\u8bf4\uff0c\u6d88\u606f\u88ab\u6d88\u8d39\u540e\u4e0d\u662f\u7acb\u9a6c\u88ab\u5220\u9664\uff0c\u6211\u4eec\u53ef\u4ee5\u5c06\u8fd9\u4e9b\u6d88\u606f\u4fdd\u7559\u4e00\u6bb5\u76f8\u5bf9\u6bd4\u8f83\u957f\u7684\u65f6\u95f4\uff08\u6bd4\u5982\u4e00\u4e2a\u661f\u671f\uff09\u3002<\/div>\n<div><\/div>\n<div>5.Kafka \u751f\u4ea7\u8005-\u6d88\u8d39\u8005<\/div>\n<\/div>\n<div>\u00a0\u00a0\u00a0\u00a0\u00a0\u6d88\u606f\u7cfb\u7edf\u901a\u5e38\u90fd\u4f1a\u7531\u751f\u4ea7\u8005\uff0c\u6d88\u8d39\u8005\uff0cBroker\u4e09\u5927\u90e8\u5206\u7ec4\u6210\uff0c\u751f\u4ea7\u8005\u4f1a\u5c06\u6d88\u606f\u5199\u5165\u5230Broker\uff0c\u6d88\u8d39\u8005\u4f1a\u4eceBroker\u4e2d\u8bfb\u53d6\u51fa\u6d88\u606f\uff0c\u4e0d\u540c\u7684MQ\u5b9e\u73b0\u7684Broker\u5b9e\u73b0\u4f1a\u6709\u6240\u4e0d\u540c\uff0c\u4e0d\u8fc7Broker\u7684\u672c\u8d28\u90fd\u662f\u8981\u8d1f\u8d23\u5c06\u6d88\u606f\u843d\u5730\u5230\u670d\u52a1\u7aef\u7684\u5b58\u50a8\u7cfb\u7edf\u4e2d\u3002\u5177\u4f53\u6b65\u9aa4\u5982\u4e0b\uff1a<\/div>\n<div>\n<ol>\n<li>\u751f\u4ea7\u8005\u5ba2\u6237\u7aef\u5e94\u7528\u7a0b\u5e8f\u4ea7\u751f\u6d88\u606f\uff1a\n<ol>\n<li>\n<div>\u5ba2\u6237\u7aef\u8fde\u63a5\u5bf9\u8c61\u5c06\u6d88\u606f\u5305\u88c5\u5230\u8bf7\u6c42\u4e2d\u53d1\u9001\u5230\u670d\u52a1\u7aef<\/div>\n<\/li>\n<li>\n<div>\u670d\u52a1\u7aef\u7684\u5165\u53e3\u4e5f\u6709\u4e00\u4e2a\u8fde\u63a5\u5bf9\u8c61\u8d1f\u8d23\u63a5\u6536\u8bf7\u6c42\uff0c\u5e76\u5c06\u6d88\u606f\u4ee5\u6587\u4ef6\u7684\u5f62\u5f0f\u5b58\u50a8\u8d77\u6765<\/div>\n<\/li>\n<li>\n<div>\u670d\u52a1\u7aef\u8fd4\u56de\u54cd\u5e94\u7ed3\u679c\u7ed9\u751f\u4ea7\u8005\u5ba2\u6237\u7aef<\/div>\n<\/li>\n<\/ol>\n<\/li>\n<li>\u6d88\u8d39\u8005\u5ba2\u6237\u7aef\u5e94\u7528\u7a0b\u5e8f\u6d88\u8d39\u6d88\u606f\uff1a\n<ol>\n<li>\n<div>\u5ba2\u6237\u7aef\u8fde\u63a5\u5bf9\u8c61\u5c06\u6d88\u8d39\u4fe1\u606f\u4e5f\u5305\u88c5\u5230\u8bf7\u6c42\u4e2d\u53d1\u9001\u7ed9\u670d\u52a1\u7aef<\/div>\n<\/li>\n<li>\n<div>\u670d\u52a1\u7aef\u4ece\u6587\u4ef6\u5b58\u50a8\u7cfb\u7edf\u4e2d\u53d6\u51fa\u6d88\u606f<\/div>\n<\/li>\n<li>\n<div>\u670d\u52a1\u7aef\u8fd4\u56de\u54cd\u5e94\u7ed3\u679c\u7ed9\u6d88\u8d39\u8005\u5ba2\u6237\u7aef<\/div>\n<\/li>\n<li>\n<div>\u5ba2\u6237\u7aef\u5c06\u54cd\u5e94\u7ed3\u679c\u8fd8\u539f\u6210\u6d88\u606f\u5e76\u5f00\u59cb\u5904\u7406\u6d88\u606f<\/div>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<div><\/div>\n<div>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u56fe4-1\u00a0\u5ba2\u6237\u7aef\u548c\u670d\u52a1\u7aef\u4ea4\u4e92<\/div>\n<div>\n<div><\/div>\n<div>5.1 \u00a0Producers<\/div>\n<div><\/div>\n<div>Producers\u76f4\u63a5\u53d1\u9001\u6d88\u606f\u5230broker\u4e0a\u7684leader partition\uff0c\u4e0d\u9700\u8981\u7ecf\u8fc7\u4efb\u4f55\u4e2d\u4ecb\u6216\u5176\u4ed6\u8def\u7531\u8f6c\u53d1\u3002\u4e3a\u4e86\u5b9e\u73b0\u8fd9\u4e2a\u7279\u6027\uff0ckafka\u96c6\u7fa4\u4e2d\u7684\u6bcf\u4e2abroker\u90fd\u53ef\u4ee5\u54cd\u5e94producer\u7684\u8bf7\u6c42\uff0c\u5e76\u8fd4\u56detopic\u7684\u4e00\u4e9b\u5143\u4fe1\u606f\uff0c\u8fd9\u4e9b\u5143\u4fe1\u606f\u5305\u62ec\u54ea\u4e9b\u673a\u5668\u662f\u5b58\u6d3b\u7684\uff0ctopic\u7684leader partition\u90fd\u5728\u54ea\uff0c\u73b0\u9636\u6bb5\u54ea\u4e9bleader partition\u662f\u53ef\u4ee5\u76f4\u63a5\u88ab\u8bbf\u95ee\u7684\u3002<\/div>\n<div><\/div>\n<div>Producer\u5ba2\u6237\u7aef\u81ea\u5df1\u63a7\u5236\u7740\u6d88\u606f\u88ab\u63a8\u9001\u5230\u54ea\u4e9bpartition\u3002\u5b9e\u73b0\u7684\u65b9\u5f0f\u53ef\u4ee5\u662f\u968f\u673a\u5206\u914d\u3001\u5b9e\u73b0\u4e00\u7c7b\u968f\u673a\u8d1f\u8f7d\u5747\u8861\u7b97\u6cd5\uff0c\u6216\u8005\u6307\u5b9a\u4e00\u4e9b\u5206\u533a\u7b97\u6cd5\u3002Kafka\u63d0\u4f9b\u4e86\u63a5\u53e3\u4f9b\u7528\u6237\u5b9e\u73b0\u81ea\u5b9a\u4e49\u7684partition\uff0c\u7528\u6237\u53ef\u4ee5\u4e3a\u6bcf\u4e2a\u6d88\u606f\u6307\u5b9a\u4e00\u4e2apartitionKey\uff0c\u901a\u8fc7\u8fd9\u4e2akey\u6765\u5b9e\u73b0\u4e00\u4e9bhash\u5206\u533a\u7b97\u6cd5\u3002\u6bd4\u5982\uff0c\u628auserid\u4f5c\u4e3apartitionkey\u7684\u8bdd\uff0c\u76f8\u540cuserid\u7684\u6d88\u606f\u5c06\u4f1a\u88ab\u63a8\u9001\u5230\u540c\u4e00\u4e2apartition\u3002<\/div>\n<div><\/div>\n<div>\u4ee5Batch\u7684\u65b9\u5f0f\u63a8\u9001\u6570\u636e\u53ef\u4ee5\u6781\u5927\u7684\u63d0\u9ad8\u5904\u7406\u6548\u7387\uff0ckafka Producer \u53ef\u4ee5\u5c06\u6d88\u606f\u5728\u5185\u5b58\u4e2d\u7d2f\u8ba1\u5230\u4e00\u5b9a\u6570\u91cf\u540e\u4f5c\u4e3a\u4e00\u4e2abatch\u53d1\u9001\u8bf7\u6c42\u3002Batch\u7684\u6570\u91cf\u5927\u5c0f\u53ef\u4ee5\u901a\u8fc7Producer\u7684\u53c2\u6570\u63a7\u5236\uff0c\u53c2\u6570\u503c\u53ef\u4ee5\u8bbe\u7f6e\u4e3a\u7d2f\u8ba1\u7684\u6d88\u606f\u7684\u6570\u91cf\uff08\u5982500\u6761\uff09\u3001\u7d2f\u8ba1\u7684\u65f6\u95f4\u95f4\u9694\uff08\u5982100ms\uff09\u6216\u8005\u7d2f\u8ba1\u7684\u6570\u636e\u5927\u5c0f(64KB)\u3002\u901a\u8fc7\u589e\u52a0batch\u7684\u5927\u5c0f\uff0c\u53ef\u4ee5\u51cf\u5c11\u7f51\u7edc\u8bf7\u6c42\u548c\u78c1\u76d8IO\u7684\u6b21\u6570\uff0c\u5f53\u7136\u5177\u4f53\u53c2\u6570\u8bbe\u7f6e\u9700\u8981\u5728\u6548\u7387\u548c\u65f6\u6548\u6027\u65b9\u9762\u505a\u4e00\u4e2a\u6743\u8861\u3002<\/div>\n<div><\/div>\n<div>Producers\u53ef\u4ee5\u5f02\u6b65\u7684\u5e76\u884c\u7684\u5411kafka\u53d1\u9001\u6d88\u606f\uff0c\u4f46\u662f\u901a\u5e38producer\u5728\u53d1\u9001\u5b8c\u6d88\u606f\u4e4b\u540e\u4f1a\u5f97\u5230\u4e00\u4e2afuture\u54cd\u5e94\uff0c\u8fd4\u56de\u7684\u662foffset\u503c\u6216\u8005\u53d1\u9001\u8fc7\u7a0b\u4e2d\u9047\u5230\u7684\u9519\u8bef\u3002\u8fd9\u5176\u4e2d\u6709\u4e2a\u975e\u5e38\u91cd\u8981\u7684\u53c2\u6570\u201cacks\u201d,\u8fd9\u4e2a\u53c2\u6570\u51b3\u5b9a\u4e86producer\u8981\u6c42leader partition \u6536\u5230\u786e\u8ba4\u7684\u526f\u672c\u4e2a\u6570\uff0c\u5982\u679cacks\u8bbe\u7f6e\u6570\u91cf\u4e3a0\uff0c\u8868\u793aproducer\u4e0d\u4f1a\u7b49\u5f85broker\u7684\u54cd\u5e94\uff0c\u6240\u4ee5\uff0cproducer\u65e0\u6cd5\u77e5\u9053\u6d88\u606f\u662f\u5426\u53d1\u9001\u6210\u529f\uff0c\u8fd9\u6837\u6709\u53ef\u80fd\u4f1a\u5bfc\u81f4\u6570\u636e\u4e22\u5931\uff0c\u4f46\u540c\u65f6\uff0cacks\u503c\u4e3a0\u4f1a\u5f97\u5230\u6700\u5927\u7684\u7cfb\u7edf\u541e\u5410\u91cf\u3002<\/div>\n<div><\/div>\n<div>\u82e5acks\u8bbe\u7f6e\u4e3a1\uff0c\u8868\u793aproducer\u4f1a\u5728leader partition\u6536\u5230\u6d88\u606f\u65f6\u5f97\u5230broker\u7684\u4e00\u4e2a\u786e\u8ba4\uff0c\u8fd9\u6837\u4f1a\u6709\u66f4\u597d\u7684\u53ef\u9760\u6027\uff0c\u56e0\u4e3a\u5ba2\u6237\u7aef\u4f1a\u7b49\u5f85\u76f4\u5230broker\u786e\u8ba4\u6536\u5230\u6d88\u606f\u3002\u82e5\u8bbe\u7f6e\u4e3a-1\uff0cproducer\u4f1a\u5728\u6240\u6709\u5907\u4efd\u7684partition\u6536\u5230\u6d88\u606f\u65f6\u5f97\u5230broker\u7684\u786e\u8ba4\uff0c\u8fd9\u4e2a\u8bbe\u7f6e\u53ef\u4ee5\u5f97\u5230\u6700\u9ad8\u7684\u53ef\u9760\u6027\u4fdd\u8bc1\u3002<\/div>\n<div><\/div>\n<div>Kafka \u6d88\u606f\u6709\u4e00\u4e2a\u5b9a\u957f\u7684header\u548c\u53d8\u957f\u7684\u5b57\u8282\u6570\u7ec4\u7ec4\u6210\u3002\u56e0\u4e3akafka\u6d88\u606f\u652f\u6301\u5b57\u8282\u6570\u7ec4\uff0c\u4e5f\u5c31\u4f7f\u5f97kafka\u53ef\u4ee5\u652f\u6301\u4efb\u4f55\u7528\u6237\u81ea\u5b9a\u4e49\u7684\u5e8f\u5217\u53f7\u683c\u5f0f\u6216\u8005\u5176\u5b83\u5df2\u6709\u7684\u683c\u5f0f\u5982Apache Avro\u3001protobuf\u7b49\u3002Kafka\u6ca1\u6709\u9650\u5b9a\u5355\u4e2a\u6d88\u606f\u7684\u5927\u5c0f\uff0c\u4f46\u6211\u4eec\u63a8\u8350\u6d88\u606f\u5927\u5c0f\u4e0d\u8981\u8d85\u8fc71MB,\u901a\u5e38\u4e00\u822c\u6d88\u606f\u5927\u5c0f\u90fd\u57281~10kB\u4e4b\u524d\u3002<\/div>\n<div><\/div>\n<div>\u53d1\u5e03\u6d88\u606f\u65f6\uff0ckafka client\u5148\u6784\u9020\u4e00\u6761\u6d88\u606f\uff0c\u5c06\u6d88\u606f\u52a0\u5165\u5230\u6d88\u606f\u96c6set\u4e2d\uff08kafka\u652f\u6301\u6279\u91cf\u53d1\u5e03\uff0c\u53ef\u4ee5\u5f80\u6d88\u606f\u96c6\u5408\u4e2d\u6dfb\u52a0\u591a\u6761\u6d88\u606f\uff0c\u4e00\u6b21\u884c\u53d1\u5e03\uff09\uff0csend\u6d88\u606f\u65f6\uff0cproducer client\u9700\u6307\u5b9a\u6d88\u606f\u6240\u5c5e\u7684topic\u3002<\/div>\n<div><\/div>\n<div>5.2 \u00a0Consumers<\/div>\n<div>Kafka\u63d0\u4f9b\u4e86\u4e24\u5957consumer api\uff0c\u5206\u4e3ahigh-level api\u548csample-api\u3002Sample-api \u662f\u4e00\u4e2a\u5e95\u5c42\u7684API\uff0c\u5b83\u7ef4\u6301\u4e86\u4e00\u4e2a\u548c\u5355\u4e00broker\u7684\u8fde\u63a5\uff0c\u5e76\u4e14\u8fd9\u4e2aAPI\u662f\u5b8c\u5168\u65e0\u72b6\u6001\u7684\uff0c\u6bcf\u6b21\u8bf7\u6c42\u90fd\u9700\u8981\u6307\u5b9aoffset\u503c\uff0c\u56e0\u6b64\uff0c\u8fd9\u5957API\u4e5f\u662f\u6700\u7075\u6d3b\u7684\u3002<\/div>\n<div><\/div>\n<div>\u5728kafka\u4e2d\uff0c\u5f53\u524d\u8bfb\u5230\u54ea\u6761\u6d88\u606f\u7684offset\u503c\u662f\u7531consumer\u6765\u7ef4\u62a4\u7684\uff0c\u56e0\u6b64\uff0cconsumer\u53ef\u4ee5\u81ea\u5df1\u51b3\u5b9a\u5982\u4f55\u8bfb\u53d6kafka\u4e2d\u7684\u6570\u636e\u3002\u6bd4\u5982\uff0cconsumer\u53ef\u4ee5\u901a\u8fc7\u91cd\u8bbeoffset\u503c\u6765\u91cd\u65b0\u6d88\u8d39\u5df2\u6d88\u8d39\u8fc7\u7684\u6570\u636e\u3002\u4e0d\u7ba1\u6709\u6ca1\u6709\u88ab\u6d88\u8d39\uff0ckafka\u4f1a\u4fdd\u5b58\u6570\u636e\u4e00\u6bb5\u65f6\u95f4\uff0c\u8fd9\u4e2a\u65f6\u95f4\u5468\u671f\u662f\u53ef\u914d\u7f6e\u7684\uff0c\u53ea\u6709\u5230\u4e86\u8fc7\u671f\u65f6\u95f4\uff0ckafka\u624d\u4f1a\u5220\u9664\u8fd9\u4e9b\u6570\u636e\u3002\uff08\u8fd9\u4e00\u70b9\u4e0eAMQ\u4e0d\u4e00\u6837\uff0cAMQ\u7684message\u4e00\u822c\u6765\u8bf4\u90fd\u662f\u6301\u4e45\u5316\u5230mysql\u4e2d\u7684\uff0c\u6d88\u8d39\u5b8c\u7684message\u4f1a\u88abdelete\u6389\uff09<\/div>\n<div><\/div>\n<div>High-level API\u5c01\u88c5\u4e86\u5bf9\u96c6\u7fa4\u4e2d\u4e00\u7cfb\u5217broker\u7684\u8bbf\u95ee\uff0c\u53ef\u4ee5\u900f\u660e\u7684\u6d88\u8d39\u4e00\u4e2atopic\u3002\u5b83\u81ea\u5df1\u7ef4\u6301\u4e86\u5df2\u6d88\u8d39\u6d88\u606f\u7684\u72b6\u6001\uff0c\u5373\u6bcf\u6b21\u6d88\u8d39\u7684\u90fd\u662f\u4e0b\u4e00\u4e2a\u6d88\u606f\u3002<\/div>\n<div><\/div>\n<div>High-level API\u8fd8\u652f\u6301\u4ee5\u7ec4\u7684\u5f62\u5f0f\u6d88\u8d39topic\uff0c\u5982\u679cconsumers\u6709\u540c\u4e00\u4e2a\u7ec4\u540d\uff0c\u90a3\u4e48kafka\u5c31\u76f8\u5f53\u4e8e\u4e00\u4e2a\u961f\u5217\u6d88\u606f\u670d\u52a1\uff0c\u800c\u5404\u4e2aconsumer\u5747\u8861\u7684\u6d88\u8d39\u76f8\u5e94partition\u4e2d\u7684\u6570\u636e\u3002\u82e5consumers\u6709\u4e0d\u540c\u7684\u7ec4\u540d\uff0c\u90a3\u4e48\u6b64\u65f6kafka\u5c31\u76f8\u5f53\u4e0e\u4e00\u4e2a\u5e7f\u64ad\u670d\u52a1\uff0c\u4f1a\u628atopic\u4e2d\u7684\u6240\u6709\u6d88\u606f\u5e7f\u64ad\u5230\u6bcf\u4e2aconsumer\u3002<\/div>\n<div><\/div>\n<div>High level api\u548cLow level api\u662f\u9488\u5bf9consumer\u800c\u8a00\u7684\uff0c\u548cproducer\u65e0\u5173\u3002<\/div>\n<div><\/div>\n<div>High level api\u662fconsumer\u8bfb\u7684partition\u7684offsite\u662f\u5b58\u5728zookeeper\u4e0a\u3002High level api\u00a0\u4f1a\u542f\u52a8\u53e6\u5916\u4e00\u4e2a\u7ebf\u7a0b\u53bb\u6bcf\u9694\u4e00\u6bb5\u65f6\u95f4\uff0coffsite\u81ea\u52a8\u540c\u6b65\u5230zookeeper\u4e0a\u3002\u6362\u53e5\u8bdd\u8bf4\uff0c\u5982\u679c\u4f7f\u7528\u4e86High level api\uff0c \u6bcf\u4e2amessage\u53ea\u80fd\u88ab\u8bfb\u4e00\u6b21\uff0c\u4e00\u65e6\u8bfb\u4e86\u8fd9\u6761message\u4e4b\u540e\uff0c\u65e0\u8bba\u6211consumer\u7684\u5904\u7406\u662f\u5426ok\u3002High level api\u7684\u53e6\u5916\u4e00\u4e2a\u7ebf\u7a0b\u4f1a\u81ea\u52a8\u7684\u628aoffiste+1\u540c\u6b65\u5230zookeeper\u4e0a\u3002\u5982\u679cconsumer\u8bfb\u53d6\u6570\u636e\u51fa\u4e86\u95ee\u9898\uff0coffsite\u4e5f\u4f1a\u5728zookeeper\u4e0a\u540c\u6b65\u3002\u56e0\u6b64\uff0c\u5982\u679cconsumer\u5904\u7406\u5931\u8d25\u4e86\uff0c\u4f1a\u7ee7\u7eed\u6267\u884c\u4e0b\u4e00\u6761\u3002\u8fd9\u5f80\u5f80\u662f\u4e0d\u5bf9\u7684\u884c\u4e3a\u3002\u56e0\u6b64\uff0cBest Practice\u662f\u4e00\u65e6consumer\u5904\u7406\u5931\u8d25\uff0c\u76f4\u63a5\u8ba9\u6574\u4e2aconusmer group\u629bException\u7ec8\u6b62\uff0c\u4f46\u662f\u6700\u540e\u8bfb\u7684\u8fd9\u4e00\u6761\u6570\u636e\u662f\u4e22\u5931\u4e86\uff0c\u56e0\u4e3a\u5728zookeeper\u91cc\u9762\u7684offsite\u5df2\u7ecf+1\u4e86\u3002\u7b49\u518d\u6b21\u542f\u52a8conusmer group\u7684\u65f6\u5019\uff0c\u5df2\u7ecf\u4ece\u4e0b\u4e00\u6761\u5f00\u59cb\u8bfb\u53d6\u5904\u7406\u4e86\u3002<\/div>\n<div><\/div>\n<div>Low level api\u662fconsumer\u8bfb\u7684partition\u7684offsite\u5728consumer\u81ea\u5df1\u7684\u7a0b\u5e8f\u4e2d\u7ef4\u62a4\u3002\u4e0d\u4f1a\u540c\u6b65\u5230zookeeper\u4e0a\u3002\u4f46\u662f\u4e3a\u4e86kafka manager\u80fd\u591f\u65b9\u4fbf\u7684\u76d1\u63a7\uff0c\u4e00\u822c\u4e5f\u4f1a\u624b\u52a8\u7684\u540c\u6b65\u5230zookeeper\u4e0a\u3002\u8fd9\u6837\u7684\u597d\u5904\u662f\u4e00\u65e6\u8bfb\u53d6\u67d0\u4e2amessage\u7684consumer\u5931\u8d25\u4e86\uff0c\u8fd9\u6761message\u7684offsite\u6211\u4eec\u81ea\u5df1\u7ef4\u62a4\uff0c\u6211\u4eec\u4e0d\u4f1a+1\u3002\u4e0b\u6b21\u518d\u542f\u52a8\u7684\u65f6\u5019\uff0c\u8fd8\u4f1a\u4ece\u8fd9\u4e2aoffsite\u5f00\u59cb\u8bfb\u3002\u8fd9\u6837\u53ef\u4ee5\u505a\u5230exactly once\u5bf9\u4e8e\u6570\u636e\u7684\u51c6\u786e\u6027\u6709\u4fdd\u8bc1\u3002<\/div>\n<div><\/div>\n<div><\/div>\n<div>\u5bf9\u4e8eConsumer group\uff1a<\/div>\n<div>1. \u5141\u8bb8consumer group\uff08\u5305\u542b\u591a\u4e2aconsumer\uff0c\u5982\u4e00\u4e2a\u96c6\u7fa4\u540c\u65f6\u6d88\u8d39\uff09\u5bf9\u4e00\u4e2atopic\u8fdb\u884c\u6d88\u8d39\uff0c\u4e0d\u540c\u7684consumer group\u4e4b\u95f4\u72ec\u7acb\u6d88\u8d39\u3002<\/div>\n<div>2. \u4e3a\u4e86\u5bf9\u51cf\u5c0f\u4e00\u4e2aconsumer group\u4e2d\u4e0d\u540cconsumer\u4e4b\u95f4\u7684\u5206\u5e03\u5f0f\u534f\u8c03\u5f00\u9500\uff0c\u6307\u5b9apartition\u4e3a\u6700\u5c0f\u7684\u5e76\u884c\u6d88\u8d39\u5355\u4f4d\uff0c\u5373\u4e00\u4e2agroup\u5185\u7684consumer\u53ea\u80fd\u6d88\u8d39\u4e0d\u540c\u7684partition\u3002<\/div>\n<div><img decoding=\"async\" src=\"https:\/\/img-blog.csdn.net\/20170107212800371?watermark\/2\/text\/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMzU3MzEzMw==\/font\/5a6L5L2T\/fontsize\/400\/fill\/I0JBQkFCMA==\/dissolve\/70\/gravity\/Center\" alt=\"\" \/><\/div>\n<div><\/div>\n<div><\/div>\n<div>Consumer\u4e0ePartition\u7684\u5173\u7cfb\uff1a<\/div>\n<div>&#8211; \u5982\u679cconsumer\u6bd4partition\u591a\uff0c\u662f\u6d6a\u8d39\uff0c\u56e0\u4e3akafka\u7684\u8bbe\u8ba1\u662f\u5728\u4e00\u4e2apartition\u4e0a\u662f\u4e0d\u5141\u8bb8\u5e76\u53d1\u7684\uff0c\u6240\u4ee5consumer\u6570\u4e0d\u8981\u5927\u4e8epartition\u6570<\/div>\n<div>&#8211; \u5982\u679cconsumer\u6bd4partition\u5c11\uff0c\u4e00\u4e2aconsumer\u4f1a\u5bf9\u5e94\u4e8e\u591a\u4e2apartitions\uff0c\u8fd9\u91cc\u4e3b\u8981\u5408\u7406\u5206\u914dconsumer\u6570\u548cpartition\u6570\uff0c\u5426\u5219\u4f1a\u5bfc\u81f4partition\u91cc\u9762\u7684\u6570\u636e\u88ab\u53d6\u7684\u4e0d\u5747\u5300<\/div>\n<div>&#8211; \u5982\u679cconsumer\u4ece\u591a\u4e2apartition\u8bfb\u5230\u6570\u636e\uff0c\u4e0d\u4fdd\u8bc1\u6570\u636e\u95f4\u7684\u987a\u5e8f\u6027\uff0ckafka\u53ea\u4fdd\u8bc1\u5728\u4e00\u4e2apartition\u4e0a\u6570\u636e\u662f\u6709\u5e8f\u7684\uff0c\u4f46\u591a\u4e2apartition\uff0c\u6839\u636e\u4f60\u8bfb\u7684\u987a\u5e8f\u4f1a\u6709\u4e0d\u540c<\/div>\n<div>&#8211; \u589e\u51cfconsumer\uff0cbroker\uff0cpartition\u4f1a\u5bfc\u81f4rebalance\uff0c\u6240\u4ee5rebalance\u540econsumer\u5bf9\u5e94\u7684partition\u4f1a\u53d1\u751f\u53d8\u5316<\/div>\n<div>&#8211; High-level\u63a5\u53e3\u4e2d\u83b7\u53d6\u4e0d\u5230\u6570\u636e\u7684\u65f6\u5019\u662f\u4f1ablock\u7684<\/div>\n<div><\/div>\n<div>\u8d1f\u8f7d\u4f4e\u7684\u60c5\u51b5\u4e0b\u53ef\u4ee5\u6bcf\u4e2a\u7ebf\u7a0b\u6d88\u8d39\u591a\u4e2apartition\u3002\u4f46\u8d1f\u8f7d\u9ad8\u7684\u60c5\u51b5\u4e0b\uff0cConsumer \u7ebf\u7a0b\u6570\u6700\u597d\u548cPartition\u6570\u91cf\u4fdd\u6301\u4e00\u81f4\u3002\u5982\u679c\u8fd8\u662f\u6d88\u8d39\u4e0d\u8fc7\u6765\uff0c\u5e94\u8be5\u518d\u5f00 Consumer \u8fdb\u7a0b\uff0c\u8fdb\u7a0b\u5185\u7ebf\u7a0b\u6570\u540c\u6837\u548c\u5206\u533a\u6570\u4e00\u81f4\u3002<\/div>\n<div><\/div>\n<div>\u6d88\u8d39\u6d88\u606f\u65f6\uff0ckafka client\u9700\u6307\u5b9atopic\u4ee5\u53capartition number\uff08\u6bcf\u4e2apartition\u5bf9\u5e94\u4e00\u4e2a\u903b\u8f91\u65e5\u5fd7\u6d41\uff0c\u5982topic\u4ee3\u8868\u67d0\u4e2a\u4ea7\u54c1\u7ebf\uff0cpartition\u4ee3\u8868\u4ea7\u54c1\u7ebf\u7684\u65e5\u5fd7\u6309\u5929\u5207\u5206\u7684\u7ed3\u679c\uff09\uff0cconsumer client\u8ba2\u9605\u540e\uff0c\u5c31\u53ef\u8fed\u4ee3\u8bfb\u53d6\u6d88\u606f\uff0c\u5982\u679c\u6ca1\u6709\u6d88\u606f\uff0cconsumer client\u4f1a\u963b\u585e\u76f4\u5230\u6709\u65b0\u7684\u6d88\u606f\u53d1\u5e03\u3002consumer\u53ef\u4ee5\u7d2f\u79ef\u786e\u8ba4\u63a5\u6536\u5230\u7684\u6d88\u606f\uff0c\u5f53\u5176\u786e\u8ba4\u4e86\u67d0\u4e2aoffset\u7684\u6d88\u606f\uff0c\u610f\u5473\u7740\u4e4b\u524d\u7684\u6d88\u606f\u4e5f\u90fd\u5df2\u6210\u529f\u63a5\u6536\u5230\uff0c\u6b64\u65f6broker\u4f1a\u66f4\u65b0zookeeper\u4e0a\u5730offset registry\u3002<\/div>\n<div><\/div>\n<\/div>\n<div>\n<div>5.3\u00a0\u00a0\u9ad8\u6548\u7684\u6570\u636e\u4f20\u8f93<\/div>\n<div>\n<div align=\"left\">\n<div><span lang=\"en-us\" xml:lang=\"en-us\">1.\u00a0\u00a0<\/span>\u53d1\u5e03\u8005\u6bcf\u6b21\u53ef\u53d1\u5e03\u591a\u6761\u6d88\u606f\uff08\u5c06\u6d88\u606f\u52a0\u5230\u4e00\u4e2a\u6d88\u606f\u96c6\u5408\u4e2d\u53d1\u5e03\uff09\uff0c<span lang=\"en-us\" xml:lang=\"en-us\">\u00a0consumer<\/span>\u6bcf\u6b21\u8fed\u4ee3\u6d88\u8d39\u4e00\u6761\u6d88\u606f\u3002<\/div>\n<\/div>\n<p align=\"left\"><span lang=\"en-us\" xml:lang=\"en-us\">2.\u00a0\u00a0<\/span>\u4e0d\u521b\u5efa\u5355\u72ec\u7684<span lang=\"en-us\" xml:lang=\"en-us\">cache<\/span>\uff0c\u4f7f\u7528\u7cfb\u7edf\u7684<span lang=\"en-us\" xml:lang=\"en-us\">page cache<\/span>\u3002\u53d1\u5e03\u8005\u987a\u5e8f\u53d1\u5e03\uff0c\u8ba2\u9605\u8005\u901a\u5e38\u6bd4\u53d1\u5e03\u8005\u6ede\u540e\u4e00\u70b9\u70b9\uff0c\u76f4\u63a5\u4f7f\u7528<span lang=\"en-us\" xml:lang=\"en-us\"><a class=\"replace_word\" title=\"Linux\u77e5\u8bc6\u5e93\" href=\"http:\/\/lib.csdn.net\/base\/linux\" target=\"_blank\" rel=\"nofollow noopener\">Linux<\/a><\/span>\u7684<span lang=\"en-us\" xml:lang=\"en-us\">page cache<\/span>\u6548\u679c\u4e5f\u6bd4\u8f83\u540e\uff0c\u540c\u65f6\u51cf\u5c11\u4e86<span lang=\"en-us\" xml:lang=\"en-us\">cache<\/span>\u7ba1\u7406\u53ca\u5783\u573e\u6536\u96c6\u7684\u5f00\u9500\u3002<\/p>\n<div align=\"left\">\n<div><span lang=\"en-us\" xml:lang=\"en-us\">3.\u00a0\u00a0<\/span>\u4f7f\u7528<span lang=\"en-us\" xml:lang=\"en-us\">sendfile<\/span>\u4f18\u5316\u7f51\u7edc\u4f20\u8f93\uff0c\u51cf\u5c11\u4e00\u6b21\u5185\u5b58\u62f7\u8d1d\u3002<\/div>\n<\/div>\n<\/div>\n<div><\/div>\n<div>6.Kafka \u4e0e Zookeeper<\/div>\n<div>\n<div align=\"left\">\n<div><span lang=\"en-us\" xml:lang=\"en-us\">6.1 Zookeeper\u00a0<\/span>\u534f\u8c03\u63a7\u5236<\/div>\n<\/div>\n<div align=\"left\">\n<div><span lang=\"en-us\" xml:lang=\"en-us\">1.\u00a0<\/span>\u7ba1\u7406<span lang=\"en-us\" xml:lang=\"en-us\">broker<\/span>\u4e0e<span lang=\"en-us\" xml:lang=\"en-us\">consumer<\/span>\u7684\u52a8\u6001\u52a0\u5165\u4e0e\u79bb\u5f00\u3002(Producer\u4e0d\u9700\u8981\u7ba1\u7406\uff0c\u968f\u4fbf\u4e00\u53f0\u8ba1\u7b97\u673a\u90fd\u53ef\u4ee5\u4f5c\u4e3aProducer\u5411Kakfa Broker\u53d1\u6d88\u606f)<\/div>\n<\/div>\n<div align=\"left\">\n<div><span lang=\"en-us\" xml:lang=\"en-us\">2.\u00a0<\/span>\u89e6\u53d1\u8d1f\u8f7d\u5747\u8861\uff0c\u5f53<span lang=\"en-us\" xml:lang=\"en-us\">broker<\/span>\u6216<span lang=\"en-us\" xml:lang=\"en-us\">consumer<\/span>\u52a0\u5165\u6216\u79bb\u5f00\u65f6\u4f1a\u89e6\u53d1\u8d1f\u8f7d\u5747\u8861\u7b97\u6cd5\uff0c\u4f7f\u5f97\u4e00<\/div>\n<\/div>\n<div align=\"left\">\n<div>\u00a0 \u00a0\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">consumer group<\/span>\u5185\u7684\u591a\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">consumer<\/span>\u7684\u6d88\u8d39\u8d1f\u8f7d\u5e73\u8861\u3002\uff08\u56e0\u4e3a\u4e00\u4e2acomsumer\u6d88\u8d39\u4e00\u4e2a\u6216\u591a\u4e2apartition\uff0c\u4e00\u4e2apartition\u53ea\u80fd\u88ab\u4e00\u4e2aconsumer\u6d88\u8d39\uff09<\/div>\n<\/div>\n<p align=\"left\"><span lang=\"en-us\" xml:lang=\"en-us\">3.\u00a0\u00a0<\/span>\u7ef4\u62a4\u6d88\u8d39\u5173\u7cfb\u53ca\u6bcf\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">partition<\/span>\u7684\u6d88\u8d39\u4fe1\u606f\u3002<\/p>\n<p align=\"left\"><span lang=\"en-us\" xml:lang=\"en-us\">6.2 Zookeeper<\/span>\u4e0a\u7684\u7ec6\u8282\uff1a<\/p>\n<p align=\"left\"><span lang=\"en-us\" xml:lang=\"en-us\">1.\u00a0<\/span>\u6bcf\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">broker<\/span>\u542f\u52a8\u540e\u4f1a\u5728<span lang=\"en-us\" xml:lang=\"en-us\">zookeeper<\/span>\u4e0a\u6ce8\u518c\u4e00\u4e2a\u4e34\u65f6\u7684<span lang=\"en-us\" xml:lang=\"en-us\">broker registry<\/span>\uff0c\u5305\u542b<span lang=\"en-us\" xml:lang=\"en-us\">broker<\/span>\u7684<span lang=\"en-us\" xml:lang=\"en-us\">ip<\/span>\u5730\u5740\u548c\u7aef\u53e3\u53f7\uff0c\u6240\u5b58\u50a8\u7684<span lang=\"en-us\" xml:lang=\"en-us\">topics<\/span>\u548c<span lang=\"en-us\" xml:lang=\"en-us\">partitions<\/span>\u4fe1\u606f\u3002<\/p>\n<p align=\"left\"><span lang=\"en-us\" xml:lang=\"en-us\">2.\u00a0<\/span>\u6bcf\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">consumer<\/span>\u542f\u52a8\u540e\u4f1a\u5728<span lang=\"en-us\" xml:lang=\"en-us\">zookeeper<\/span>\u4e0a\u6ce8\u518c\u4e00\u4e2a\u4e34\u65f6\u7684<span lang=\"en-us\" xml:lang=\"en-us\">consumer registry<\/span>\uff1a\u5305\u542b<span lang=\"en-us\" xml:lang=\"en-us\">consumer<\/span>\u6240\u5c5e\u7684<span lang=\"en-us\" xml:lang=\"en-us\">consumer group<\/span>\u4ee5\u53ca\u8ba2\u9605\u7684<span lang=\"en-us\" xml:lang=\"en-us\">topics<\/span>\u3002<\/p>\n<p align=\"left\"><span lang=\"en-us\" xml:lang=\"en-us\">3.\u00a0<\/span>\u6bcf\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">consumer group<\/span>\u5173\u8054\u4e00\u4e2a\u4e34\u65f6\u7684<span lang=\"en-us\" xml:lang=\"en-us\">owner registry<\/span>\u548c\u4e00\u4e2a\u6301\u4e45\u7684<span lang=\"en-us\" xml:lang=\"en-us\">offset registry<\/span>\u3002\u5bf9\u4e8e\u88ab\u8ba2\u9605\u7684\u6bcf\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">partition<\/span>\u5305\u542b\u4e00\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">owner registry<\/span>\uff0c\u5185\u5bb9\u4e3a\u8ba2\u9605\u8fd9\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">partition<\/span>\u7684<span lang=\"en-us\" xml:lang=\"en-us\">consumer id<\/span>\uff1b\u540c\u65f6\u5305\u542b\u4e00\u4e2a<span lang=\"en-us\" xml:lang=\"en-us\">offset registry<\/span>\uff0c\u5185\u5bb9\u4e3a\u4e0a\u4e00\u6b21\u8ba2\u9605\u7684<span lang=\"en-us\" xml:lang=\"en-us\">offset<\/span>\u3002<\/p>\n<p align=\"left\">\u8f6c\u81ea\uff1ahttps:\/\/blog.csdn.net\/YChenFeng\/article\/details\/74980531<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Kafka Kafka\u662f\u6700\u521d\u7531Linkedin\u516c\u53f8\u5f00\u53d1\uff0c\u662f\u4e00\u4e2a\u5206\u5e03\u5f0f\u3001\u652f\u6301\u5206\u533a\u7684\uff08partition\uff09\u3001\u591a\u526f\u672c\u7684\uff08replica\uff09\uff0c\u57fa\u4e8ezookeeper\u534f\u8c03\u7684\u5206\u5e03\u5f0f\u6d88\u606f\u7cfb\u7edf\uff0c\u5b83\u7684\u6700\u5927\u7684\u7279\u6027\u5c31\u662f\u53ef\u4ee5\u5b9e\u65f6\u7684\u5904\u7406\u5927\u91cf\u6570\u636e\u4ee5\u6ee1\u8db3\u5404\u79cd\u9700\u6c42\u573a\u666f\uff1a\u6bd4\u5982\u57fa\u4e8ehadoop\u7684\u6279\u5904\u7406\u7cfb\u7edf\u3001\u4f4e\u5ef6\u8fdf\u7684\u5b9e\u65f6\u7cfb\u7edf\u3001storm\/Spark\u6d41\u5f0f\u5904\u7406\u5f15\u64ce\uff0cweb\/nginx\u65e5\u5fd7\u3001\u8bbf\u95ee\u65e5\u5fd7\uff0c\u6d88\u606f\u670d\u52a1\u7b49\u7b49\uff0c\u7528scala\u8bed\u8a00\u7f16\u5199\uff0cLinkedin\u4e8e2010\u5e74\u8d21\u732e\u7ed9\u4e86Apache\u57fa\u91d1\u4f1a\u5e76\u6210\u4e3a\u9876\u7ea7\u5f00\u6e90 \u9879\u76ee\u3002 1.\u524d\u8a00 \u6d88\u606f\u961f\u5217\u7684\u6027\u80fd\u597d\u574f\uff0c\u5176\u6587\u4ef6\u5b58\u50a8\u673a\u5236\u8bbe\u8ba1\u662f\u8861\u91cf\u4e00\u4e2a\u6d88\u606f\u961f\u5217\u670d\u52a1\u6280\u672f\u6c34\u5e73\u548c\u6700\u5173\u952e\u6307\u6807\u4e4b\u4e00\u3002\u4e0b\u9762\u5c06\u4eceKafka\u6587\u4ef6\u5b58\u50a8\u673a\u5236\u548c\u7269\u7406\u7ed3\u6784\u89d2\u5ea6\uff0c\u5206\u6790Kafka\u662f\u5982\u4f55\u5b9e\u73b0\u9ad8\u6548\u6587\u4ef6\u5b58\u50a8\uff0c\u53ca\u5b9e\u9645\u5e94\u7528\u6548\u679c\u3002 \u00a01.1 \u00a0Kafka\u7684\u7279\u6027: &#8211; \u9ad8\u541e\u5410\u91cf\u3001\u4f4e\u5ef6\u8fdf\uff1akafka\u6bcf\u79d2\u53ef\u4ee5\u5904\u7406\u51e0\u5341\u4e07\u6761\u6d88\u606f\uff0c\u5b83\u7684\u5ef6\u8fdf\u6700\u4f4e\u53ea\u6709\u51e0\u6beb\u79d2\uff0c\u6bcf\u4e2atopic\u53ef\u4ee5\u5206\u591a\u4e2apartition, consumer group \u5bf9partition\u8fdb\u884cconsume\u64cd\u4f5c\u3002 &#8211; \u53ef\u6269\u5c55\u6027\uff1akafka\u96c6\u7fa4\u652f\u6301\u70ed\u6269\u5c55 &#8211; \u6301\u4e45\u6027\u3001\u53ef\u9760\u6027\uff1a\u6d88\u606f\u88ab\u6301\u4e45\u5316\u5230\u672c\u5730\u78c1\u76d8\uff0c\u5e76\u4e14\u652f\u6301\u6570\u636e\u5907\u4efd\u9632\u6b62\u6570\u636e\u4e22\u5931 &#8211; \u5bb9\u9519\u6027\uff1a\u5141\u8bb8\u96c6\u7fa4\u4e2d\u8282\u70b9\u5931\u8d25\uff08\u82e5\u526f\u672c\u6570\u91cf\u4e3an,\u5219\u5141\u8bb8n-1\u4e2a\u8282\u70b9\u5931\u8d25\uff09 &#8211; \u9ad8\u5e76\u53d1\uff1a\u652f\u6301\u6570\u5343\u4e2a\u5ba2\u6237\u7aef\u540c\u65f6\u8bfb\u5199 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1118","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=\/wp\/v2\/posts\/1118","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1118"}],"version-history":[{"count":0,"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=\/wp\/v2\/posts\/1118\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1118"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1118"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.jsjs.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}