kafka集群性能测试
kafka集群性能测试
生产者写入测试
向一个只有3个分区和3个副本的Topic主题perf-producer-test 中发送 100 万条消息，并且每条消息大小为 1024B 生产者，对应的 acks 参数为1
bin/kafka-topics.sh --bootstrap-server=host117:19092,host120:19092,host121:19092 \
--create --partitions 3 --replication-factor 3 --topic perf-producer-test
bin/kafka-producer-perf-test.sh \
--topic perf-producer-test \
--num-records 1000000 \
--record-size 1024 \
--throughput -1 \
--print-metrics \
--producer-props bootstrap.servers=host117:19092,host120:19092,host121:19092 \
acks=1
# --nurn-records：用来指定发送消息的总条数
# --record-size：用来设置每条消息的字节数
# --throughput：用来进行限流控制，当设定的值小于0不限流，当设定的值大于0时，当发送的吞吐量大于该值时就会被阻塞一段时间。
# --print-metrics：测试完成后的Metrics信息
# --producer-props：参数用来指定生产者的配置，可同时指定多组配置，各组配置之间以空格分隔，与producer-props 参数对应的还有一个producer-config参数，它用来指定生产者的配置文件
测试结果
262093 records sent, 52418.6 records/sec (51.19 MB/sec), 5.1 ms avg latency, 577.0 ms max latency.
302967 records sent, 60593.4 records/sec (59.17 MB/sec), 1.2 ms avg latency, 23.0 ms max latency.
299259 records sent, 59851.8 records/sec (58.45 MB/sec), 1.1 ms avg latency, 21.0 ms max latency.
1000000 records sent, 58132.775259 records/sec (56.77 MB/sec), 2.16 ms avg latency, 577.00 ms max latency, 1 ms 50th, 3 ms 95th, 50 ms 99th, 83 ms 99.9th.
Metric Name Value
app-info:commit-id:{client-id=perf-producer-client} : e23c59d00e687ff5
app-info:start-time-ms:{client-id=perf-producer-client} : 1672883275668
app-info:version:{client-id=perf-producer-client} : 3.3.1
kafka-metrics-count:count:{client-id=perf-producer-client} : 136.000
producer-metrics:batch-size-avg:{client-id=perf-producer-client} : 5652.279
producer-metrics:batch-size-max:{client-id=perf-producer-client} : 15556.000
producer-metrics:batch-split-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:batch-split-total:{client-id=perf-producer-client} : 0.000
producer-metrics:buffer-available-bytes:{client-id=perf-producer-client} : 33554432.000
producer-metrics:buffer-exhausted-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:buffer-exhausted-total:{client-id=perf-producer-client} : 0.000
producer-metrics:buffer-total-bytes:{client-id=perf-producer-client} : 33554432.000
producer-metrics:bufferpool-wait-ratio:{client-id=perf-producer-client} : 0.000
producer-metrics:bufferpool-wait-time-ns-total:{client-id=perf-producer-client} : 0.000
producer-metrics:bufferpool-wait-time-total:{client-id=perf-producer-client} : 0.000
producer-metrics:compression-rate-avg:{client-id=perf-producer-client} : 1.000
producer-metrics:connection-close-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:connection-close-total:{client-id=perf-producer-client} : 0.000
producer-metrics:connection-count:{client-id=perf-producer-client} : 4.000
producer-metrics:connection-creation-rate:{client-id=perf-producer-client} : 0.085
producer-metrics:connection-creation-total:{client-id=perf-producer-client} : 4.000
producer-metrics:failed-authentication-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:failed-authentication-total:{client-id=perf-producer-client} : 0.000
producer-metrics:failed-reauthentication-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:failed-reauthentication-total:{client-id=perf-producer-client} : 0.000
producer-metrics:flush-time-ns-total:{client-id=perf-producer-client} : 3747524.000
producer-metrics:incoming-byte-rate:{client-id=perf-producer-client} : 271857.204
producer-metrics:incoming-byte-total:{client-id=perf-producer-client} : 12749831.000
producer-metrics:io-ratio:{client-id=perf-producer-client} : 0.079
producer-metrics:io-time-ns-avg:{client-id=perf-producer-client} : 10222.730
producer-metrics:io-time-ns-total:{client-id=perf-producer-client} : 3747877657.000
producer-metrics:io-wait-ratio:{client-id=perf-producer-client} : 0.147
producer-metrics:io-wait-time-ns-avg:{client-id=perf-producer-client} : 18953.205
producer-metrics:io-wait-time-ns-total:{client-id=perf-producer-client} : 6948661856.000
producer-metrics:io-waittime-total:{client-id=perf-producer-client} : 6948661856.000
producer-metrics:iotime-total:{client-id=perf-producer-client} : 3747877657.000
producer-metrics:metadata-age:{client-id=perf-producer-client} : 16.771
producer-metrics:metadata-wait-time-ns-total:{client-id=perf-producer-client} : 459611281.000
producer-metrics:network-io-rate:{client-id=perf-producer-client} : 7878.427
producer-metrics:network-io-total:{client-id=perf-producer-client} : 369514.000
producer-metrics:outgoing-byte-rate:{client-id=perf-producer-client} : 22548555.094
producer-metrics:outgoing-byte-total:{client-id=perf-producer-client} : 1057572331.000
producer-metrics:produce-throttle-time-avg:{client-id=perf-producer-client} : 0.000
producer-metrics:produce-throttle-time-max:{client-id=perf-producer-client} : 0.000
producer-metrics:reauthentication-latency-avg:{client-id=perf-producer-client} : NaN
producer-metrics:reauthentication-latency-max:{client-id=perf-producer-client} : NaN
producer-metrics:record-error-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:record-error-total:{client-id=perf-producer-client} : 0.000
producer-metrics:record-queue-time-avg:{client-id=perf-producer-client} : 0.434
producer-metrics:record-queue-time-max:{client-id=perf-producer-client} : 90.000
producer-metrics:record-retry-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:record-retry-total:{client-id=perf-producer-client} : 0.000
producer-metrics:record-send-rate:{client-id=perf-producer-client} : 21405.026
producer-metrics:record-send-total:{client-id=perf-producer-client} : 1000000.000
producer-metrics:record-size-avg:{client-id=perf-producer-client} : 1110.000
producer-metrics:record-size-max:{client-id=perf-producer-client} : 1110.000
producer-metrics:records-per-request-avg:{client-id=perf-producer-client} : 5.413
producer-metrics:request-latency-avg:{client-id=perf-producer-client} : 0.884
producer-metrics:request-latency-max:{client-id=perf-producer-client} : 74.000
producer-metrics:request-rate:{client-id=perf-producer-client} : 3939.214
producer-metrics:request-size-avg:{client-id=perf-producer-client} : 5724.126
producer-metrics:request-size-max:{client-id=perf-producer-client} : 15628.000
producer-metrics:request-total:{client-id=perf-producer-client} : 184757.000
producer-metrics:requests-in-flight:{client-id=perf-producer-client} : 0.000
producer-metrics:response-rate:{client-id=perf-producer-client} : 3939.550
producer-metrics:response-total:{client-id=perf-producer-client} : 184757.000
producer-metrics:select-rate:{client-id=perf-producer-client} : 7766.593
producer-metrics:select-total:{client-id=perf-producer-client} : 366622.000
producer-metrics:successful-authentication-no-reauth-total:{client-id=perf-producer-client} : 0.000
producer-metrics:successful-authentication-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:successful-authentication-total:{client-id=perf-producer-client} : 0.000
producer-metrics:successful-reauthentication-rate:{client-id=perf-producer-client} : 0.000
producer-metrics:successful-reauthentication-total:{client-id=perf-producer-client} : 0.000
producer-metrics:txn-abort-time-ns-total:{client-id=perf-producer-client} : 0.000
producer-metrics:txn-begin-time-ns-total:{client-id=perf-producer-client} : 0.000
producer-metrics:txn-commit-time-ns-total:{client-id=perf-producer-client} : 0.000
producer-metrics:txn-init-time-ns-total:{client-id=perf-producer-client} : 0.000
producer-metrics:txn-send-offsets-time-ns-total:{client-id=perf-producer-client} : 0.000
producer-metrics:waiting-threads:{client-id=perf-producer-client} : 0.000
producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client, node-id=node--2} : 14.627
producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client, node-id=node-1} : 117341.309
producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client, node-id=node-2} : 80083.062
producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client, node-id=node-3} : 75486.908
producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client, node-id=node--2} : 686.000
producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client, node-id=node-1} : 5482538.000
producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client, node-id=node-2} : 3740840.000
producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client, node-id=node-3} : 3525767.000
producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client, node-id=node--2} : 2.878
producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client, node-id=node-1} : 7141605.329
producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client, node-id=node-2} : 7649327.142
producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client, node-id=node-3} : 7845067.953
producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client, node-id=node--2} : 135.000
producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client, node-id=node-1} : 333691509.000
producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client, node-id=node-2} : 357445408.000
producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client, node-id=node-3} : 366435279.000
producer-node-metrics:request-latency-avg:{client-id=perf-producer-client, node-id=node--2} : NaN
producer-node-metrics:request-latency-avg:{client-id=perf-producer-client, node-id=node-1} : 0.511
producer-node-metrics:request-latency-avg:{client-id=perf-producer-client, node-id=node-2} : 1.100
producer-node-metrics:request-latency-avg:{client-id=perf-producer-client, node-id=node-3} : 1.234
producer-node-metrics:request-latency-max:{client-id=perf-producer-client, node-id=node--2} : NaN
producer-node-metrics:request-latency-max:{client-id=perf-producer-client, node-id=node-1} : 42.000
producer-node-metrics:request-latency-max:{client-id=perf-producer-client, node-id=node-2} : 74.000
producer-node-metrics:request-latency-max:{client-id=perf-producer-client, node-id=node-3} : 48.000
producer-node-metrics:request-rate:{client-id=perf-producer-client, node-id=node--2} : 0.043
producer-node-metrics:request-rate:{client-id=perf-producer-client, node-id=node-1} : 1700.454
producer-node-metrics:request-rate:{client-id=perf-producer-client, node-id=node-2} : 1160.093
producer-node-metrics:request-rate:{client-id=perf-producer-client, node-id=node-3} : 1093.858
producer-node-metrics:request-size-avg:{client-id=perf-producer-client, node-id=node--2} : 67.500
producer-node-metrics:request-size-avg:{client-id=perf-producer-client, node-id=node-1} : 4199.913
producer-node-metrics:request-size-avg:{client-id=perf-producer-client, node-id=node-2} : 6593.717
producer-node-metrics:request-size-avg:{client-id=perf-producer-client, node-id=node-3} : 7171.927
producer-node-metrics:request-size-max:{client-id=perf-producer-client, node-id=node--2} : 75.000
producer-node-metrics:request-size-max:{client-id=perf-producer-client, node-id=node-1} : 15628.000
producer-node-metrics:request-size-max:{client-id=perf-producer-client, node-id=node-2} : 15628.000
producer-node-metrics:request-size-max:{client-id=perf-producer-client, node-id=node-3} : 15628.000
producer-node-metrics:request-total:{client-id=perf-producer-client, node-id=node--2} : 2.000
producer-node-metrics:request-total:{client-id=perf-producer-client, node-id=node-1} : 79452.000
producer-node-metrics:request-total:{client-id=perf-producer-client, node-id=node-2} : 54210.000
producer-node-metrics:request-total:{client-id=perf-producer-client, node-id=node-3} : 51093.000
producer-node-metrics:response-rate:{client-id=perf-producer-client, node-id=node--2} : 0.043
producer-node-metrics:response-rate:{client-id=perf-producer-client, node-id=node-1} : 1700.527
producer-node-metrics:response-rate:{client-id=perf-producer-client, node-id=node-2} : 1160.540
producer-node-metrics:response-rate:{client-id=perf-producer-client, node-id=node-3} : 1093.928
producer-node-metrics:response-total:{client-id=perf-producer-client, node-id=node--2} : 2.000
producer-node-metrics:response-total:{client-id=perf-producer-client, node-id=node-1} : 79452.000
producer-node-metrics:response-total:{client-id=perf-producer-client, node-id=node-2} : 54210.000
producer-node-metrics:response-total:{client-id=perf-producer-client, node-id=node-3} : 51093.000
producer-topic-metrics:byte-rate:{client-id=perf-producer-client, topic=perf-producer-test} : 22353102.126
producer-topic-metrics:byte-total:{client-id=perf-producer-client, topic=perf-producer-test} : 1044269872.000
producer-topic-metrics:compression-rate:{client-id=perf-producer-client, topic=perf-producer-test} : 1.000
producer-topic-metrics:record-error-rate:{client-id=perf-producer-client, topic=perf-producer-test} : 0.000
producer-topic-metrics:record-error-total:{client-id=perf-producer-client, topic=perf-producer-test} : 0.000
producer-topic-metrics:record-retry-rate:{client-id=perf-producer-client, topic=perf-producer-test} : 0.000
producer-topic-metrics:record-retry-total:{client-id=perf-producer-client, topic=perf-producer-test} : 0.000
producer-topic-metrics:record-send-rate:{client-id=perf-producer-client, topic=perf-producer-test} : 21405.484
producer-topic-metrics:record-send-total:{client-id=perf-producer-client, topic=perf-producer-test} : 1000000.000
消费者性能测试
bin/kafka-consumer-perf-test.sh \
--topic perf-producer-test \
--messages 1000000 \
--print-metrics \
--broker-list host117:19092,host120:19092,host121:19092
测试结果
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-01-05 09:51:00:099, 2023-01-05 09:51:11:269, 976.5625, 87.4273, 1000000, 89525.5148, 3896, 7274, 134.2538, 137475.9417
Metric Name Value
consumer-coordinator-metrics:assigned-partitions:{client-id=perf-consumer-client} : 0.000
consumer-coordinator-metrics:commit-latency-avg:{client-id=perf-consumer-client} : 19.500
consumer-coordinator-metrics:commit-latency-max:{client-id=perf-consumer-client} : 34.000
consumer-coordinator-metrics:commit-rate:{client-id=perf-consumer-client} : 0.062
consumer-coordinator-metrics:commit-total:{client-id=perf-consumer-client} : 2.000
consumer-coordinator-metrics:failed-rebalance-rate-per-hour:{client-id=perf-consumer-client} : 177.056
consumer-coordinator-metrics:failed-rebalance-total:{client-id=perf-consumer-client} : 2.000
consumer-coordinator-metrics:heartbeat-rate:{client-id=perf-consumer-client} : 0.058
consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=perf-consumer-client} : 10.000
consumer-coordinator-metrics:heartbeat-total:{client-id=perf-consumer-client} : 2.000
consumer-coordinator-metrics:join-rate:{client-id=perf-consumer-client} : 0.027
consumer-coordinator-metrics:join-time-avg:{client-id=perf-consumer-client} : 3028.000
consumer-coordinator-metrics:join-time-max:{client-id=perf-consumer-client} : 3028.000
consumer-coordinator-metrics:join-total:{client-id=perf-consumer-client} : 1.000
consumer-coordinator-metrics:last-heartbeat-seconds-ago:{client-id=perf-consumer-client} : 1.000
consumer-coordinator-metrics:last-rebalance-seconds-ago:{client-id=perf-consumer-client} : 7.000
consumer-coordinator-metrics:partition-assigned-latency-avg:{client-id=perf-consumer-client} : 0.000
consumer-coordinator-metrics:partition-assigned-latency-max:{client-id=perf-consumer-client} : 0.000
consumer-coordinator-metrics:partition-lost-latency-avg:{client-id=perf-consumer-client} : NaN
consumer-coordinator-metrics:partition-lost-latency-max:{client-id=perf-consumer-client} : NaN
consumer-coordinator-metrics:partition-revoked-latency-avg:{client-id=perf-consumer-client} : 0.000
consumer-coordinator-metrics:partition-revoked-latency-max:{client-id=perf-consumer-client} : 0.000
consumer-coordinator-metrics:rebalance-latency-avg:{client-id=perf-consumer-client} : 3297.000
consumer-coordinator-metrics:rebalance-latency-max:{client-id=perf-consumer-client} : 3297.000
consumer-coordinator-metrics:rebalance-latency-total:{client-id=perf-consumer-client} : 3297.000
consumer-coordinator-metrics:rebalance-rate-per-hour:{client-id=perf-consumer-client} : 96.228
consumer-coordinator-metrics:rebalance-total:{client-id=perf-consumer-client} : 1.000
consumer-coordinator-metrics:sync-rate:{client-id=perf-consumer-client} : 0.027
consumer-coordinator-metrics:sync-time-avg:{client-id=perf-consumer-client} : 44.000
consumer-coordinator-metrics:sync-time-max:{client-id=perf-consumer-client} : 44.000
consumer-coordinator-metrics:sync-total:{client-id=perf-consumer-client} : 1.000
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client, topic=perf-producer-test} : 27690658.089
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client} : 27690658.089
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client, topic=perf-producer-test} : 1033000000.000
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client} : 1033000000.000
consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client} : 19.617
consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client} : 501.000
consumer-fetch-manager-metrics:fetch-rate:{client-id=perf-consumer-client} : 27.105
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client, topic=perf-producer-test} : 1031968.032
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client} : 1031968.032
consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client, topic=perf-producer-test} : 1041264.000
consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client} : 1041264.000
consumer-fetch-manager-metrics:fetch-throttle-time-avg:{client-id=perf-consumer-client} : 0.000
consumer-fetch-manager-metrics:fetch-throttle-time-max:{client-id=perf-consumer-client} : 0.000
consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client} : 1012.000
consumer-fetch-manager-metrics:preferred-read-replica:{client-id=perf-consumer-client, topic=perf-producer-test, partition=0} : -1
consumer-fetch-manager-metrics:preferred-read-replica:{client-id=perf-consumer-client, topic=perf-producer-test, partition=1} : -1
consumer-fetch-manager-metrics:preferred-read-replica:{client-id=perf-consumer-client, topic=perf-producer-test, partition=2} : -1
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client, topic=perf-producer-test} : 26806.777
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client} : 26806.058
consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client, topic=perf-producer-test} : 1000000.000
consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client} : 1000000.000
consumer-fetch-manager-metrics:records-lag-avg:{client-id=perf-consumer-client, topic=perf-producer-test, partition=0} : 162268.590
consumer-fetch-manager-metrics:records-lag-avg:{client-id=perf-consumer-client, topic=perf-producer-test, partition=1} : 172422.787
consumer-fetch-manager-metrics:records-lag-avg:{client-id=perf-consumer-client, topic=perf-producer-test, partition=2} : 176060.943
consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client, topic=perf-producer-test, partition=0} : 312302.000
consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client, topic=perf-producer-test, partition=1} : 338547.000
consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client, topic=perf-producer-test, partition=2} : 347656.000
consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client} : 347656.000
consumer-fetch-manager-metrics:records-lag:{client-id=perf-consumer-client, topic=perf-producer-test, partition=0} : 0.000
consumer-fetch-manager-metrics:records-lag:{client-id=perf-consumer-client, topic=perf-producer-test, partition=1} : 0.000
consumer-fetch-manager-metrics:records-lag:{client-id=perf-consumer-client, topic=perf-producer-test, partition=2} : 0.000
consumer-fetch-manager-metrics:records-lead-avg:{client-id=perf-consumer-client, topic=perf-producer-test, partition=0} : 150533.410
consumer-fetch-manager-metrics:records-lead-avg:{client-id=perf-consumer-client, topic=perf-producer-test, partition=1} : 166624.213
consumer-fetch-manager-metrics:records-lead-avg:{client-id=perf-consumer-client, topic=perf-producer-test, partition=2} : 172090.057
consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client, topic=perf-producer-test, partition=0} : 500.000
consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client, topic=perf-producer-test, partition=1} : 500.000
consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client, topic=perf-producer-test, partition=2} : 495.000
consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client} : 495.000
consumer-fetch-manager-metrics:records-lead:{client-id=perf-consumer-client, topic=perf-producer-test, partition=0} : 312802.000
consumer-fetch-manager-metrics:records-lead:{client-id=perf-consumer-client, topic=perf-producer-test, partition=1} : 339047.000
consumer-fetch-manager-metrics:records-lead:{client-id=perf-consumer-client, topic=perf-producer-test, partition=2} : 348151.000
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client, topic=perf-producer-test} : 999.001
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client} : 999.001
kafka-metrics-count:count:{client-id=perf-consumer-client} : 75.000
Powered by Waline v2.14.1