Kafka shell scripts
kafka-topics.sh : 토픽 생성, 조회, 수정 등 역할
- --bootstrap-server : 토픽관련 명령어를 수행할 대상 카프카 클러스터
- --replication-factor : 레플리카 개수 지정
- --partitions : 파티션 개수 설정
- --config : 각종 토픽 설정 가능(retention.ms, segment.byte 등)
- --create : 토픽 생성
- --delete : 토픽 제거
- --describe: 토픽 상세 확인
- --list : 카프카 클러스터의 토픽 리스트 확인
- --version : 대상 카프카 클러스터 버전 확인
kafka-console-consumer.sh : 토픽의 레코드 즉시 조회
kafka-console-producer.sh : 토픽의 레코드를 전달
kafka-consumer-groups.sh : 컨슈머그룹 조회, 컨슈머 오프셋 확인, 수정
- --bootstrap-server
- --describe
- --group
자주 사용하는 명령
토픽 생성
kafka-topics.sh --create
--zookeeper localhost:2181
--replication-factor 1
--partitions 1
--topic mytopic토픽 리스트 조회
bash-4.4# kafka-topics.sh --list --zookeeper 192.168.0.1:2181
__consumer_offsets
micro-service-topic-0
micro-service-topic-1토픽 정보 조회
bash-4.4# kafka-topics.sh --zookeeper 192.168.0.1:2181 --topic micro-service-0 --describe
Topic: micro-service-0 PartitionCount: 2 ReplicationFactor: 2 Configs: retention.ms=1000
Topic: micro-service-0 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: micro-service-0 Partition: 1 Leader: 2 Replicas: 2,1 Isr: 1,2
Retention 변경
kafka-topics.sh --zookeeper 192.168.0.1:2181 --alter
--topic micro-service-0
--config retention.ms=1000
컨슈머 그룹 조회
bash-4.4# kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
micro-service-group
컨슈머 그룹 상세 조회
kafka-consumer-groups.sh --bootstrap-server localhost:9092
--group micro-service-group --describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
micro-service-group micro-service-0 2 334 334 0 consumer-micro-service-group-1-xxx /192.168.0.3 consumer-micro-service-group-1-. TOPIC: 토픽 이름
-. PARTITION: consumer group 내의 각 consumer가 할당된 파티션 번호
-. CURRENT-OFFSET: 현재 consumer group의 consumer가 각 파티션에서 마지막으로 offset을 commit한 값
-. LOG-END-OFFSET: producer쪽에서 마지막으로 생성한 레코드의 offset
-. LAG: LOG-END-OFFSET에서 CURRENT-OFFSET를 뺀 값
Replication factor 변경
변경전
bash-4.4# kafka-topics.sh --zookeeper 192.168.0.1:2181 --topic micro-service-0 --describe
Topic: micro-service-0 PartitionCount: 1 ReplicationFactor: 1 Configs:
Topic: micro-service-0 Partition: 0 Leader: 1 Replicas: 1 Isr: 1변경후
// rf.json
{
"version": 1,
"partitions": [{
"topic":"micro-service-0", "partition": 0, "replicas":[1,2]
}]
}bash-4.4# kafka-reassign-partitions.sh --zookeeper 192.168.0.1:2181 --reassignment-json-file rf.json --execute
Warning: --zookeeper is deprecated, and will be removed in a future version of Kafka.
Current partition replica assignment
{"version":1,"partitions":[{"topic":"micro-service-0","partition":0,"replicas":[1],"log_dirs":["any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignment for micro-service-0bash-4.4# kafka-topics.sh --zookeeper 192.168.0.1:2181 --topic micro-service-0 --describe
Topic: micro-service-0 PartitionCount: 1 ReplicationFactor: 2 Configs:
Topic: micro-service-0 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2파티션 개수 증가
bash-4.4# kafka-topics.sh --alter --topic micro-service-0 --partitions 4 --zookeeper 192.168.0.1:2181
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
bash-4.4# kafka-topics.sh --zookeeper 192.168.0.1:2181 --topic micro-service-0 --describe
Topic: micro-service-0 PartitionCount: 4 ReplicationFactor: 2 Configs:
Topic: micro-service-0 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: micro-service-0 Partition: 1 Leader: 2 Replicas: 2,1 Isr: 2,1
Topic: micro-service-0 Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: micro-service-0 Partition: 3 Leader: 2 Replicas: 2,1 Isr: 2,1
'빅데이터 > Kafka Cluster' 카테고리의 다른 글
| Kafka restart과 retention (0) | 2022.02.28 |
|---|---|
| Kafka 스트레스 테스트 (0) | 2022.02.28 |
| Kafka Consumer/Producer (0) | 2022.02.28 |
| Kafka Overview (0) | 2022.02.28 |
| Kafka cluster failover (0) | 2022.02.28 |
