Kafka 자주 사용하는 명령

Kafka shell scripts

kafka-topics.sh : 토픽 생성, 조회, 수정 등 역할

• --bootstrap-server : 토픽관련 명령어를 수행할 대상 카프카 클러스터
• --replication-factor : 레플리카 개수 지정
• --partitions : 파티션 개수 설정
• --config : 각종 토픽 설정 가능(retention.ms, segment.byte 등)
• --create : 토픽 생성
• --delete : 토픽 제거
• --describe: 토픽 상세 확인
• --list : 카프카 클러스터의 토픽 리스트 확인
• --version : 대상 카프카 클러스터 버전 확인

kafka-console-consumer.sh : 토픽의 레코드 즉시 조회

kafka-console-producer.sh : 토픽의 레코드를 전달

kafka-consumer-groups.sh : 컨슈머그룹 조회, 컨슈머 오프셋 확인, 수정

• --bootstrap-server
• --describe
• --group

 

자주 사용하는 명령

토픽 생성

kafka-topics.sh --create 
    --zookeeper localhost:2181 
    --replication-factor 1 
    --partitions 1 
    --topic mytopic

 

토픽 리스트 조회

bash-4.4# kafka-topics.sh --list --zookeeper 192.168.0.1:2181
__consumer_offsets
micro-service-topic-0
micro-service-topic-1

 

토픽 정보 조회

bash-4.4# kafka-topics.sh --zookeeper 192.168.0.1:2181 --topic micro-service-0 --describe
Topic: micro-service-0	PartitionCount: 2	ReplicationFactor: 2	Configs: retention.ms=1000
	Topic: micro-service-0	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2
	Topic: micro-service-0	Partition: 1	Leader: 2	Replicas: 2,1	Isr: 1,2

 

Retention 변경

kafka-topics.sh --zookeeper 192.168.0.1:2181 --alter 
    --topic micro-service-0 
    --config retention.ms=1000

 

컨슈머 그룹 조회

bash-4.4# kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
micro-service-group

 

컨슈머 그룹 상세 조회

kafka-consumer-groups.sh --bootstrap-server localhost:9092 
   --group micro-service-group --describe
   
GROUP                      TOPIC              PARTITION  CURRENT-OFFSET  LOG-END-OFFSET LAG  CONSUMER-ID                            HOST            CLIENT-ID
micro-service-group        micro-service-0    2          334             334            0    consumer-micro-service-group-1-xxx     /192.168.0.3    consumer-micro-service-group-1

-. TOPIC: 토픽 이름
-. PARTITION: consumer group 내의 각 consumer가 할당된 파티션 번호
-. CURRENT-OFFSET: 현재 consumer group의 consumer가 각 파티션에서 마지막으로 offset을 commit한 값
-. LOG-END-OFFSET: producer쪽에서 마지막으로 생성한 레코드의 offset
-. LAG: LOG-END-OFFSET에서 CURRENT-OFFSET를 뺀 값

 

Replication factor 변경

변경전

bash-4.4# kafka-topics.sh --zookeeper 192.168.0.1:2181 --topic micro-service-0 --describe
Topic: micro-service-0	PartitionCount: 1	ReplicationFactor: 1	Configs: 
	Topic: micro-service-0	Partition: 0	Leader: 1	Replicas: 1	Isr: 1

변경후

// rf.json
{
   "version": 1,
   "partitions": [{
       "topic":"micro-service-0", "partition": 0, "replicas":[1,2]
    }]
}

bash-4.4# kafka-reassign-partitions.sh --zookeeper 192.168.0.1:2181 --reassignment-json-file rf.json --execute
Warning: --zookeeper is deprecated, and will be removed in a future version of Kafka.
Current partition replica assignment

{"version":1,"partitions":[{"topic":"micro-service-0","partition":0,"replicas":[1],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignment for micro-service-0

bash-4.4# kafka-topics.sh --zookeeper 192.168.0.1:2181 --topic micro-service-0 --describe
Topic: micro-service-0	PartitionCount: 1	ReplicationFactor: 2	Configs: 
	Topic: micro-service-0	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

 

파티션 개수 증가

bash-4.4# kafka-topics.sh --alter --topic micro-service-0 --partitions 4 --zookeeper 192.168.0.1:2181
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
bash-4.4# kafka-topics.sh --zookeeper 192.168.0.1:2181 --topic micro-service-0 --describe
Topic: micro-service-0	PartitionCount: 4	ReplicationFactor: 2	Configs: 
	Topic: micro-service-0	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2
	Topic: micro-service-0	Partition: 1	Leader: 2	Replicas: 2,1	Isr: 2,1
	Topic: micro-service-0	Partition: 2	Leader: 1	Replicas: 1,2	Isr: 1,2
	Topic: micro-service-0	Partition: 3	Leader: 2	Replicas: 2,1	Isr: 2,1