数据平台安装配置-Kafka

Posted on   |   In

前提: 对应的 Scala 版本已安装, ZooKeeper 已启动.

Kafka 安装

解压压缩文件到 ~/workspace 下

修改环境变量

1
2
3
4
$ vim ~/.bashrc
    export KAFKA_HOME=/home/hadoop/workspace/kafka_2.11-0.8.2.2 
    export CLASSPATH=.:$KAFKA_HOME/lib:$CLASSPATH
    export PATH=$KAFKA_HOME/bin:$PATH

配置生效:

1
source ~/.bashrc

修改 Kafka 配置

  1. ~/workspace/kafka_2.11-0.8.2.2/config
    1
    2
    3
    4
    5
    6
    7
    8
    9
    $ vim server.properties     #添加配置内容
        
        broker.id=1 # 对应集群中唯一数字
        port=9092
        host.name=h16 # hostname
        advertised.host.name=h16
        log.dirs=/home/data/kafka/kafka-logs
        num.partitions=2
        zookeeper.connect=h16:2181,h17:2181,h18:2181

Kafka 启动

在 h16 上启动 Kafka

1
2
bin/kafka-server-start.sh config/server.properties (正常启动)
bin/kafka-server-start.sh -daemon config/server.properties (后台启动)

验证启动情况, 在服务器上查看:

1
2
3
jps  # h16
11013 Kafka
561 Jps

运行示例

  1. 创建topic:

    1
    bin/kafka-topics.sh --create --zookeeper h16::2181 --replication-factor 3 --partitions 1 --topic mykafka

  2. 查看Topic:

    1
    bin/kafka-topics.sh --list --zookeeper h16::2181

  3. 查看详细信息:

    1
    2
    3
    4
    bin/kafktopics.sh --describe --zookeeper h16:2181
    Topic:mykafka	PartitionCount:1	ReplicationFactor:3	Configs:
    Topic: mykafka	Partition: 0	Leader: 133	Replicas: 133,134,132	Isr: 134

  4. 发送消息:

    1
    2
    3
    4
    bin/kafka-console-producer.sh --broker-list 192.168.40.134:9092 --topic mykafka
    23423
    bin/kafka-console-producer.sh --brokelist 192.168.40.134:9092 --topic mykafka
    4533

    如果出现以下信息

    1
    2
    3
    SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".  
    SLF4J: Defaulting to no-operation (NOP) logger implementation  
    SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

    则需要下载slftj-nop-1.5.jar, 并将其cp至kafka的libs目录下。

  5. 接收消息(注意–zookeeper配置与server.properties保持一致, 使用域名方式):

    1
    bin/kafka-console-consumer.sh --zookeeper h16:2181 --topic mykafka --from-beginning

  6. 更改topic过期时间,alter只对Topic-level级别的配置有效参考

    1
    2
    3
    bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic withdata --config retention.ms=86400000  (一天)
    bin/kafka-topics.sh --zookeeper localhost:2181 --describe -topic withdata
    bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic withdata --deleteConfig retention.ms

参考资料

Kafka 安装配置