首页 > 代码库 > Centos6.5下kafka部署安装

Centos6.5下kafka部署安装

2024-08-16 06:02:40   222人阅读

一）安装JDK


步骤1：新建路径/usr/Java, 并在其下解压 jdk-7u71-linux-x64.tar.gz


# mkdir -p /usr/java

# tar -zxvf jdk-7u71-linux-x64.tar.gz


步骤2：添加JDK到系统环境变量


# vi /etc/profile

新增以下内容：

export JAVA_HOME=/usr/java/jdk1.7.0_71

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar


步骤3：使配置生效

source /etc/profile #使配置文件立即生效


# java -version       //JDK版本java version "1.7.0_79" 也可以，目前JDK1.8版本不支持；

Java version "1.7.0_71"

Java(TM) SE Runtime Environment (build 1.7.0_71-b14)

Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)


二）安装zookeeper


步骤1：解压zookeeper安装包(路径无关，安装路径可以自定义)


# tar -zxvf zookeeper-3.3.6.tar.gz


步骤2：配置环境变量


# vim /etc/profile 

export zookeeper_home=/home/laoyang/zookeeper


#source /etc/profile


# sh $zookeeper_home/bin/zkServer.sh start

JMX enabled by default

Using config: /home/laoyang/zookeeper/bin/../conf/zoo.cfg

grep: /home/laoyang/zookeeper/bin/../conf/zoo.cfg: No such file or directory

Starting zookeeper ... STARTED


三）安装kafka


步骤1：解压&移动到/usr/local


# tar -xzvf kafka_2.11-0.9.0.1.tgz

# mv kafka_2.11-0.9.0.1 /usr/local/kafka


步骤2：配置kafka，修改server.properties


# mkdir /usr/local/kafka/log/kafka #创建kafka日志目录

# cd /usr/local/kafka/config #进入配置目录

# vim server.properties #编辑修改相应的参数

  1. broker.id=0

  2. port=9092 #端口号

  3. host.name=10.8.5.101  #服务器IP地址，修改为自己的服务器IP

  4. log.dirs=/usr/local/kafka/log/kafka  #日志存放路径，上面创建的目录

  5. zookeeper.connect=localhost:2181  #zookeeper地址和端口，单机配置部署，localhost:2181



步骤3、配置kafka下的zookeeper


# mkdir /usr/local/kafka/zookeeper #创建zookeeper目录

# mkdir /usr/local/kafka/log/zookeeper #创建zookeeper日志目录

# cd /usr/local/kafka/config #进入配置目录

# vim zookeeper.properties #编辑修改相应的参数

  1. dataDir=/usr/local/kafka/zookeeper #zookeeper数据目录

  2. dataLogDir=/usr/local/kafka/log/zookeeper #zookeeper日志目录

  3. clientPort=2181

  4. maxClientCnxns=100

  5. tickTime=2000

  6. initLimit=10

  7. syncLimit=5


步骤4：创建启动、关闭kafka脚本


1）创建启动kafka脚本

#cd  /usr/local/kafka/bin/

# vim kafkastart.sh

  1. #!/bin/bash

  2. #启动zookeeper

  3. /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &

  4. sleep 3

  5. #Start kafka

  6. /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &


2）创建关闭kafka脚本


# vim kafkastop.sh

  1. #!/bin/bash

  2. #stop zookeeper

  3. /usr/local/kafka/bin/zookeeper-server-stop.sh /usr/local/kafka/config/zookeeper.properties &

  4. sleep 3

  5. #Stop kafka

  6. /usr/local/kafka/bin/kafka-server-stop.sh /usr/local/kafka/config/server.properties &


3）#添加脚本执行权限


chmod +x kafkastart.sh

chmod +x kafkastop.sh


4）设置脚本开机自动执行


# vim /etc/rc.d/rc.local #编辑，在最后添加一行

  1. sh /usr/local/kafka/kafkastart.sh & ＃设置开机自动在后台运行脚本

  2. sh /usr/local/kafka/kafkastart.sh #启动kafka

  3. sh /usr/local/kafka/kafkastop.sh #关闭kafka


到此为止CentOs6.5下Kafka安装配置完成。


成功标志：


四）测试kafka


步骤1：启动zookeeper


# sh $zookeeper_home/bin/zkServer.sh start

JMX enabled by default

Using config: /home/laoyang/zookeeper/bin/../conf/zoo.cfg

grep: /home/laoyang/zookeeper/bin/../conf/zoo.cfg: No such file or directory

Starting zookeeper ... STARTED


步骤2：启动kafka


# ./kafkastart.sh


步骤3：测试创建topic


# cd /usr/local/kafka/bin

# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test


步骤4：通过list命令查看创建的topic


# cd /usr/local/kafka/bin

# ./kafka-topics.sh –list –zookeeper localhost:2181


步骤5：生产消息测试


# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test

Kafka test!    //这里的“Kafka test!”为手动输入的，输入后“ctrl +c”退出；


步骤6：消费消息测试


# ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

I‘m laoyang  #之前测试输入的内容

kafka test!


经过上面几步，代表kafka安装成功。


五）停止kafka

步骤1：停止Kafka


cd /usr/local/kafka

# ./kafkastop.sh   //这里的kafka进程并没有关掉，需要手动kill


步骤2：停止Zookeeper server


# sh $zookeeper_home/bin/zkServer.sh stop


至此，启动，测试，停止全过程ok。


kafka启动过程中的报错：

 WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)

EndOfStreamException: Unable to read additional data from client sessionid 0x157ac94cd940002, likely client has closed socket

    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)

    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

    at java.lang.Thread.run(Thread.java:745)


报错原因：

          zookeeper服务端和客户端的jar版本不一致导致的 ；

解决办法：

          从zookeeper服务端安装目录的根目录复制zookeeper-xxx.jar到kafka安装目录下的libs目录下即可，将kafka下libs目录下的jar包删除；


报错：

java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/1. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
        at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:205)
        at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:57)
        at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:44)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:103)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
        at kafka.Kafka$.main(Kafka.scala:46)
        at kafka.Kafka.main(Kafka.scala)


报错原因：

          该错误是由于server.properties 中的broker.id 重复


解决办法：

# vim server.properties

# The id of the broker. This must be set to a unique integer for each broker.

broker.id=0


本文出自 “11009859” 博客，请务必保留此出处http://11019859.blog.51cto.com/11009859/1860213

Centos6.5下kafka部署安装


联系
我们