首页 > 代码库 > log4j实时将数据写入到kafka,Demo和相关的配置详解

log4j实时将数据写入到kafka,Demo和相关的配置详解

一:在项目中引入对应的JAR包,如下,注意对应的包与之前包的冲突

<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.10</version>
</dependency>
<dependency>
<groupId>org.apache.flume</groupId>
<artifactId>flume-ng-core</artifactId>
<version>1.7.0</version>
</dependency>
<dependency>
<groupId>org.apache.flume.flume-ng-clients</groupId>
<artifactId>flume-ng-log4jappender</artifactId>
<version>1.7.0</version>
</dependency>

<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.2</version>

</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-log4j-appender</artifactId>
<version>0.10.2.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.5</version>
</dependency>

</dependencies>

 

二:log4j配置文件配置

 log4j.rootLogger=INFO, stdout, R, L,flume,KAFKA  log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flume.Hostname = 127.0.0.1log4j.appender.flume.Port = 5555log4j.appender.flume.UnsafeMode = true## appender KAFKAlog4j.appender.KAFKA=kafka.producer.KafkaLog4jAppenderlog4j.appender.KAFKA.topic=testsonglog4j.appender.KAFKA.brokerList=127.0.0.1:9092log4j.appender.KAFKA.pa=127.0.0.1:9092log4j.appender.KAFKA.compressionType=nonelog4j.appender.KAFKA.syncSend=truelog4j.appender.KAFKA.layout=org.apache.log4j.PatternLayoutlog4j.appender.KAFKA.ThresholdFilter.level=INFOlog4j.appender.KAFKA.ThresholdFilter.onMatch=ACCEPTlog4j.appender.KAFKA.ThresholdFilter.onMismatch=DENYlog4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n## appender consolelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.outlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n

三:测试代码

package com.songflume.flume;import java.util.Date;import org.apache.log4j.Logger;public class WriteLog {    private static Logger logger = Logger.getLogger(WriteLog.class);    public static void main(String[] args) throws InterruptedException {                logger.debug("This is debug message.");                logger.info("This is info message.");                logger.error("This is error message.");        int i = 0;        while (i < 1000) {            logger.debug(new Date().getTime());            logger.info("songceshi" + i);            Thread.sleep(100);            i++;        }    }}

四:kafka消费测试结果(由于是,支持了双通道 :一种是将数据打入到Flume然后再写入kafka,另一种是log4j直接写入kafka),所以kafka消费端出现了2次数据。

技术分享

log4j实时将数据写入到kafka,Demo和相关的配置详解