首页 > 代码库 > log4j实时将数据写入到kafka,Demo和相关的配置详解
log4j实时将数据写入到kafka,Demo和相关的配置详解
一:在项目中引入对应的JAR包,如下,注意对应的包与之前包的冲突
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.10</version>
</dependency>
<dependency>
<groupId>org.apache.flume</groupId>
<artifactId>flume-ng-core</artifactId>
<version>1.7.0</version>
</dependency>
<dependency>
<groupId>org.apache.flume.flume-ng-clients</groupId>
<artifactId>flume-ng-log4jappender</artifactId>
<version>1.7.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-log4j-appender</artifactId>
<version>0.10.2.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.5</version>
</dependency>
</dependencies>
二:log4j配置文件配置
log4j.rootLogger=INFO, stdout, R, L,flume,KAFKA log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flume.Hostname = 127.0.0.1log4j.appender.flume.Port = 5555log4j.appender.flume.UnsafeMode = true## appender KAFKAlog4j.appender.KAFKA=kafka.producer.KafkaLog4jAppenderlog4j.appender.KAFKA.topic=testsonglog4j.appender.KAFKA.brokerList=127.0.0.1:9092log4j.appender.KAFKA.pa=127.0.0.1:9092log4j.appender.KAFKA.compressionType=nonelog4j.appender.KAFKA.syncSend=truelog4j.appender.KAFKA.layout=org.apache.log4j.PatternLayoutlog4j.appender.KAFKA.ThresholdFilter.level=INFOlog4j.appender.KAFKA.ThresholdFilter.onMatch=ACCEPTlog4j.appender.KAFKA.ThresholdFilter.onMismatch=DENYlog4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n## appender consolelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.outlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n
三:测试代码
package com.songflume.flume;import java.util.Date;import org.apache.log4j.Logger;public class WriteLog { private static Logger logger = Logger.getLogger(WriteLog.class); public static void main(String[] args) throws InterruptedException { logger.debug("This is debug message."); logger.info("This is info message."); logger.error("This is error message."); int i = 0; while (i < 1000) { logger.debug(new Date().getTime()); logger.info("songceshi" + i); Thread.sleep(100); i++; } }}
四:kafka消费测试结果(由于是,支持了双通道 :一种是将数据打入到Flume然后再写入kafka,另一种是log4j直接写入kafka),所以kafka消费端出现了2次数据。
log4j实时将数据写入到kafka,Demo和相关的配置详解