首页 > 代码库 > java调用API操作HDFS

java调用API操作HDFS

2024-11-24 02:29:39   202人阅读

java调用API操作HDFS

本文介绍Java调用API从hdfs读取数据

package mongodb;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URI;
import java.util.Arrays;

import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.compress.CompressionCodec;
import org.apache.hadoop.io.compress.CompressionCodecFactory;
import org.apache.hadoop.io.compress.CompressionInputStream;
import org.apache.hadoop.util.ReflectionUtils;


class Item implements Comparable
{
	String value;
	double weight;
	public Item(String v)
	{
		value=http://www.mamicode.com/v;>


下面是一个shell脚本，用Java命令执行上面的程序的话，需要加载各种Hadoop相关的jar包。
（PS：后来发现有种更好的方法，就是用Hadoop命令执行，因为用Hadoop命令执行，Hadoop自己会加载一些jar包，无需自己再手动加载）
hd_core="/usr/local/hadoop/hadoop-core-1.1.2.jar";
s4j="/usr/local/hadoop/lib/slf4j-log4j12-1.6.1.jar";
s4japi="/usr/local/hadoop/lib/slf4j-api-1.6.1.jar";
log4j="/usr/local/hadoop/lib/log4j-1.2.17.jar";
guva="/usr/local/hadoop/lib/guava-11.0.2.jar";
clog="/usr/local/hadoop/lib/commons-logging-1.1.1.jar";
cconf="/usr/local/hadoop/lib/commons-configuration-1.6.jar";
cl="/usr/local/hadoop/lib/commons-lang-2.5.jar";
ccli="/usr/local/hadoop/lib/commons-cli-1.2.jar";
protobuf="/usr/local/hadoop/lib/protobuf-java-2.4.0a.jar";
hdfs="/usr/local/hadoop/lib/hadoop-hdfs-1.1.2.jar";
mongodb="/data/home/liulian/linger/jars/mongo-java-driver-2.12.4.jar";
libs=(
$hd_core
$s4j
$s4japi
$log4j
$guva
$clog
$cconf
$cl
$ccli
$protobuf
$hdfs
$mongodb
)
libstr=""
for jarlib in ${libs[@]};
do
    libstr=${jarlib}":"${libstr}
done
echo $libstr;
java -Xbootclasspath/a:${libstr}  -jar ../jars/updateSim.jar hdfs://10.200.91.164:9000 tv_sim/result/000000_0 tvItemsimColl



本文链接：http://blog.csdn.net/lingerlanlan/article/details/42178675
本文作者：linger



java调用API操作HDFS


联系
我们