首页 > 代码库 > Hadoop HDFS文件常用操作及注意事项
Hadoop HDFS文件常用操作及注意事项
1.Copy a file from the local file system to HDFS
The srcFile variable needs to contain the full name (path + file name) of the file in the local file system.
The dstFile variable needs to contain the desired full name of the file in the Hadoop file system.
1 Configuration config = new Configuration();2 FileSystem hdfs = FileSystem.get(config);3 Path srcPath = new Path(srcFile);4 Path dstPath = new Path(dstFile);5 hdfs.copyFromLocalFile(srcPath, dstPath);
2.Create HDFS file
The fileName variable contains the file name and path in the Hadoop file system.
The content of the file is the buff variable which is an array of bytes.
1 //byte[] buff - The content of the file 2 //创建了一个HDFS文件，并且把buff数组的内容写到了HDFS文件中。3 Configuration config = new Configuration();4 FileSystem hdfs = FileSystem.get(config);5 Path path = new Path(fileName);6 FSDataOutputStream outputStream = hdfs.create(path);7 outputStream.write(buff, 0, buff.length);
3.Rename HDFS file
In order to rename a file in Hadoop file system, we need the full name (path + name) of
the file we want to rename. The rename method returns true if the file was renamed, otherwise false.
1 Configuration config = new Configuration();2 FileSystem hdfs = FileSystem.get(config);3 Path fromPath = new Path(fromFileName);4 Path toPath = new Path(toFileName);5 boolean isRenamed = hdfs.rename(fromPath, toPath);
4.Delete HDFS file
In order to delete a file in Hadoop file system, we need the full name (path + name)
of the file we want to delete. The delete method returns true if the file was deleted, otherwise false.
1 Configuration config = new Configuration(); 2 FileSystem hdfs = FileSystem.get(config); 3 Path path = new Path(fileName); 4 boolean isDeleted = hdfs.delete(path, false); 5 6 //Recursive delete:估计true是递归删除该目录下面的文件。 7 Configuration config = new Configuration(); 8 FileSystem hdfs = FileSystem.get(config); 9 Path path = new Path(fileName);10 boolean isDeleted = hdfs.delete(path, true);
5.Get HDFS file last modification time
In order to get the last modification time of a file in Hadoop file system,
we need the full name (path + name) of the file.
1 Configuration config = new Configuration();2 FileSystem hdfs = FileSystem.get(config);3 Path path = new Path(fileName);4 FileStatus fileStatus = hdfs.getFileStatus(path);5 long modificationTime = fileStatus.getModificationTime
6.Check if a file exists in HDFS
In order to check the existance of a file in Hadoop file system,
we need the full name (path + name) of the file we want to check.
The exists methods returns true if the file exists, otherwise false.
1 Configuration config = new Configuration();2 FileSystem hdfs = FileSystem.get(config);3 Path path = new Path(fileName);4 boolean isExists = hdfs.exists(path);
7.Get the locations of a file in the HDFS cluster
A file can exist on more than one node in the Hadoop file system cluster for two reasons:
Based on the HDFS cluster configuration, Hadoop saves parts of files on different nodes in the cluster.
Based on the HDFS cluster configuration, Hadoop saves more than one copy of each file on different nodes for redundancy (The default is three).
1 Configuration config = new Configuration(); 2 FileSystem hdfs = FileSystem.get(config); 3 Path path = new Path(fileName); 4 FileStatus fileStatus = hdfs.getFileStatus(path); 5 BlockLocation[] blkLocations = hdfs.getFileBlockLocations(fileStatus, 0, fileStatus.getLen()); 6 int blkCount = blkLocations.length; 7 for (int i = 0; i < blkCount; i++) { 8 String[] hosts = blkLocations[i].getHosts(); 9 // Do something with the block hosts10 }
8. Get a list of all the nodes host names in the HDFS cluster
his method casts the FileSystem Object to a DistributedFileSystem Object.
This method will work only when Hadoop is configured as a cluster.
Running Hadoop on the local machine only, in a non cluster configuration will cause this method to throw an Exception.
1 Configuration config = new Configuration();2 FileSystem fs = FileSystem.get(config);3 DistributedFileSystem hdfs = (DistributedFileSystem) fs;4 DatanodeInfo[] dataNodeStats = hdfs.getDataNodeStats();5 String[] names = new String[dataNodeStats.length];6 for (int i = 0; i < dataNodeStats.length; i++) {7 names[i] = dataNodeStats[i].getHostName();8 }
遇到的问题及解决：
1.文件append的问题
hadoop的版本1.0.4以后，API中已经有了追加写入的功能，但不建议在生产环境中使用，原因如下：Does HDFS allow appends to files? This is currently set to false because there are bugs in the "append code" and is not supported in any prodction cluster.
如果要测试的话，需要把dfs.support.appen的参数设置为true，不然客户端写入的时候会报错：
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: Append to hdfs not supported. Please refer to dfs.support.append configuration parameter.
解决：修改namenode节点上的hdfs-site.xml。
<property> <name>dfs.support.append</name> <value>true</value> </property>
2.在Hadoop-1.0.4和Hadoop-2.2的使用append时，需求：追加写入文件，如果文件不存在，需求先创建。
异常：
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /huangq/dailyRolling/mommy-dailyRolling for DFSClient_-1456545217 on client 10.1.85.243 because current leaseholder is trying to recreate file. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1374) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1246) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1426) at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:643) at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)代码版本1：（出现上述报错的代码）
1 FileSystem fs = FileSystem.get(conf);2 Path dstPath = new Path(dst);3 if (!fs.exists(dstPath)) {4 fs.create(dstPath);5 } 6 FSDataOutputStream fsout = fs.append(dstPath);7 BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fsout));
异常的原因：fs句柄的原因，改成下面就ok了。
解决：创建完文件之后，关闭流fs，再次获得一次新的fs。
1 FileSystem fs = FileSystem.get(conf);2 Path dstPath = new Path(dst);3 if (!fs.exists(dstPath)) {4 fs.create(dstPath);5 fs.close();6 fs = FileSystem.get(conf);7 } 8 FSDataOutputStream fsout = fs.append(dstPath);