首页 > 代码库 > HDFS NFS Gateway

HDFS NFS Gateway

2024-10-22 19:59:39   208人阅读

1、mount hdfs，关闭 linux自带的几个和 hdfs需要启动冲突的服务
        参考:(1) service nfs stop and service rpcbind stop 
  (2) hadoop portmap or hadoop-daemon.sh start portmap


[hsu@server01 mnt]$ service portmap stop 
[hsu@server01 mnt]$ sudo service rpcbind stop
[hsu@server01 mnt]$ sudo hdfs portmap &
[hsu@server01 mnt]$ jobs 验证后台运行任务
[1]+  Running                 sudo hdfs portmap &
[hsu@server01 mnt]$ ps aux | grep portmap
root     224585  0.0  0.0 148020  2068 pts/0    S    14:09   0:00 sudo hdfs portmap

[hsu@server01 /]$ ls -la /sbin/mount.* 查看mount
-rwxr-xr-x. 1 root root  40592 Apr 17  2012 /sbin/mount.cifs
-rwxr-xr-x. 1 root root  11616 Nov  3  2011 /sbin/mount.fuse
-rwsr-xr-x. 1 root root 122880 May 30  2012 /sbin/mount.nfs
  lrwxrwxrwx. 1 root root      9 Jul  2 18:22 /sbin/mount.nfs4 -> mount.nfs
-rwxr-xr-x. 1 root root   1338 Apr  6  2012 /sbin/mount.tmpfs


2、挂在nfs3
       参考：hadoop nfs3  OR   hadoop-daemon.sh start nfs3


[hsu@server01 mnt]$ sudo -u hdfs hdfs nfs3 &后台运行nfs3
[hsu@server01 mnt]$ jobs 验证后台任务
[1]-  Running                 sudo hdfs portmap &
[2]+  Running                 sudo -u hdfs hdfs nfs3 &
[hsu@server01 mnt]$ ps aux | grep nfs3
root     225450  0.0  0.0 148020  2064 pts/0    S    14:14   0:00 sudo -u hdfs hdfs nfs3


3、Execute the following command to verify if all the services are up and running
[hsu@server01 mnt]$ rpcinfo -p server01
   program vers proto   port  service
   100005    2   tcp   4242  mountd
   100000    2   udp    111  portmapper
   100000    2   tcp    111  portmapper
   100005    1   tcp   4242  mountd
   100003    3   tcp   2049  nfs
   100005    1   udp   4242  mountd
   100005    3   udp   4242  mountd
   100005    3   tcp   4242  mountd
   100005    2   udp   4242  mountd
4、Verify if the HDFS namespace is exported and can be mounted
[hsu@server01 mnt]$ showmount -e server01       
Export list for server01:
/ *


5、mount -t nfs -o vers=3,proto=tcp,nolock $server:/  $mount_point


[hsu@server01 /]$ hdfs -help | egrep ‘portmap|nfs3‘
 portmap              run a portmap service
 nfs3                 run an NFS version 3 gateway
[hsu@server01 /]$ sudo mount -t nfs -o vers=3,proto=tcp,nolock server01:/ /mnt/hdfs  挂载成功！
[hsu@server01 /]$ mount | grep hdfs
server01:/ on /mnt/hdfs type nfs (rw,vers=3,proto=tcp,nolock,addr=135.33.5.53)

问题1：查看文件的时候都会显示：14/12/30 14:23:55 INFO nfs3.IdUserGroup: Can‘t map group supergroup. Use its string hashcode:-1710818332 
java.io.IOException: No such group:supergroup。

原因1：nfsServer linux本地没有supergroup这个组？cat /etc/group | grep supergroup

解决1：
[hsu@server01 /]$ sudo umount /mnt/hdfs    取消挂载
[hsu@server01 /]$ sudo which groupadd          
/usr/sbin/groupadd
[hsu@server01 /]$ sudo /usr/sbin/groupadd supergroup
[hsu@server01 /]$ sudo cat /etc/group | grep supergroup
supergroup:x:1005:
[hsu@server01 /]$ sudo mount -t nfs -o vers=3,proto=tcp,nolock server01:/ /mnt/hdfs再次挂载
14/12/30 15:07:35 INFO mount.RpcProgramMountd: Giving handle (fileId:16385) to client for export /
14/12/30 15:07:35 WARN oncrpc.RpcProgram: Invalid RPC call program 100227
14/12/30 15:07:35 INFO nfs3.IdUserGroup: Update cache now
14/12/30 15:07:35 INFO nfs3.IdUserGroup: Not doing static UID/GID mapping because ‘/etc/nfs.map‘ does not exist.
14/12/30 15:07:35 INFO nfs3.IdUserGroup: Updated user map size: 63
14/12/30 15:07:35 INFO nfs3.IdUserGroup: Updated group map size: 89
[hsu@server01 /]$ showmount -e server01
Export list for server01:
/ *
[hsu@server01 /]$ ls /mnt/hdfs/
hbase  system  tmp  user  zeus

6、stop service
(1) hadoop-daemon.sh stop nfs3      
(2) hadoop-daemon.sh stop portmap

7、参考工作参数

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html


大数据讨论群：288396468 

HDFS NFS Gateway


联系
我们