首页 > 代码库 > HDFS 搭建记录
HDFS 搭建记录
1. 三台服务：
172.17.0.62（namenode）
172.17.0.68（datanode）
172.17.0.76（datanode）
/etc/hosts包含的内容：
三台都包含的域名映射：（注意：域名中不要有下划线（_），否则 datanode起不来，呵呵：。。）
172.17.0.76 tomcattomcatmobileplatformtest.bjdv #不要有下划线（_）
172.17.0.62 tomcatmobileplatform.bjdv
172.17.0.68 mobileplatformweblogic.bjdv
#各个服务器映射自己的主机名
127.0.0.1 cd6d8c70882c
2. namenode与datanode配置互相的不用密码的ssh，以及登陆localhost的不用密码的ssh
每台服务器上执行一下：#localhost的ssh
1）ssh-keygen -t dsa -P ‘‘ -f /root/.ssh/id_dsa
2）cat /root/id_dsa.pub >> /root/.ssh/authorized_keys
3）chmod 0600 ~/.ssh/authorized_keys
把namenode和datanode上的id_dsa.pub拷到对方，并在对方上执行一下2，3 #namenode与datanode互相的ssh
3.下载并拷贝hadoop(本文使用的是hadoop-2.7.2)到三台服务器。
4.配置文件：
etc/hadoop/core-site.xml：
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="http://www.mamicode.com/configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration><property> <name>fs.defaultFS</name> <value>hdfs://172.17.0.62:9000</value></property></configuration>
etc/hadoop/hadoop-env.sh：
加入JAVA_HOME,HADOOP_PID_DIR
export JAVA_HOME=/root/jdk1.7.0_60export HADOOP_PID_DIR=/george/h/pidsetc/hadoop/hdfs-site.xml:
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="http://www.mamicode.com/configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/george/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/george/datanode</value> </property></configuration>
etc/hadoop/slaves:
172.17.0.68172.17.0.76
namenode服务器上：
# sbin/start-dfs.sh
检查服务
# jps27831 Jps27353 NameNode27584 SecondaryNameNode
# bin/hdfs dfsadmin -report-bash: ./hadoop-2.7.2/bin/hdfs: No such file or directoryroot@acc82464149d:/george/hadoop-2.7.2# ./bin/hdfs dfsadmin -reportConfigured Capacity: 20869324800 (19.44 GB)Present Capacity: 14557261824 (13.56 GB)DFS Remaining: 14557155328 (13.56 GB)DFS Used: 106496 (104 KB)DFS Used%: 0.00%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0-------------------------------------------------Live datanodes (2):Name: 172.17.0.68:50010 (mobileplatformweblogic.bjdv)Hostname: localhostDecommission Status : NormalConfigured Capacity: 10434662400 (9.72 GB)DFS Used: 49152 (48 KB)Non DFS Used: 4756729856 (4.43 GB)DFS Remaining: 5677883392 (5.29 GB)DFS Used%: 0.00%DFS Remaining%: 54.41%Configured Cache Capacity: 0 (0 B)Cache Used: 0 (0 B)Cache Remaining: 0 (0 B)Cache Used%: 100.00%Cache Remaining%: 0.00%Xceivers: 1Last contact: Mon Aug 29 14:55:37 CST 2016Name: 172.17.0.76:50010 (tomcattomcatmobileplatformtest.bjdv)Hostname: localhostDecommission Status : NormalConfigured Capacity: 10434662400 (9.72 GB)DFS Used: 57344 (56 KB)Non DFS Used: 1555333120 (1.45 GB)DFS Remaining: 8879271936 (8.27 GB)DFS Used%: 0.00%DFS Remaining%: 85.09%Configured Cache Capacity: 0 (0 B)Cache Used: 0 (0 B)Cache Remaining: 0 (0 B)Cache Used%: 100.00%Cache Remaining%: 0.00%Xceivers: 1Last contact: Mon Aug 29 14:55:38 CST 2016（ NameNode 的webUI信息 似乎不正确，只能看到一个Datanode， http://172.17.0.62:50070/dfshealth.html， 上面的 bin/hdfs dfsadmin -report 是正确的）
测试：
# bin/hadoop fs -put LICENSE.txt /
datanode服务器上：
检查服务
# jps 21622 DataNode21774 Jps
日志查看：
# tail -f logs/hadoop-root-datanode-cd6d8c70882c.log
datanode上：
没有压缩的情况下，往hdfs上传文件后在其中的一个datanode上可以看到（如：blk_1073741828），dfs.replication设置为1，所以在所有datanode上只有一份文件。
# ls -l datanode/current/BP-333745209-172.17.0.62-1472441549732/current/finalized/subdir0/subdir0/total 8-rw-r--r-- 1 root root 1366 Aug 29 14:55 blk_1073741828-rw-r--r-- 1 root root 19 Aug 29 14:55 blk_1073741828_1004.meta
以上是自己搭建hdfs时测试与总结，有不对的地方，欢迎指正。
HDFS 搭建记录