首页 > 代码库 > 史上最牛hadoop2.2伪分布式hdfs数据迁移

史上最牛hadoop2.2伪分布式hdfs数据迁移

2024-08-04 20:16:03   228人阅读

1.需求:

由于hadoop的数据文件所在磁盘出现99%占用,导致hdfs无法上传文件

异常:org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/root/input could only be replicated to 0 nodes, instead of 1

[root@idata-slave3 hadoop]# df -h 

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_greenbigdata4-lv_root

                       50G   49.2G   880M  99% /

tmpfs                 7.8G     0  7.8G   0% /dev/shm

/dev/sda1             485M   64M  396M  14% /boot

/dev/mapper/vg_greenbigdata4-lv_home

                      860G   36G  781G   5% /home

可以看见/下目录50G而/home下860G,所以迁移目录

2.原hadoop配置

<configuration>

<property> 

  <name>fs.defaultFS</name> 

  <value>hdfs://idata-slave3:9010</value> 

</property>

<property> 

  <name>hadoop.tmp.dir</name> 

  <value>/usr/dfs/tmp</value> 

</property> 

</configuration>

3.停掉hadoop集群

Stop-all.sh

4.新hadoop配置

<configuration>

<property> 

  <name>fs.defaultFS</name> 

  <value>hdfs://idata-slave3:9010</value> 

</property>

<property> 

  <name>hadoop.tmp.dir</name> 

  <value>/home/dfs/tmp</value> 

</property> 

</configuration>

5.数据迁移

原始数据目录:

[root@idata-slave3 ~]# cd /usr/dfs/tmp/

[root@idata-slave3 tmp]# pwd

/usr/dfs/tmp

[root@idata-slave3 tmp]# ls

dfs                              hadoop-unjar5031704250755207519  hadoop-unjar7185041879037748832

hadoop-unjar2609195906982650560  hadoop-unjar5034268358752828831  nm-local-dir

新的路径:

[root@idata-slave3 hadoop]# mkdir /home/dfs

复制:

[root@idata-slave3 hadoop]# cp -r /usr/dfs/tmp/ /home/dfs/

6.启动hadoop集群

Start-all.sh

输出信息:

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Starting namenodes on [idata-slave3]

idata-slave3: starting namenode, logging to /usr/lib/hadoop-2.2.0/logs/hadoop-root-namenode-idata-slave3.out

idata-slave3: SLF4J: Class path contains multiple SLF4J bindings.

idata-slave3: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

idata-slave3: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

idata-slave3: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

idata-slave3: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

idata-slave3: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

idata-slave3: starting datanode, logging to /usr/lib/hadoop-2.2.0/logs/hadoop-root-datanode-idata-slave3.out

idata-slave3: SLF4J: Class path contains multiple SLF4J bindings.

idata-slave3: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

idata-slave3: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

idata-slave3: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

idata-slave3: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

idata-slave3: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Starting secondary namenodes [0.0.0.0]

0.0.0.0: reverse mapping checking getaddrinfo for localhost [127.0.0.1] failed - POSSIBLE BREAK-IN ATTEMPT!

0.0.0.0: starting secondarynamenode, logging to /usr/lib/hadoop-2.2.0/logs/hadoop-root-secondarynamenode-idata-slave3.out

0.0.0.0: SLF4J: Class path contains multiple SLF4J bindings.

0.0.0.0: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

0.0.0.0: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

0.0.0.0: SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

0.0.0.0: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

0.0.0.0: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

starting yarn daemons

starting resourcemanager, logging to /usr/lib/hadoop-2.2.0/logs/yarn-root-resourcemanager-idata-slave3.out

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

idata-slave3: starting nodemanager, logging to /usr/lib/hadoop-2.2.0/logs/yarn-root-nodemanager-idata-slave3.out

 

启动成功

7.测试

[root@idata-slave3 hadoop]# hdfs dfs -ls /

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Found 6 items

drwxr-xr-x   - root supergroup          0 2014-10-24 15:57 /flume

drwxr-xr-x   - root supergroup          0 2014-11-14 14:41 /hbase

drwxr-xr-x   - root supergroup          0 2014-11-11 12:58 /test

drwxr-xr-x   - root supergroup          0 2014-11-27 17:11 /tmp

drwxr-xr-x   - root supergroup          0 2014-10-11 11:14 /tmp-output101

drwxr-xr-x   - root supergroup          0 2014-10-28 16:02 /user

 

上传:

[root@idata-slave3 ~]# hdfs dfs -put sparkdata/ ./

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

 

drwxr-xr-x   - root supergroup          0 2014-11-04 15:14 _sqoop

drwxr-xr-x   - root supergroup          0 2014-10-23 17:08 flume

drwxr-xr-x   - root supergroup          0 2014-10-16 16:22 forest

drwxr-xr-x   - root supergroup          0 2014-10-16 16:25 od

drwxr-xr-x   - root supergroup          0 2014-11-11 15:10 pfp

drwxr-xr-x   - root supergroup          0 2014-11-07 16:07 recommend

drwxr-xr-x   - root supergroup          0 2014-11-27 17:44 sparkdata

drwxr-xr-x   - root supergroup          0 2014-11-18 17:19 temp

 

这样数据可以迁移了,不用格式化hdfs,防止丢失数据

史上最牛hadoop2.2伪分布式hdfs数据迁移


联系
我们