HDFS-Could not obtain block

?

MapReduce?Total cumulative CPU time: 33 seconds 380 msec

Ended Job = job_201308291142_4635 with errors

Error?during job, obtaining debugging information...

Job Tracking URL:?http://xxx?/jobdetails.jsp?jobid=job_201308291142_4635

Examining task ID: task_201308291142_4635_m_000019 (and more) from job job_201308291142_4635

Examining task ID: task_201308291142_4635_m_000007 m(and more) from job job_201308291142_4635

Examining task ID: task_201308291142_4635_m_000009 (and more) from job job_201308291142_4635

?

Task with the most failures(5):

-----

Task ID:

? task_201308291142_4635_m_000009

?

URL:

??http://xxxxxxx:50030/taskdetails.jsp?jobid=job_201308291142_4635&tipid=task_201308291142_4635_m_000009

-----

Diagnostic Messages for this Task:

java.io.IOException:?java.io.IOException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1555036314-10.115.5.16-1375773346340:blk_-2678705702538243931_541142 file=/user/hive/warehouse/playtime/dt=20131119/access_pt.log.2013111904.log

? ? ? ? at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)

? ? ? ? at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)

? ? ? ? at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:330)

? ? ? ? at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:246)

? ? ? ? at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:215)

? ? ? ? at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:200)

? ? ? ? at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)

? ? ? ? at org.apache.hadoop.mapred.MapTask.runOldMa?

?

• ? Reson
• ?Solution?

? ? ?　HDFS?FILE?

? ? ? ? ? ? - If?HDFS?block is missing?

? ? ? ? ?1. confirm status

? ? ? ? ? ? ? Confirm missing block is exit or not.

? ? ? ? ? ? ? If missing block is over 1, file is not able to read.?

?$?hadoop?dfsadmin -report

?

?Configured Capacity: 411114887479296 (373.91 TB)

Present Capacity: 411091477784158 (373.89 TB)

DFS Remaining: 411068945908611 (373.87 TB)

DFS Used: 22531875547 (20.98 GB)

DFS Used%: 0.01%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

?

-------------------------------------------------

Datanodes available: 20 (20 total, 0 dead)

?

? ? ? ? ?? ? 2. detail block file

? ? ? ? ? ? ? ?hadoop fsck

? ? ??hadoop?fsck?/ -files -blocks

? ??

...

Status: HEALTHY

?Total size: ? ?4056908575 B (Total open files size: 3505453 B)

?Total dirs: ? ?533

?Total files: ? 15525 (Files currently being written: 2)

?Total blocks (validated): ?15479 (avg. block size 262091 B) (Total open file blocks (not validated): 2)

?Minimally replicated blocks: ? 15479 (100.0 %)

?Over-replicated blocks: ? ?0 (0.0 %)

?Under-replicated blocks: ? 0 (0.0 %)

?Mis-replicated blocks: ? ? 0 (0.0 %)

?Default replication factor: ? ?3

?Average block replication: 3.0094967

?Corrupt blocks: ? ? ? ?0

?Missing replicas: ? ? ?0 (0.0 %)

?Number of data-nodes: ? ? ?20

?Number of racks: ? ? ? 1

FSCK?ended at Tue Nov 19 10:17:19 KST 2013 in 351 milliseconds

?

The filesystem under path ‘/‘ is HEALTHY

?

? ? ? ? ? ? 3. ?remove corrupted file

?$?hadoop?fsck?-delete

?

.....

.........................Status: HEALTHY

?Total size: ? ?4062473881 B (Total open files size: 3505453 B)

?Total dirs: ? ?533

?Total files: ? 15525 (Files currently being written: 2)

?Total blocks (validated): ? ? ?15479 (avg. block size 262450 B) (Total open file blocks (not validated): 2)

?Minimally replicated blocks: ? 15479 (100.0 %)

?Over-replicated blocks: ? ? ? ?0 (0.0 %)

?Under-replicated blocks: ? ? ? 0 (0.0 %)

?Mis-replicated blocks: ? ? ? ? 0 (0.0 %)

?Default replication factor: ? ?3

?Average block replication: ? ? 3.0094967

?Corrupt blocks: ? ? ? ? ? ? ? ?0

?Missing replicas: ? ? ? ? ? ? ?0 (0.0 %)

?Number of data-nodes: ? ? ? ? ?20

?Number of racks: ? ? ? ? ? ? ? 1

FSCK?ended at Tue Nov 19 10:21:41 KST 2013 in 294 milliseconds

?

?

The filesystem under path ‘/‘ is HEALTHY

? ? ?

? ? ? ? ? ?HIVE FILE?

? ? ? ? ? ? ? ?- ?If hive block is missing?

? ? ? ?alter?table drop partition?

?

hadoop 突然断电数据丢失问题