首页 > 代码库 > hadoop挂载多硬盘，ZZ

hadoop挂载多硬盘，ZZ

2024-08-02 23:44:12   217人阅读

hadoop挂载多硬盘 ...multiple disks per node

 multiple disks per node Read more at: http://www.queryhome.com/24784/how-to-set-hadoop-tmp-dir-if-i-have-multiple-disks-per-node
 multiple disks per node Read more at: http://www.queryhome.com/24784/how-to-set-hadoop-tmp-dir-if-i-have-multiple-disks-per-node

http://blog.sina.com.cn/s/blog_b88e09dd01013rd4.html

 

Ubuntu - 硬盘分区、格式化、自动挂载配置 | Hard disk add new partition, format, auto mount in ubuntu 

http://aofengblog.blog.163.com/blog/static/6317021201101502540117/

 

http://my.oschina.net/leejun2005/blog/290073

 

proper-care-and-feeding-of-drives-in-a-hadoop-cluster-a-conversation-with-stackiqs-dr-bruno

http://hortonworks.com/blog/proper-care-and-feeding-of-drives-in-a-hadoop-cluster-a-conversation-with-stackiqs-dr-bruno/

Utilizing-multiple-hard-disks-for-hadoop-HDFS

http://lucene.472066.n3.nabble.com/Utilizing-multiple-hard-disks-for-hadoop-HDFS-td3553851.html

 

=================

First, Hadoop requires at least two locations for storing it’s files: mapred.local.dir, where MapReduce stores intermediary files,

and dfs.data.dir, where HDFS stores the HDFS data (there are other locations as well, like hadoop.tmp.dir, where Hadoop and components stores its temporary data).  

Both of them can cover multiple partitions.  

While the two locations can be placed on physically different partitions, Cloudera recommends to configure them across the same set of partitions to maximize disk-level parallelism (this might not be an issue if the number of disk is much larger than the number of cores).

==

Hadoop多磁碟設定(ubuntu)
Hadoop預設只會使用hadoop資料夾的那個磁碟，然而要使用到所有的磁碟必須要做額外的設定，才可以讓HDFS使用到。設定步驟如下：
  1. 設定新磁碟的權限為777（也許不用這麼高），讓Hadoop擁有讀寫的權限
     sudo chmod -c 777 /media/diskName
  2. 修改conf資料夾內的hdfs-site.xml文件，在<configuration></configuration>之間加入property
<property>
<name>dfs.data.dir</name>  
<value> ~/dfs/data,media/diskName/dfs/data</value>
</property>

多磁碟間的路徑使用半形逗點分隔，並請確定路徑是否正確

最後重新啟動HDFS
============================

Adding disk to the Hadoop Data nodes.(Repeat for all nodes)

http://girishkathalagiri.blogspot.com/2012/09/adding-disk-to-hadoop-data-nodesrepeat.html
==================
You need to apply comma-separated lists only to
dfs.data.dir (HDFS) and mapred.local.dir (MR) directly.
 
Make sure the subdirectories are different for each, else you may accidentally wipe away your data when you restart MR services.

The hadoop.tmp.dir property does not accept multiple paths and you should avoid using it in production - its more of a utility property that acts as a default base path for other properties.
stop-dfs.sh
start-dfs.sh

hadoop挂载多硬盘，ZZ


联系
我们