首页 > 代码库 > Hadoop 1.2.1 安装笔记01 : Linux 与免密码

Hadoop 1.2.1 安装笔记01 : Linux 与免密码

2024-07-17 02:28:39   226人阅读

目标： 配置一个hadoop 1.2.1 测试环境 微笑

用的JDK是: jdk-7u65-linux-x64.gz

选的hadoop 是： hadoop-1.2.1.tar.gz 

均来源Apache和oracle网站

主机规划：

image

Linux版本 ：  Centos 6.5 x64位 

/boot：用来存放与Linux系统启动有关的程序，比如启动引导装载程序等，建议大小为100MB。

/usr：用来存放Linux系统中的应用程序，其相关数据较多，建议大于3GB以上。

/var：用来存放Linux系统中经常变化的数据以及日志文件，建议大于1GB以上。

/home：存放普通用户的数据，是普通用户的宿主目录，建议大小为剩下的空间。

/：Linux系统的根目录，所有的目录都挂在这个目录下面，建议大小为5GB以上。

/tmp：将临时盘在独立的分区，可避免在文件系统被塞满时影响到系统的稳定性。建议大小为500MB以上。

swap：实现虚拟内存，建议大小是物理内存的1~2倍。

clipboard

Boot Loader密码是 D*****2014

安装选择 Basic Server 不然所有的包得一个个打，确实很麻烦.

   

  • 配置sudo ，赋予hadoop用户权限 ，并配置sudo免密

useradd hadoop

passwd hadoop

vi /etc/sudoers

root ALL=(ALL) ALL

hadoop ALL=(ALL) ALL 添加一行

hadoop ALL=(ALL) NOPASSWD: ALL 添加一行免密码

clipboard[1]

  •   配置本机的解析

[hadoop@master /]$ cat /etc/hosts    
10.15.5.200 master.hadoop    
10.15.5.201 slave01.hadoop    
10.15.5.202 slave02.hadoop

  • 配置主机名

vi etc/sysconfig/network

HOSTNAME=master.hadoop

  •    配置IP 地址

[hadoop@master /]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0    
DEVICE=eth0    
TYPE=Ethernet    
UUID=721f9261-45d5-4335-9b47-64459173b2a9    
ONBOOT=yes    
NM_CONTROLLED=yes    
BOOTPROTO=none    
HWADDR=00:50:56:82:00:0F    
IPADDR=10.15.5.200    
PREFIX=24    
GATEWAY=10.15.5.1    
DEFROUTE=yes    
IPV4_FAILURE_FATAL=yes    
IPV6INIT=no    
NAME="System eth0"

  • 挂载 安装盘，以便所需要的安装包。打上ftp包 方便安装软件

[root@master home]# mount -t auto /dev/cdrom /home/cdrom 
mount: block device /dev/sr0 is write-protected, mounting read-only

[root@master Packages]# rpm -ivh ftp-0.17-54.el6.x86_64.rpm

warning: ftp-0.17-54.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY    
Preparing... ########################################### [100%]    
1:ftp ########################################### [100%]

目标： master和slaves  hadoop账户无密码互访

原理：

可以粗糙的理解为，我制造一把钥匙，把样子发给你，你在门禁系统里记下我钥匙的样子，我拿这把钥匙去开门时候你的门禁系统直接放行。

  • Master上，登录hadoop用户，并在/home/hadoop目录下操作

[hadoop@master ~]$ ssh-keygen -t rsa    
Generating public/private rsa key pair.    
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):    
Created directory ‘/home/hadoop/.ssh‘.    
Enter passphrase (empty for no passphrase):    
Enter same passphrase again:    
Your identification has been saved in /home/hadoop/.ssh/id_rsa.    
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.    
The key fingerprint is:    
8d:81:09:c8:45:3f:c0:fb:3b:a0:cf:95:b6:dd:e9:b1 hadoop@master.hadoop    
The key‘s randomart image is:    
+--[ RSA 2048]----+    
| . == |    
| o .+ o |    
| .= . |    
| . . + |    
| . S . |    
| . .. |    
| . .+. . |    
| .. ooo . + |    
| .o ....E |    
+-----------------+

  • 该隐藏文件夹下，生成了一个 id_rsa.pub文件

[hadoop@master .ssh]$ ls -al /home/hadoop/.ssh    
total 16    
drwx------. 2 hadoop hadoop 4096 Jul 30 22:09 .    
drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 ..    
-rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa    
-rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub

[hadoop@master .ssh]$ cat id_rsa.pub    
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAugxDdYElWX8URmERVTPsNtSW2mZoEMZyzmIKEe3UQjMbgYOCx0jV15/vtnOXJf4K+S6hccajih2oemcC4BmMI99nFWcyd9zcrvfJvn/DZHnw0Yog6MyMd9Qw2BQWUL265dhW2FNCaECuyG2U1CxR0W9wZlVz54JlToCRX6yUvWZzzGqUw/oR3ZWE7pUpQiOHV0ZNypUtbWEw/zj7N01LbVskNLqyY164APIVfZBONPMXjs/H6B8/VcxsA0lDwAqndnMxJ1iQHFkMntNgqcLkhs8oAmvl+/A6NH2I0ZBW+VOCUIjbNKNRO9BKwGvUQUZGcKthEPU0jK5ErsS6RPBrmQ== hadoop@master.hadoop

  • 把这个文件的内容，追加到 authorized_keys 的文件里

[hadoop@master .ssh]$ cat id_rsa.pub >> authorized_keys    
[hadoop@master .ssh]$ ls    
authorized_keys id_rsa id_rsa.pub    
[hadoop@master .ssh]$ cat authorized_keys    
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAugxDdYElWX8URmERVTPsNtSW2mZoEMZyzmIKEe3UQjMbgYOCx0jV15/vtnOXJf4K+S6hccajih2oemcC4BmMI99nFWcyd9zcrvfJvn/DZHnw0Yog6MyMd9Qw2BQWUL265dhW2FNCaECuyG2U1CxR0W9wZlVz54JlToCRX6yUvWZzzGqUw/oR3ZWE7pUpQiOHV0ZNypUtbWEw/zj7N01LbVskNLqyY164APIVfZBONPMXjs/H6B8/VcxsA0lDwAqndnMxJ1iQHFkMntNgqcLkhs8oAmvl+/A6NH2I0ZBW+VOCUIjbNKNRO9BKwGvUQUZGcKthEPU0jK5ErsS6RPBrmQ== hadoop@master.hadoop

  • 此时需要修改authorized_keys的文件权限 ，不安全的设置安全设置，会让你不能使用RSA功能

[hadoop@master .ssh]$ ls -al  
total 20  
drwx------. 2 hadoop hadoop 4096 Jul 30 22:20 .  
drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 ..  
-rw-rw-r--. 1 hadoop hadoop 402 Jul 30 22:20 authorized_keys  
-rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa  
-rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub  
[hadoop@master .ssh]$ sudo chmod 600 authorized_keys  
[hadoop@master .ssh]$ ls -al  
total 20  
drwx------. 2 hadoop hadoop 4096 Jul 30 22:20 .  
drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 ..  
-rw-------. 1 hadoop hadoop 402 Jul 30 22:20 authorized_keys  
-rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa  
-rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub

  • 修改 /etc/ssh/sshd_config    并且确认其中三个选项 然后sudo service  sshd restart 服务

[hadoop@master .ssh]$ sudo cat /etc/ssh/sshd_config   
RSAAuthentication yes 
PubkeyAuthentication yes    
AuthorizedKeysFile .ssh/authorized_keys

  • 在所有Slave上建立 .ssh文件夹,改变权限

[hadoop@slave02 ~]$ mkdir .ssh    
[hadoop@slave02 ~]$ ls -al 
total 32    
drwx------. 3 hadoop hadoop 4096 Jul 30 22:47 .    
drwxr-xr-x. 4 root root 4096 Jul 30 20:10 ..    
-rw-------. 1 hadoop hadoop 401 Jul 30 20:58 .bash_history    
-rw-r--r--. 1 hadoop hadoop 18 Jul 18 2013 .bash_logout    
-rw-r--r--. 1 hadoop hadoop 176 Jul 18 2013 .bash_profile    
-rw-r--r--. 1 hadoop hadoop 124 Jul 18 2013 .bashrc    
drwxrwxr-x. 2 hadoop hadoop 4096 Jul 30 22:47 .ssh    
-rw-------. 1 hadoop hadoop 557 Jul 30 20:28 .viminfo

[hadoop@slave02 ~]$ sudo chmod 700 .ssh    
[hadoop@slave02 ~]$ ls -al    
total 32    
drwx------. 3 hadoop hadoop 4096 Jul 30 22:47 .    
drwxr-xr-x. 4 root root 4096 Jul 30 20:10 ..    
-rw-------. 1 hadoop hadoop 401 Jul 30 20:58 .bash_history    
-rw-r--r--. 1 hadoop hadoop 18 Jul 18 2013 .bash_logout    
-rw-r--r--. 1 hadoop hadoop 176 Jul 18 2013 .bash_profile    
-rw-r--r--. 1 hadoop hadoop 124 Jul 18 2013 .bashrc    
drwx------. 2 hadoop hadoop 4096 Jul 30 22:47 .ssh    
-rw-------. 1 hadoop hadoop 557 Jul 30 20:28 .viminfo    
[hadoop@slave02 ~]$

  • 把id_rsa.pub发送给slaves

[hadoop@master .ssh]$ scp id_rsa.pub hadoop@slave01.hadoop:/home/hadoop/.ssh/id_rsa_frommaster.pub  
The authenticity of host ‘slave01.hadoop (10.15.5.201)‘ can‘t be established.  
RSA key fingerprint is 76:14:2f:f9:d9:03:07:17:7c:d1:ad:1e:af:55:45:00.  
Are you sure you want to continue connecting (yes/no)? yes  
Warning: Permanently added ‘slave01.hadoop,10.15.5.201‘ (RSA) to the list of known hosts.  
hadoop@slave01.hadoop‘s password:  
id_rsa.pub 100% 402 0.4KB/s 00:00  
[hadoop@master .ssh]$ scp id_rsa.pub hadoop@slave02.hadoop:/home/hadoop/.ssh/id_rsa_frommaster.pub  
The authenticity of host ‘slave02.hadoop (10.15.5.202)‘ can‘t be established.  
RSA key fingerprint is 76:14:2f:f9:d9:03:07:17:7c:d1:ad:1e:af:55:45:00.  
Are you sure you want to continue connecting (yes/no)? yes  
Warning: Permanently added ‘slave02.hadoop,10.15.5.202‘ (RSA) to the list of known hosts.  
hadoop@slave02.hadoop‘s password:  
id_rsa.pub 100% 402 0.4KB/s 00:00

  • 把这个值追加到slaves上的authorized_keys里面,并修改安全权限

[hadoop@slave02 .ssh]$ cat id_rsa_frommaster.pub >> authorized_keys    
[hadoop@slave02 .ssh]$ sudo chmod 600 authorized_keys

  • 修改slaves主机上的 /etc/ssh/sshd_config内容，并重启sshd服务

[hadoop@slave01 .ssh]$ sudo cat /etc/ssh/sshd_config   
RSAAuthentication yes 
PubkeyAuthentication yes 
AuthorizedKeysFile .ssh/authorized_keys

[hadoop@slave01 .ssh]$ sudo service sshd restart    
Stopping sshd: [ OK ]    
Starting sshd: [ OK ]

   

   

最终验证方法

[hadoop@master .ssh]$ ssh slave01.hadoop

Last login: Wed Jul 30 23:00:25 2014 from master.hadoop

[hadoop@slave01 ~]$ logout

Connection to slave01.hadoop closed.

[hadoop@master .ssh]$ ssh slave02.hadoop

[hadoop@slave02 ~]$ logout

Connection to slave02.hadoop closed.

同理： 从slaves 到master

  1. slaves 生成公钥 

  2. 把slaves把Slave的公钥追加到Master的".ssh"文件夹下的"authorized_keys"中

[hadoop@slave01 .ssh]$ cat id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqkRUrgiyD3W36WPRpL9ifJjgnVe+1R4x4mRhimKXOfmW+D3dcbi9FwE2j0/H+NaMdLWjQYw685ITdfHni0X5La7YXy6EIe0FQB/NxDKVslc44RuzJQJKQkosqSY/HGVfefF7OzIFCEcqVQDqN+OpWlrbXNtmO1UUhG2TFvj3MsbgwWHf7fc+uSN7Y7bMzLJpKhAVEjljjyTaHekJ8WMNzgT160vgr0mIZawDlRxKRlS2HtWQnndf74ZJDCeQKgJA6RUKuqBLop5x/0lvHBbUkn3MADnJUlIzd5pJW/aFEVySypfkGZTspj3+m8gNfQkqYJiB6sBfFgUYt1iPL/GNRw== hadoop@slave01.hadoop

[hadoop@slave02 ~]$ cat .ssh/id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3mHGmqOGjIkC/Elc/4CopgvKCQ7AdtqV87dhQPWnDdlljePVY/GbCWYqp4h8CmtmsMNreTva8RLBPSIP3jGMrbvNDCCuMo+mZGa034yR6JwekI9zVikzYscCTqWCk6w5Hs3u/pnb1Ym6a46ho+DNEm42qIaYzrhRn9Fc1F9Hd3/dXRq0KZlO/5xMKuhFt1GKy+gs+l7MiK6y7pTISx+ox/mDjDQfzxfpguRO68XX54+dD0gONSb/MaVVMdqiWK3Fh88ouN23sKi/cnzD1vLSm55kpT6ZRo792qcFmMe7CIYc6dwdkKZBgkdheayWy5PPfsZrGfDd/erVCWrmofDhiw== hadoop@slave02.hadoop

可以用cat >>的方式追加，或者vi直接写入。钥匙么，让master知道就行了.

clipboard[6]


联系
我们