Spark安装过程

首页 > 代码库 > Spark安装过程

2024-08-03 02:44:04 214人阅读

1. expect安装

Expect是基于Tcl语言的一种脚本语言，其实无论是交互还是非交互的应用场合，Expect都可以大显身手，但是对于交互式的特定场合，还非Except莫属。

第1步：使用root用户登录

第2步：下载安装文件expect-5.43.0.tar.gz 和 tcl8.4.11-src.tar.gz

第3步：解压安装包
       解压tcl8.4.11-src.tar.gz
       tar –xvf tcl8.4.11-src.tar.gz
       解压后将创建tcl8.4.11 文件夹

       解压expect-5.43.0.tar.gz
       tar –xvf expect-5.43.0.tar.gz
       解压后将创建expect-5.43 文件夹

第4步：安装tcl
       进入tcl8.4.11/unix 目录
        a.执行sed -i "s/relid‘/relid/" configure
        b.执行./configure --prefix=/expect
        c.执行make
        d.执行make install
        e.执行mkdir -p /tools/lib
        f.执行cp tclConfig.sh /tools/lib/
        g. 将/tools/bin目录export到环境变量
           tclpath=/tools/bin
           export tclpath

第5步：安装Expect
        进入/soft/expect-5.43目录
        执行./configure --prefix=/tools --with-tcl=/tools/lib --with-x=no
        如果最后一行提示：
        configure: error: Can‘t find Tcl private headers
        需要添加一个头文件目录参数
        --with-tclinclude=../tcl8.4.11/generic,即
        ./configure --prefix=/tools --with-tcl=/tools/lib --with-x=no --with-tclinclude=../tcl8.4.11/generic
        ../tcl8.4.11/generic 就是tcl解压安装后的路径，一定确保该路径存在
        执行make
        执行make install
        编译完成后会生在/tools/bin内生成expect命令
        执行/tools/bin/expect出现expect1.1>提示符说明expect安装成功.

第6步：创建一个符号链接
        ln -s /tools/bin/expect /usr/bin/expect
        查看符号连接
        ls -l /usr/bin/expect
        lrwxrwxrwx 1 root root 17 06-09 11:38 /usr/bin/expect -> /tools/bin/expect

        这个符号链接将在编写expect脚本文件时用到，例如在expect文件头部会指定用于执行该脚本的shell
        #!/usr/bin/expect

2. SSH免输入密码登陆

主机Master操作如下：

[Spark@Master ~]$ ssh-keygen -t rsa -P ‘‘ -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /home/Spark/.ssh/id_rsa.
Your public key has been saved in /home/Spark/.ssh/id_rsa.pub.
The key fingerprint is:
c9:d0:1f:92:43:42:85:f1:c5:23:76:f8:df:80:e5:66 Spark@Master
The key‘s randomart image is:
+--[ RSA 2048]----+
| .++oo. |
| .=+o+ . |
| ..*+.= |
| o =o.E |
| S .+ o |
| . . |
| |
| |
| |
+-----------------+
[Spark@Master ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

3. 然后执行如下自动化传输公钥脚本SSH.sh，将主机Master上的公钥传输给各个从节点Slave1，Slave2......

（Note：将SSH.sh和NoPwdAccessSSH.exp脚本文件添加执行权限，如下：）

[Spark@Master test]$ chmod +x SSH.sh

[Spark@Master test]$ chmod +x NoPwdAccessSSH.exp

//执行自动化无密码访问脚本SSH.sh

[Spark@Master test]$ ./SSH.sh
spawn ssh-copy-id -i /home/Spark/.ssh/id_rsa.pub Spark@Master
The authenticity of host ‘master (192.168.8.29)‘ can‘t be established.
RSA key fingerprint is f0:3f:04:51:36:b5:91:c7:fa:47:5a:49:bc:fd:fe:40.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘master,192.168.8.29‘ (RSA) to the list of known hosts.
Now try logging into the machine, with "ssh ‘Spark@Master‘", and check in:

.ssh/authorized_keys

to make sure we haven‘t added extra keys that you weren‘t expecting.

No Password Access Master is Succeed!!!
spawn ssh-copy-id -i /home/Spark/.ssh/id_rsa.pub Spark@Slave1
Spark@slave1‘s password:
Now try logging into the machine, with "ssh ‘Spark@Slave1‘", and check in:

.ssh/authorized_keys

to make sure we haven‘t added extra keys that you weren‘t expecting.

No Password Access Slave1 is Succeed!!!
spawn ssh-copy-id -i /home/Spark/.ssh/id_rsa.pub Spark@Slave2
Spark@slave2‘s password:
Now try logging into the machine, with "ssh ‘Spark@Slave2‘", and check in:

.ssh/authorized_keys

to make sure we haven‘t added extra keys that you weren‘t expecting.

No Password Access Slave2 is Succeed!!!
[Spark@Master test]$ ssh Slave1
Last login: Wed Nov 19 02:35:28 2014 from 192.168.8.29
Welcome to your pre-built HUSOR STANDARD WEB DEVELOP VM.

PHP5.3 (/usr/local/php-cgi) service:php-fpm
PHP5.4 (/usr/local/php-54) service:php54-fpm
Tengine1.4.6, mysql-5.5.29, memcached 1.4.15, tokyocabinet-1.4.48, tokyotyrant-1.1.41, httpsqs-1.7, coreseek-4.1

WEBROOT: /data/webroot/www/

[Spark@Slave1 ~]$ exit
logout
Connection to Slave1 closed.
[Spark@Master test]$ ssh Slave2
Last login: Wed Nov 19 01:48:01 2014 from 192.168.8.1
Welcome to your pre-built HUSOR STANDARD WEB DEVELOP VM.

WEBROOT: /data/webroot/www/

[Spark@Slave2 ~]$

以上自动化执行脚本文件如下：

SSH.sh

#!/bin/bashbin=`which $0`bin=`dirname ${bin}`bin=`cd "$bin"; pwd`if [ ! -x "$bin/NoPwdAccessSSH.exp" ]; then  echo "Sorry, $bin/NoPwdAccessSSH.exp is not executable file,please chmod +x $bin/NoPwdAccessSSH.exp."  exit 1fifor hostInfo in $(cat $bin/SparkCluster);do        host_name=$(echo "$hostInfo"|cut -f1 -d":")    user_name=$(echo "$hostInfo"|cut -f2 -d":")    user_pwd=$(echo "$hostInfo"|cut -f3 -d":")        local_host=`ifconfig eth0 | grep "Mask" | cut -d: -f2 | awk ‘{print $1}‘`    if [ $host_name = $local_host ]; then        continue;    else         expect $bin/NoPwdAccessSSH.exp $host_name $user_name $user_pwd //调用expect应答式脚本NoPwdAccessSSH.exp    fi    if [ $? -eq 0 ]    then        echo "No Password Access $host_name is Succeed!!!"    else        echo "No Password Access $host_name is failed!!!"    fidone

NoPwdAccessSSH.exp

#!/usr/bin/expect -f# auto ssh loginif { $argc<3} {  puts stderr "Usage: $argv0(hostname) $argv1(username) $argv2(userpwd).\n "　exit 1}set hostname [lindex $argv 0]set username [lindex $argv 1]set userpwd [lindex $argv 2]spawn ssh-copy-id -i /home/Spark/.ssh/id_rsa.pub $username@$hostnameexpect {   "*yes/no*" { send "yes\r";exp_continue }   "*password*" { send "$userpwd\r";exp_continue }   "*password*" { send "$userpwd\r"; }}

其中的SparkCluster文件内容如下：

Master:Spark:111111
Slave1:Spark:111111
Slave2:Spark:111111

Spark安装过程

声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉：投诉/举报工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。

联系
我们

首页 > 代码库 > Spark安装过程

Spark安装过程

看完仍有疑问？有类似问题直接问程序猿