首页 > 代码库 > 数据采集+调度:cdh5.8.0+mysql5.7.17+hadoop+sqoop+hbase+oozie+hue
数据采集+调度:cdh5.8.0+mysql5.7.17+hadoop+sqoop+hbase+oozie+hue
一、前期准备
1.操作系统版本
# cat /etc/redhat-release
CentOS release 6.5 (Final)
# uname –a
Linux enc-bigdata05 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
2.前期软件准备
2-1. jdk版本: jdk-7u80-linux-x64.tar.gz
[jdk1.7.80下载地址][1]
2-2. mysql版本: mysql-5.7.17-linux-glibc2.5-x86_64.tar.gz
[mysql-5.7.17-x86_64下载地址][2]
[mysql所有版本的下载地址][3]
2-3. cloudera manager版本: cloudera-manager-el6-cm5.8.0_x86_64.tar.gz
[cloudera-manager-el6-cm5.8.0_x86_64.tar.gz下载地址][4]
[cloudera-manager5所有版本的下载地址][5]
2-4. parecel包版本:
CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel
CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha1
manifest.json
[CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel下载地址][6]
注意: 因为linux操作版本是centos6.5 x64,所以选择的cloudera manager和parecel包的版本都是对应el6版本,
如果您的linux系统版本是centos7、ubuntu等等其他版本,更下载对应的cloudera manager和parecel版本进行安装;
[cloudera官网关于CM5.8.0的安装说明][7]
----------
二、cm5.8.0安装前环境的配置
1.修改集群主机名hostname
# vim /etc/sysconfig/network
:HOSTNAME=enc-bigdata05
重启服务器shutdown -r now,是配置文件network生效
剩下的主机依次修改hostname, enc-bigdata0[1-6]
----------
2. 修改系统ip和hostname的映射: /etc/hosts
2-1.修改映射关系
# vim /etc/hosts
2-2.将文件hosts复制发送到集群的每一个节点
# for a in {1..6} ; do scp /etc/hosts enc-bigdata0$a:/etc/hosts ; done
注意: 将localhost的本地映射关系删除,或者放在文件最后面
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
2-3. 检查是否集群每一个节点的hosts都已经修改过来了
# for a in {1..6} ; do ssh enc-bigdata0$a cat /etc/hosts ; done
----------
3.设置ssh免密码登入,打通集群的每一个节点
3-1.创建公钥和私钥
# ssh-keygen -t rsa -P ‘‘
注意: 在集群的每一台节点上运行上述命令,创建当前用户的公钥和私钥;
每一个用户都可以创建自己的ssh公钥和私钥,打通每一个节点,可以用其它用户(非root用户)来安装CDH5.8.0;
3-2.将每一个节点的公钥id_rsa.pub放入到认证文件中authorized_keys;
# for a in {1..6}; do ssh enc-bigdata0$a cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys; done
3-3.将认证文件authorized_keys复制发送到每一个节点上去: /root/.ssh/authorized_keys
# for a in {1..6}; do scp /root/.ssh/authorized_keys enc-bigdata0$a:/root/.ssh/authorized_keys ; done
----------
4.安装jdk1.7.0_80
4-1.解压jdk到指定路径下
# tar -zxvf jdk-7u80-linux-x64.tar.gz -C /usr/local/
4-2.将解压后的jdk1.7.0_80目录发送到每一台节点的指定目录下
# for a in {1..6}; scp -r /usr/local/jdk1.7.0_80/ enc-bigdata0$a:/usr/local/; done
4-3.修改配置文件 /etc/profile
# vim /etc/profile
#install jdk-7u80-linux-x64
export JAVA_HOME=/usr/local/jdk1.7.0_80
export JAVA_BIN=$JAVA_HOME/bin
export JAVA_LIB=$JAVA_HOME/lib
export CLASSPATH=.:$JAVA_LIB/tools.jar:$JAVA_LIB/dt.jar
export PATH=$JAVA_BIN:$PATH
4-4.将配置文件复制发送到每一个节点下
# for a in {1..6}; scp -r /etc/profile enc-bigdata0$a:/etc/profile; done
# source /etc/profile
# for a in {1..6}; do ssh enc-bigdata0$a /usr/local/jdk1.7.0_80/bin/java -version ; done
5.修改系统参数
5-1.关闭防火墙
# service iptables status/stop
# for a in {1..6}; do ssh enc-bigdata0$a /etc/init.d/iptables status; done
# for a in {1..6}; do ssh enc-bigdata0$a /etc/init.d/iptables stop; done
# for a in {1..6}; do ssh enc-bigdata0$a chkconfig iptables off; done
# for a in {1..6}; do ssh enc-bigdata0$a chkconfig iptables --list; done
5-2.关闭SELINUX
# setenforce 0 (临时生效)
# vim /etc/selinux/config (永久生效)
: SELINUX=disabled
# for a in {1..6}; scp /etc/selinux/config enc-bigdata0$a: /etc/selinux/config; done
5-3.修改swappiness
# echo "vm.swappiness=0" >> /etc/sysctl.conf (永久生效)
# echo 0 > /proc/sys/vm/swappiness (临时生效)
# for a in {1..6}; do ssh enc-bigdata0$a echo "vm.swappiness=0" >> /etc/sysctl.conf; done
5-4.编辑/etc/security/limits.conf
# vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535
# for a in {1..6}; scp /etc/security/limits.conf enc-bigdata0$a: /etc/security/limits.conf; done
5-5.修改/etc/rc.local (开机运行的命令)
#vim /etc/rc.local (永久生效)
: echo never > /sys/kernel/mm/transparent_hugepage/defrag
# for a in {1..6}; do ssh enc-bigdata0$a echo never > /sys/kernel/mm/transparent_hugepage/defrag ; done (临时生效)
三、安装mysql5.7.17
1. 检查之前是否安装过mysql,删除之前安装的mysql
# rpm -qa | grep mysql
mysql-libs-5.1.71-1.el6.x86_64
# rpm -e mysql-libs-5.1.71-1.el6.x86_64 --nodeps
# find / -name mysql
# rm -rf …
注意:卸载不了就强制卸载,--nodeps --force
2. 新增用户,指定家目录,指定登陆级别
# usedel mysql
# groupdel mysql
# mkdir /usr/local/mysql
# useradd -d /usr/local/mysql/ -s /sbin/nologin -m mysql
# chown -R mysql:mysql /usr/local/mysql
# chmod 755 /usr/local/mysql
3. 解压到制定路径下: /usr/local/mysql
# tar -zxvf mysql-5.7.17-linux-glibc2.5-x86_64.tar.gz -C /tmp/
# mv /tmp/mysql-5.7.17-linux-glibc2.5-x86_64/* /usr/local/mysql/
说明:将解压后的安装包里的文件拷贝到mysql中
# mkdir /usr/local/mysql/data
说明:创建数据存放路径
# chown -R mysql:mysql /usr/local/mysql/
4.安装和初始化
# /usr/local/mysql/bin/mysql_install_db --user=mysql --basedir=/usr/local/mysql/ --datadir=/usr/local/mysql/data/
# cp -a /usr/local/mysql/support-files/my-default.cnf /etc/my.cnf
# cp -a /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld
# /usr/local/mysql/bin/mysqld_safe --user=mysql &
5.启动mysqld
# /etc/init.d/mysqld status
# /etc/init.d/mysqld restart
# chkconfig mysqld on
# chkconfig mysqld --list
添加快捷方式:ln -s /usr/local/mysql/bin/mysql /usr/bin/
6.初始密码
mysql5.7会在root用户家目录下生成一个初始化密码。
# cat /root/.mysql_secret
Password set for user ‘root@localhost‘ at 2016-06-01 15:23:25 FhCmc,k50X/y
# mysql -uroot -p (输入初始化密码)
//修改密码
mysql> SET PASSWORD = PASSWORD(‘123qwe‘);
mysql> flush privileges;
7. 添加远程访问
第一种方法: 改表
mysql> use mysql;
mysql> update user set host = ‘%‘ where user = ‘root‘;
第二种方法: 授权
mysql>GRANT ALL PRIVILEGES ON *.* TO ‘root‘@‘%‘;
mysql>FLUSH PRIVILEGES
8. 问题: 忘记初始化密码
# service mysqld stop
# vim /etc/my.cnf 在末尾添加 skip-grant-tables
# service mysqld start
# mysql -uroot -p (可以免密码登入)
mysql> set password for ‘root‘@‘localhost‘=password(‘new password‘);
mysql>flush privileges;
9. 配置/etc/my.cnf
---------------------------------------------------------------------
[mysql]
# 设置mysql客户端默认字符集
default-character-set=utf8
[mysqld]
#设置3306端口
port = 3306
# 设置mysql的安装目录
basedir=/usr/local/mysql
# 设置mysql数据库的数据的存放目录
datadir=/usr/local/mysql/data
tmpdir=/usr/local/mysql/tmp
# 允许最大连接数
max_connections=2000
#服务端使用的字符集默认为8比特编码的latin1字符集
character-set-server=utf8
# 创建新表时将使用的默认存储引擎
default-storage-engine=INNODB
#设置不区分大小写(0:大小写敏感;1:大小写不敏感)
lower_case_table_names=1
#安装模式:update mysql.user set authentication_string=password(‘123qwe‘) where user=‘root‘;
#skip-grant-tables #忘记初始化密码
--------------------------------------------------------------------
10. 创建数据库
mysql>create database amon default charset utf8 collate utf8_general_ci; #activity monitor
mysql>create database hive default charset utf8 collate utf8_general_ci; #hive
mysql>create database oozie default charset utf8 collate utf8_general_ci; #oozie
mysql>create database hue default charset utf8 collate utf8_general_ci; #hue
11. 新建用户(也可以延用root用户)
mysql>create user ‘admin‘@‘%‘identified by ‘Admin@123‘;
mysql>grant all privileges on *.* to ‘admin‘@‘%‘;
mysql>flush privileges;
----------
四、安装cloudera manager
1.解压cloudera-manager-el6-cm5.8.0_x86_64.tar.gz到指定路径:/opt
# tar -zxvf cloudera-manager-el6-cm5.8.0_x86_64.tar.gz -C /opt/
# for a in {1..6}; do scp -r /opt/cm-5.8.0/ enc-bigdata0$a:/opt/;done
2.修改配置文件
# vim /opt/cm-5.8.0/etc/cloudera-scm-agent/config.ini
server_host=enc-bigdata05
# for a in {1..6}; do scp -r /opt/cm-5.8.0/etc/cloudera-scm-agent/config.ini enc-bigdata0$a:/opt/cm-5.8.0/etc/cloudera-scm-agent/config.ini ; done
3.创建CDH内置数据库scm
# mv mysql-connector-java-5.1.40-bin.jar /opt/cm-5.7.0/share/cmf/lib
# /opt/cm-5.7.1/share/cmf/schemascm_prepare_database.sh mysql -hlocalhost -uroot –pAdmin@123 --scm-host localhost scm scm scm
scm_prepare_database.sh 数据库类型 服务器ip username password -scm-host [cloudera manager server所在服务器] scm scm scm;
4.将parecel包复制到cdh-server的指定目录下,修改CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha1文件名
# cp CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel /opt/cloudera/parcel-repo/
# cp CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha1 /opt/cloudera/parcel-repo/
# cp manifest.json/opt/cloudera/parcel-repo/
# mv /opt/cloudera/parcel-repo/CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha
4.启动cloudera-scm-server
# /opt/cm-5.8.0/etc/init.d/cloudera-scm-server start/status/stop
5.启动cloudera-scm-agent
# for a in {1..6}; do ssh enc-bigdata0$a /opt/cm-5.8.0/etc/init.d/cloudera-scm-agent start; done
6.问题:cloudera-scm-agent 启动失败: Unable to create the pidfile
原因: 无法创建 /opt/cm-5.8.0/run/cloudera-scm-agent
解决办法:
# mkdir /opt/cm-5.8.0/run/cloudera-scm-agent
# chown -R cloudera-scm:cloudera-scm /opt/cm-5.8.0/run/cloudera-scm-agent
7.进入网址: http://IP:7180/ (配置CDH5.8.0)
enc-bigdata0[1-6].enc.cn ##点击模式
注意: 修改jdk主目录很重要,以防止集群主机无法检测到jdk;
注意: 要保证所有检查都成功,以防止后面报错
注意 : hive的元数据库如果是安装在mysql中的话,需要将相应数据库驱动拷贝到hive家目录下的lib;
# mv mysql-connector-java-5.1.40-bin.jar /opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hive/lib
将相应的数据库驱动拷贝到对应的服务家目录下,以防止数据库连接失败;
五、问题
1. 关于namenode的内存问题,namenode的内存如果不够的话,会导致namenode服务进程死掉
问题描述:can‘t open /opt/cm-5.7.0/run/cloudera-scm-agent/process/1033-hdfs-NAMENODE/supervisord.conf , Permission deny
解决办法:增加内存
2. 无法检测到jdk存在
解决办法:设置JDK主目录
3. oozie web 页面不能显示
解决办法:上传ext-2.2.zip到指定路径:/var/lib/oozie
六、总结
**本篇章仅供参考,具体的请以官方文档为准,谢谢!!**
数据采集+调度:cdh5.8.0+mysql5.7.17+hadoop+sqoop+hbase+oozie+hue