RedHat(Centos)6.4全自动安装Ambari2.6.0详细教程(图文一步步)

Install Hortonworks Hadoop 2.6 Using Ambari

变更历史

操作 创建

责任人 日期 版本 V1.0 创建 变更内容 szdxxhb_hairy 2017/4/12 目录

1. 2.

Hadoop集群安装规划 .................................................................................................................................................................................................. 3 下载软件包 .................................................................................................................................................................................................................... 3 2.1. 下载JDK ............................................................................................................................................................................................................. 3 2.2. 下载Ambari ....................................................................................................................................................................................................... 3 2.3. 下载Hadoop ...................................................................................................................................................................................................... 3 安装操作系统 ................................................................................................................................................................................................................ 3 2.4. 最小化安装CentOS 6.6 ..................................................................................................................................................................................... 3 2.5. 配置本地YUM源 .............................................................................................................................................................................................. 4 2.6. 安装软件包 ........................................................................................................................................................................................................ 6 2.7. 修改系统配置参数 ............................................................................................................................................................................................ 7 克隆节点及设置SSH互信 ............................................................................................................................................................................................ 7 4.1. 虚机管理软件克隆 .......................................................................................................................................................... 错误！未定义书签。 4.2. 修改主机名及网卡绑定 .................................................................................................................................................. 错误！未定义书签。 4.3. 设置SSH互信.................................................................................................................................................................................................... 7 安装Ambari ................................................................................................................................................................................................................... 8 5.1. 安装和设置HTTP Server ................................................................................................................................................. 错误！未定义书签。 5.2. 准备Ambari Local Repository ........................................................................................................................................................................... 8 5.3. 修改和分发YUM Repository文件 .................................................................................................................................................................... 9 5.4. 安装Ambari Agent和Server ............................................................................................................................................................................ 9 创建和设置Hadoop集群 ........................................................................................................................................................................................... 12 6.1. 创建Hive Metastore Repository ...................................................................................................................................................................... 12 6.2. 创建Oozie Server Repository .......................................................................................................................................................................... 12 6.3. 重启Ambari Server .......................................................................................................................................................... 错误！未定义书签。 6.4. 创建集群 .......................................................................................................................................................................................................... 12 6.5. 选择安装服务 .................................................................................................................................................................................................. 17 安装问题FAQ .............................................................................................................................................................................................................. 32 7.1. 安装挂载CentOS 6.6 DVD ............................................................................................................................................................................... 32 7.2. Hive 启动失败................................................................................................................................................................. 错误！未定义书签。 7.3. Hbase启动失败 ............................................................................................................................................................... 错误！未定义书签。

3.

4.

5.

6.

7.

1. Hadoop集群安装规划

主机名 hadoop-db hadoop01 hadoop02 hadoop03 hadoop04 hadoop05 hadoop06 hadoop07 hadoop08

172.29.0.25 172.29.0.226 172.29.0.227 172.29.0.228 172.29.0.229 172.29.0.21 172.29.0.22 172.29.0.23 172.29.0.24 IP地址 DataNode DataNode DataNode DataNode DataNode DataNode DataNode DataNode 说明 Namenode/Ambari Server/Oracle 2. 下载离线软件包 2.1. 下载JDK

JDK:jdk-7u67-linux-x64.tar.gz

Oracle JDBC:ojdbc6-11.2.0.3.jar

2.2. 下载Ambari离线安装包（RedHat 6）

下载repo文件 wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3/ambari.repo -O /etc/yum.repos.d/ambari.repo 下载Ambari Server文件 http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos6.tar.gz 2.3. 下载Hadoop

下载Hadoop组件 http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.0.3/HDP-2.6.0.3-centos6-rpm.tar.gz 下载HDP-UTILS组件 http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6/HDP-UTILS-1.1.0.21-centos6.tar.gz 2.4. 下载CentOS6.4

CentOS-6.4-x86_64-bin-DVD1.iso CentOS-6.4-x86_64-bin-DVD2.iso 2.5. 下载依赖RPM包（依系统而定）

http://rpm.pbone.net/ 1、python-argparse-1.2.1-36.1.x86_64.rpm 3. 安装操作系统 3.1. 安装Redhat6.4

IT运维部提供机器，机器发布模板要求：

（1、关闭防火墙，且不自启动 2、selinux 为disabled）

3.2. 安装和设置HTTP Server

本文选择HADOOP-DB为Ambari Server，故需在HADOOP-DB上安装httpd [root@hadoop-db ~]# chkconfig --levels 345 httpd on [root@hadoop-db ~]# service httpd start 建立软连接 [root@hadoop-db html]# pwd /var/www/html [root@hadoop-db html]# cd /var/www/html/ [root@hadoop-db html]# ll 总用量 0 [root@hadoop-db html]# ln -s /mnt/redhat redhat [root@hadoop-db html]# ll 总用量 0 lrwxrwxrwx 1 root root 11 4月 12 14:43 redhat -> /mnt/redhat 3.3. 配置在线YUM源

加载Redhat和Centos6.4安装介质，并配置yum #mkdir –p /mnt/redhat #cd /etc/yum.repos.d #mkdir bak #mv *.repo bak #vi rhel-source.repo 建立本地私有库 [root@hadoop-db html]# yum -y install createrepo* Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. base | 4.0 kB 00:00 Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package createrepo.noarch 0:0.9.9-17.el6 will be installed --> Processing Dependency: python-deltarpm for package: createrepo-0.9.9-17.el6.noarch --> Running transaction check ---> Package python-deltarpm.x86_64 0:3.5-0.5.20090913git.el6 will be installed --> Processing Dependency: deltarpm = 3.5-0.5.20090913git.el6 for package: python-deltarpm-3.5-0.5.20090913git.el6.x86_64 --> Running transaction check ---> Package deltarpm.x86_64 0:3.5-0.5.20090913git.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ==================================================================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================================================================== Installing: createrepo noarch 0.9.9-17.el6 base 94 k Installing for dependencies: deltarpm x86_64 3.5-0.5.20090913git.el6 base 71 k python-deltarpm x86_64 3.5-0.5.20090913git.el6 base 27 k Transaction Summary ==================================================================================================================================================================================================== Install 3 Package(s) Total download size: 192 k Installed size: 518 k Downloading Packages: (1/3): python-deltarpm-3.5-0.5.20090913git.el6.x86_64.rpm | 27 kB 00:00 (2/3): createrepo-0.9.9-17.el6.noarch.rpm | 94 kB 00:00 (3/3): deltarpm-3.5-0.5.20090913git.el6.x86_64.rpm | 71 kB 00:00 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 7.8 MB/s | 192 kB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : deltarpm-3.5-0.5.20090913git.el6.x86_64 1/3 Installing : python-deltarpm-3.5-0.5.20090913git.el6.x86_64 2/3 Installing : createrepo-0.9.9-17.el6.noarch 3/3 Verifying : python-deltarpm-3.5-0.5.20090913git.el6.x86_64 1/3 Verifying : createrepo-0.9.9-17.el6.noarch 2/3 Verifying : deltarpm-3.5-0.5.20090913git.el6.x86_64 3/3 Installed: createrepo.noarch 0:0.9.9-17.el6 Dependency Installed: deltarpm.x86_64 0:3.5-0.5.20090913git.el6 python-deltarpm.x86_64 0:3.5-0.5.20090913git.el6 Complete! [root@hadoop-db html]# createrepo -v /mnt/others/ Spawning worker 0 with 1 pkgs Worker 0: reading python-argparse-1.2.1-36.1.x86_64.rpm Workers Finished Gathering worker results Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Starting other db creation: Thu Apr 13 14:29:32 2017 Ending other db creation: Thu Apr 13 14:29:32 2017 Starting filelists db creation: Thu Apr 13 14:29:32 2017 Ending filelists db creation: Thu Apr 13 14:29:32 2017 Starting primary db creation: Thu Apr 13 14:29:32 2017 Ending primary db creation: Thu Apr 13 14:29:33 2017 Sqlite DBs complete [root@hadoop-db html]# cd /mnt/others/ [root@hadoop-db others]# ll 总用量 264 -rw-r--r-- 1 root root 263604 4月 13 14:22 python-argparse-1.2.1-36.1.x86_64.rpm drwxr-xr-x 2 root root 4096 4月 13 14:29 repodata [root@hadoop-db others]# cd repodata/ 建立文件和html的目录链接 [root@hadoop-db html]# ll

总用量 0 lrwxrwxrwx 1 root root 27 4月 12 14:58 ambari -> /home/hadoop/package/ambari lrwxrwxrwx 1 root root 13 4月 13 14:07 centos6 -> /mnt/centos6/ lrwxrwxrwx 1 root root 24 4月 12 14:58 HDP -> /home/hadoop/package/HDP lrwxrwxrwx 1 root root 12 4月 13 14:30 others -> /mnt/others/ lrwxrwxrwx 1 root root 11 4月 12 14:43 redhat -> /mnt/redhat

rhel-source.repo文件内容如下 [base] name=redhat server 6.4 baseurl=http://172.29.0.25/redhat enable=1 gpgcheck=0 [addons] name=centos6.4 baseurl=http://172.29.0.25/centos6 enable=1 gpgcheck=0 [extras] name=others baseurl=http://172.29.0.25/others enable=1 gpgcheck=0

yum repolist [root@hadoop-db yum.repos.d]# yum repolist Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. repo id repo name status rhel-source Red Hat Enterprise Linux 6Server - x86_64 - Source 3,648 repolist: 3,648

yum clean all yum makecache [root@hadoop-db yum.repos.d]# yum clean all Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Cleaning repos: InstallMedia rhel-source Cleaning up Everything [root@hadoop-db yum.repos.d]# yum makecache Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. rhel-source | 3.9 kB 00:00 ... rhel-source/group_gz | 204 kB 00:00 ... rhel-source/filelists_db | 3.7 MB 00:00 ... rhel-source/primary_db | 3.1 MB 00:00 ... rhel-source/other_db | 1.5 MB 00:00 ... Metadata Cache Created 3.4. 安装软件包

检查&安装scp, curl, unzip, tar, ntp and wget,http

[root@hadoop-db ~]# yum -y install openssh-clients [root@hadoop-db ~]# yum -y install wget [root@hadoop-db ~]# yum -y install unzip [root@hadoop-db ~]# yum -y install wget [root@hadoop-db ~]# yum -y install ntp [root@hadoop-db ~]# yum -y install http* [root@hadoop-db ~]# yum -y install expect 安装JDK，准备目录,如果存在，则不创建 mkdir /usr/java 上传jdk-7u67-linux-x64.tar.gz到/usr/java目录下，如果存在，则不创建 tar -xvzf jdk-7u67-linux-x64.tar.gz ln -s /usr/java/jdk1.7.0_67 /usr/java/default 配置所有用户Java变量，编辑/etc/profile文件，增加如下行， export JAVA_HOME=/usr/java/default export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$CLASSPATH

3.5. 修改系统配置参数

编辑/etc/hosts文件，增加3行 172.29.0.25hadoop-db.xxxxx.com.cnhadoop-db 172.29.0.226hadoop01-sit.xxxxx.com.cn hadoop01 172.29.0.227hadoop02-sit.xxxxx.com.cn hadoop02 172.29.0.228hadoop03-sit.xxxxx.com.cn hadoop03 172.29.0.229hadoop04-sit.xxxxx.com.cn hadoop04 172.29.0.21hadoop05-sit.xxxxx.com.cn hadoop05 172.29.0.22hadoop06-sit.xxxxx.com.cn hadoop06 172.29.0.23hadoop07-sit.xxxxx.com.cn hadoop07 172.29.0.24hadoop08-sit.xxxxx.com.cn hadoop08 关闭SELINUX，编辑/etc/selinux/config文件 SELINUX=disabled 配置Linux服务 chkconfig iptables off chkconfig ip6tables off chkconfig --levels 345 ntpd on

4. Slave节点及设置SSH互信 4.1. 设置SSH互信

以下操作在HADOOP-DB主机（第一台主机）上进行操作 [root@hadoop-db ~]# ssh-keygen -t rsa [root@hadoop-db ~]# cd .ssh [root@hadoop-db .ssh]# cat id_rsa.pub >> authorized_keys [root@hadoop-db .ssh]# cd .. [root@hadoop-db ~]# chmod 700 .ssh [root@hadoop-db ~]# chmod 600 .ssh/authorized_keys 执行如下Shell脚本， 1、auto_ssh.shhosts 完成后进行连通性测试 2、测试连通性： #!/bin/sh if [ $# -ne 1 ]; then echo \ echo \ hostsFile\ exit 1 fi HOSTS_FILE=$1 LOCALDATE=`date +%Y%m%d` for ip in `cat ${HOSTS_FILE}` do REMOTEDATE=`ssh ${ip} date +%Y%m%d` if [ ${LOCALDATE}==${REMOTEDATE} ];then echo \主机与远程IP${ip} 连通测试成功\ else fi done 结果： [root@hadoop-db script]# sh test_ssh.sh hosts 主机与远程IP172.29.0.25 连通测试成功 主机与远程IP172.29.0.226 连通测试成功 主机与远程IP172.29.0.227 连通测试成功 主机与远程IP172.29.0.228 连通测试成功 主机与远程IP172.29.0.229 连通测试成功 主机与远程IP172.29.0.21 连通测试成功 主机与远程IP172.29.0.22 连通测试成功 主机与远程IP172.29.0.23 连通测试成功 主机与远程IP172.29.0.24 连通测试成功 echo \主机与远程IP${ip} 连通测试失败\ 4.2. 各节点系统配置

1、 分发rhel-source.repo到各个节点 2、 安装NTP服务 3、 发布host解析文件 4、 设置Hostname（刚分到机器的时候就应该定义好，此处不再修改hostname） 5、 关闭主机防火墙和自启动选项 6、 关闭selinux，设置为disabled 7、 设置UMASK eachsetting.sh

5. 安装Ambari

5.1. 准备Ambari Local Repository

上传ambari-2.5.0.3-centos6.tar.gz，HDP-2.6.0.3-centos6-rpm.tar.gz, HDP-UTILS-1.1.0.21-centos6.tar.gz到 /home/hadoop/package目录，并解压 [root@hadoop-db html]# mkdir –p /home/hadoop/package [root@hadoop-dbpackage]# mkdir -p HDP/HDP-UTILS [root@hadoop-dbpackage]# mkdir -p ambari/centos6/2.x/updates/2.5.0.3 [root@hadoop-dbpackage]# mkdir -p HDP/HDP-2.6.0 [root@hadoop-dbpackage]# tar -xzf HDP-UTILS-1.1.0.21-centos6.tar.gz -C/home/hadoop/package/HDP/HDP-UTILS/ [root@hadoop-dbpackage]# tar -xzf HDP-2.6.0.3-centos6-rpm.tar.gz-C /home/hadoop/package/HDP/ HDP-2.6.0/ [root@hadoop-dbpackage]# tar -xzf ambari-2.5.0.3-centos6.tar.gz -C ambari/centos6/2.x/updates/2.5.0.3 [root@hadoop-db html]# ln -s /home/hadoop/package/ambari ambari [root@hadoop-db html]# ln -s /home/hadoop/package/HDP HDP [root@hadoop-db html]# ll 总用量 0 lrwxrwxrwx 1 root root 27 4月 12 14:58 ambari -> /home/hadoop/package/ambari lrwxrwxrwx 1 root root 24 4月 12 14:58 HDP -> /home/hadoop/package/HDP lrwxrwxrwx 1 root root 11 4月 12 14:43 redhat -> /mnt/redhat 打开浏览器测试http://172.29.0.25/redhat/, http://172.29.0.25/ambari/, http://172.29.0.25/HDP/

5.2. 修改和分发YUM Repository文件

将ambari.repo文件修改 #VERSION_NUMBER=2.5.0.3-7 [ambari-2.5.0.3] name=ambari Version - ambari-2.5.0.3 baseurl=http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3 gpgcheck=1 gpgkey=http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 修改为： #VERSION_NUMBER=2.5.0.3-7 [ambari-2.5.0.3] name=ambari Version - ambari-2.5.0.3 baseurl=http://172.29.0.25/ambari/centos6/2.x/updates/2.5.0.3 gpgcheck=1 gpgkey=http://172.29.0.25/ambari/centos6/2.x/updates/2.5.0.3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1

5.3. 创建Ambari的Oracle用户

db_ambari

5.4. 安装AmbariServer

在HADOOP-DB节点，安装Ambari Server [root@hadoop-db ~]# yum -y install ambari-server

配置Ambari Server

[root@hadoop-db ~]# ambari-server setup #1、设置ambariserver 用户 [root@hadoop-db ~]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? n #2、设置JAVA_HOME： Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /usr/java/jdk1.7.0_71 Validating JDK on Ambari Server...done. Completing setup... #3、设置元数据库，将Oracle JDBC 放入/usr/share/java Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 2 Hostname (localhost): hadoop-db Port (1521): 1521 Select Oracle identifier type: 1 - Service Name 2 - SID (1): 2 SID (ambari): xxxxx Username (ambari): db_ambari Enter Database Password (bigdata): Re-enter password: Configuring ambari database... WARNING: Before starting Ambari Server, you must copy the Oracle JDBC driver JAR file to /usr/share/java and set property \Press to continue. #4、手动处理/var/lib/ambari-server/resources/oracle-jdbc-driver.jar Press to continue. Copying JDBC drivers to server resources... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-Oracle-CREATE.sql' Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... .......ambari-admin-2.5.0.3.7.jar .... Adjusting ambari-server permissions and ownership...

WARNING: Command chown -R -L ambari /var/lib/ambari-server returned exit code /var/lib/ambari-server with message: chown: 无法取消\的引用: 没有那个文件或目录 Ambari Server 'setup' completed successfully. 解决办法： [root@hadoop-db resources]# cp ojdbc6-11.2.0.3.jar /var/lib/ambari-server/resources/ojdbc6.jar #5、再次重新确认设置（前面没有错误，可以忽略） [root@hadoop-db ~]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Ambari-server daemon is configured to run under user 'ambari'. Change this setting [y/n] (n)? n Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... Do you want to change Oracle JDK [y/n] (n)? n Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? n Configuring database... Configuring ambari database... Copying JDBC drivers to server resources... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-Oracle-CREATE.sql' Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... ............ Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully. 6、# 执行/var/lib/ambari-server/resources/Ambari-DDL-Oracle-CREATE.sql（启动之前一定要在元数据库db_ambari执行此脚本，否则报错） WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-Oracle-CREATE.sql' 启动Ambari Server，使用tail –f /var/log/ambari-server/ambari-server.log查看日志 [root@hadoop-db ~]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start............................................................ DB configs consistency check: no errors and warnings were found. ERROR: Exiting with exit code 1. REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.

客户端浏览器登录（admin/admin） http://172.29.0.25:8080/

6. 创建和设置Hadoop集群 6.1. 创建Hive Metastore Repository

CREATE USER db_hive IDENTIFIED BY *** DEFAULT TABLESPACE TBS_AMBARI TEMPORARY TABLESPACE TEMP PROFILE \ QUOTA UNLIMITED ON TBS_AMBARI; GRANT UNLIMITED TABLESPACE TO db_hive; GRANT \GRANT \ALTER USER db_hive DEFAULT ROLE \ 6.2. 创建Oozie Server Repository

CREATE USER db_oozie IDENTIFIED BY *** DEFAULT TABLESPACE TBS_AMBARI TEMPORARY TABLESPACE TEMP PROFILE \ QUOTA UNLIMITED ON TBS_AMBARI; GRANT UNLIMITED TABLESPACE TO db_oozie; GRANT \GRANT \ALTER USER db_oozie DEFAULT ROLE \ 6.3. 创建集群

浏览器登录Ambari管理界面

输入：xxxxx_hadoop_sit

点击“Use Local Repository”

删除其余操作系统的Repository

按照HTTP yum 地址填入进去

安装配置表

项目 Name your cluster Stacks Advanced Repository Options redhat6HDP-2.2 redhat6HDP-UTILS-1.1.0.20 http://172.29.0.25/HDP/HDP-2.6.0/HDP/centos6/ http://172.29.0.25/HDP/HDP-UTILS/ HDP001 HDP 2.6 值 说明 Skip Repository Base URL validation No (Advanced) Install Options Target Hosts Host Registration Information hadoop-db hadoop[01-08] HADOOP-DB /root/.ssh/id_rsa文件

解决办法：每台主机上执行此脚本 if [ $# -ne 1 ]; then echo \ echo \ hostsFile\ exit 1 fi HOSTS_FILE=$1 LOCALDATE=`date +%Y%m%d` for ip in `cat ${HOSTS_FILE}` do ssh $ip echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag ssh $ip echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled ssh $ip echo never > /sys/kernel/mm/transparent_hugepage/enabled ssh $ip echo never > /sys/kernel/mm/transparent_hugepage/defrag ssh $ip python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=users done

6.4. 选择安装服务

Installed Service Version Description Yes HDFS 2.6.0.2.2.0.0 Apache Hadoop Distributed File System Yes YARN + MapReduce2 2.6.0.2.2.0.0 Apache Hadoop NextGen MapReduce (YARN) Yes Tez 0.5.2.2.2.0.0 Tez is the next generation Hadoop Query Processing framework written on top of YARN. Yes Nagios 3.5.0 Nagios Monitoring and Alerting system Yes Ganglia 3.5.0 Ganglia Metrics Collection system (RRDTool will be installed too) Yes Hive 0.14.0.2.2.0.0 Data warehouse system for ad-hoc queries & analysis of large datasets and table & storage management service Yes HBase 0.98.4.2.2.0.0 Non-relational distributed database and centralized service for configuration management & synchronization Yes Pig 0.14.0.2.2.0.0 Scripting platform for analyzing large datasets Yes Sqoop 1.4.5.2.2.0.0 Tool for transferring bulk data between Apache Hadoop and structured data stores such as relational databases Yes Oozie 4.1.0.2.2.0.0 System for workflow coordination and execution of Apache Hadoop jobs. This also includes the installation of the optional Oozie Web Console which relies on and will install the ExtJS Library. Yes ZooKeeper 3.4.6.2.2.0.0 Centralized service which provides highly reliable distributed coordination Yes Falcon 0.6.0.2.2.0.0 Data management and processing platform Yes Storm 0.9.3.2.2.0.0 Apache Hadoop Stream processing framework Yes Flume 1.5.2.2.2.0.0 A distributed service for collecting, aggregating, and moving large amounts of streaming data into HDFS Yes Slider 0.60.0.2.2.0.0 A framework for deploying, managing and monitoring existing distributed applications on YARN. Yes Knox 0.5.0.2.2.0.0 Provides a single point of authentication and access for Apache Hadoop services in a cluster Yes Kafka 0.8.1.2.2.0.0 A high-throughput distributed messaging system 根据需要选择，可以根据硬件资源的情况选择服务，平衡服务在各个节点的分布

分配节点和客户端 修复错误

[root@hadoop-db jdk1.7.0_71]# ambari-server setup --jdbc-db=oracle --jdbc-driver=/usr/share/java/ojdbc6-11.2.0.3.jar Using python /usr/bin/python Setup ambari-server Copying /usr/share/java/ojdbc6-11.2.0.3.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for oracle with ojdbc6-11.2.0.3.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully.

Summary Admin Name : admin Cluster Name : xxxxx_hadoop_sit Total Hosts : 9 (9 new) Repositories: ? redhat6 (HDP-2.6): http://172.29.0.25/HDP/HDP-2.6.0/HDP/centos6/ redhat6 (HDP-UTILS-1.1.0.21): http://172.29.0.25/HDP/HDP-UTILS/ ? Services: ? HDFS o DataNode : 7 hosts o NameNode : hadoop-db o NFSGateway : 7 hosts o SNameNode : hadoop01 ?

? ?

?

? ? ?

? ?

? ? ?

? ? ?

?

?

YARN + MapReduce2

o App Timeline Server : hadoop01 o NodeManager : 7 hosts

o ResourceManager : hadoop01 Tez

o Clients : 9 hosts Hive

o Metastore : hadoop01 o HiveServer2 : hadoop-db

o WebHCat Server : hadoop-db

o Database : Existing Oracle Database HBase

o Master : hadoop01 o RegionServer : 3 hosts

o Phoenix Query Server : 3 hosts Pig

o Clients : 9 hosts Sqoop

o Clients : 9 hosts Oozie

o Server : hadoop01

o Database : Existing Oracle Database ZooKeeper

o Server : 3 hosts Storm

o DRPC Server : hadoop01 o Nimbus : hadoop01 o UI Server : hadoop01 o Supervisor : 4 hosts Flume

o Flume : 3 hosts Ambari Infra

o Infra Solr Instance : hadoop-db Ambari Metrics

o Metrics Collector : hadoop01 o Grafana : hadoop-db Kafka

o Broker : 4 hosts Knox

o Gateway : hadoop-db SmartSense

o Activity Analyzer : hadoop-db o Activity Explorer : hadoop-db o HST Server : hadoop-db Spark2

o Livy for Spark2 Server : 7 hosts o History Server : hadoop01 o Thrift Server : 1 host Slider

o Clients : 9 hosts

查看日志解决问题

6.5. 升级集群为HA集群

检查集群没有告警后，点击HDFS-service actions-namnode HA选项，开始升级。基本没有什么好选择的，根据步骤来就行了。注意即使使用root用户执行的命令也必须全部copy！！

在一路好运的陪伴下，你终于可以看见下图了。恭喜恭喜！！！！！！

检查集群没有告警后，点击Yarn-service actions- Enable ResourceManager HA选项，开始升级。基本没有什么好选择的，根据步骤来就行了。注意即使使用root用户执行的命令也必须全部copy！！

7. 给HDP集群安装Kerberos 7.1. 安装KDC

在集群中找台机器安装KDC，这里我们使用Hadoop01 [root@hadoop01 ~]# yum install krb5-server krb5-libs krb5-workstation Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Repository HDP-UTILS-1.1.0.21 is listed more than once in the configuration Repository InstallMedia is listed more than once in the configuration HDP-2.6 | 2.9 kB 00:00 HDP-UTILS-1.1.0.21 | 2.9 kB 00:00 InstallMedia | 2.9 kB 00:00 addons | 4.0 kB 00:00 ambari-2.5.0.3 | 2.9 kB 00:00 base | 3.9 kB 00:00 extras | 2.9 kB 00:00 Setting up Install Process Package krb5-libs-1.10.3-10.el6.x86_64 already installed and latest version Package krb5-workstation-1.10.3-10.el6.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package krb5-server.x86_64 0:1.10.3-10.el6 will be installed HDP-2.6/filelists_db | 455 kB 00:00 HDP-UTILS-1.1.0.21/filelists_db | 95 kB 00:00 InstallMedia/filelists_db | 455 kB 00:00 addons/filelists_db | 5.9 MB 00:00 ambari-2.5.0.3/filelists_db | 147 kB 00:00 base/filelists_db | 3.7 MB 00:00 extras/filelists_db | 2.2 kB 00:00 --> Finished Dependency Resolution Dependencies Resolved ==================================================================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================================================================== Installing: krb5-server x86_64 1.10.3-10.el6 addons 2.0 M Transaction Summary ==================================================================================================================================================================================================== Install 1 Package(s) Total download size: 2.0 M Installed size: 2.8 M Is this ok [y/N]: y^H^CExiting on user Command [root@hadoop01 ~]# ^C [root@hadoop01 ~]# yum install krb5-server krb5-libs krb5-workstation Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Repository HDP-UTILS-1.1.0.21 is listed more than once in the configuration Repository InstallMedia is listed more than once in the configuration Setting up Install Process Package krb5-libs-1.10.3-10.el6.x86_64 already installed and latest version Package krb5-workstation-1.10.3-10.el6.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package krb5-server.x86_64 0:1.10.3-10.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================= Package Arch Version Repository Size ================================================================================================================================================= Installing: krb5-server x86_64 1.10.3-10.el6 addons 2.0 M Transaction Summary ================================================================================================================================================= Install 1 Package(s) Total download size: 2.0 M Installed size: 2.8 M Is this ok [y/N]: y Downloading Packages: krb5-server-1.10.3-10.el6.x86_64.rpm | 2.0 MB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : krb5-server-1.10.3-10.el6.x86_64 1/1 Verifying : krb5-server-1.10.3-10.el6.x86_64 1/1 Installed: krb5-server.x86_64 0:1.10.3-10.el6 Complete!

7.2. 检测主机名与IP映射

[root@hadoop01 ~]# cat /etc/hosts 172.29.0.25 hadoop-db 172.29.0.226 hadoop01 172.29.0.227 hadoop02 172.29.0.228 hadoop03 172.29.0.229 hadoop04 172.29.0.21 hadoop05 172.29.0.22 hadoop06 172.29.0.23 hadoop07

172.29.0.24 hadoop08 7.3. 修改配置文件

1、/etc/krb5.conf [root@hadoop01 ~]# cat /etc/krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] default_realm = EXAMPLE.COM dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true [realms] EXAMPLE.COM = { kdc = hadoop01 admin_server = hadoop01 } [domain_realm] .example.com = EXAMPLE.COM example.com = EXAMPLE.COM

7.4. 创建Kerberos数据库

[root@hadoop01 ~]# kdb5_util create -s Loading random data Initializing database '/var/kerberos/krb5kdc/principal' for realm 'EXAMPLE.COM', master key name 'K/M@EXAMPLE.COM' You will be prompted for the database Master Password. It is important that you NOT FORGET this password. Enter KDC database master key: Re-enter KDC database master key to verify:

7.5. 启动 KDC

[root@hadoop01 ~]# service krb5kdc start 正在启动 Kerberos 5 KDC：[确定] [root@hadoop01 ~]# service kadmin start 正在启动 Kerberos 5 Admin Server：[确定]

7.6. 设置KDC自启动

[root@hadoop01 ~]# chkconfig krb5kdc on [root@hadoop01 ~]# chkconfig kadmin on

7.7. 创建Kerberos管理员

[root@hadoop01 ~]# kadmin.local -q \Authenticating as principal root/admin@EXAMPLE.COM with password. WARNING: no policy specified for admin/admin@EXAMPLE.COM; defaulting to no policy Enter password for principal \ Re-enter password for principal \ Principal \ 7.8. 重启KDC

[root@hadoop01 ~]# service kadmin restart 停止 Kerberos 5 Admin Server：[确定] 正在启动 Kerberos 5 Admin Server：[确定]

7.9. 安装JCE

JCE（Java Cryptography Extension）是一组包，它们提供用于加密、密钥生成和协商以及 Message Authentication Code（MAC）算法的框架和实现。 它提供对对称、不对称、块和流密码的加密支持，它还支持安全流和密封的对象。它不对外出口，用它开发完成封装后将无法调用。 补充2：

If you are using Oracle JDK, you must distribute and install the JCE on all hosts in the cluster, including the Ambari Server. Be sure to restart Ambari Server after installng the JCE. If you are using OpenJDK, some distributions of the OpenJDK come with unlimited strength JCE automatically and therefore, installation of JCE is not required. For Oracle JDK 1.8:

http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

For Oracle JDK 1.7:

http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html 12.将下载的JCE解压并覆盖$JAVA_HOME/jre/lib/security/目录下的文件（所有节点） [root@hadoop-db ~]# unzip UnlimitedJCEPolicyJDK7.zip Archive: UnlimitedJCEPolicyJDK7.zip creating: UnlimitedJCEPolicy/

inflating: UnlimitedJCEPolicy/US_export_policy.jar inflating: UnlimitedJCEPolicy/local_policy.jar inflating: UnlimitedJCEPolicy/README.txt [root@hadoop-db ~]# cd UnlimitedJCEPolicy [root@hadoop-db UnlimitedJCEPolicy]# ll 总用量 16

-rw-rw-r-- 1 root root 2500 6月 1 2011 local_policy.jar -rw-r--r-- 1 root root 7289 6月 1 2011 README.txt

-rw-rw-r-- 1 root root 2487 6月 1 2011 US_export_policy.jar

[root@hadoop-db UnlimitedJCEPolicy]# cp *.jar /opt/java/jre/lib/security/ cp：是否覆盖\？ y

cp：是否覆盖\？ y [root@hadoop-db UnlimitedJCEPolicy]# cd /opt/java/jre/lib/security/ [root@hadoop-db security]# ll 总用量 136

-rw-r--r-- 1 root root 3890 9月 30 09:55 blacklist -rw-r--r-- 1 root root 92776 9月 30 09:55 cacerts

-rw-r--r-- 1 root root 158 9月 30 09:55 javafx.policy -rw-r--r-- 1 root root 2593 9月 30 09:55 java.policy -rw-r--r-- 1 root root 17838 9月 30 09:55 java.security -rw-r--r-- 1 root root 98 9月 30 09:55 javaws.policy -rw-r--r-- 1 root root 2500 12月 7 17:01 local_policy.jar -rw-r--r-- 1 root root 0 9月 30 09:55 trusted.libraries -rw-r--r-- 1 root root 2487 12月 7 17:01 US_export_policy.jar

#替换其它节点的JCE

[root@hadoop-db security]# scp *.jar hadoop01:/opt/java/jre/lib/security/

local_policy.jar 100% 2500 2.4KB/s 00:00

US_export_policy.jar 100% 2487 2.4KB/s 00:00

[root@hadoop-db security]# scp *.jar ws1dn1:/opt/java/jre/lib/security/ The authenticity of host 'ws1dn1 (192.168.1.61)' can't be established. RSA key fingerprint is b4:59:18:46:76:2e:27:e2:2c:5b:36:9b:49:b8:72:7b. Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'ws1dn1,192.168.1.61' (RSA) to the list of known hosts.

local_policy.jar 100% 2500 2.4KB/s 00:00

US_export_policy.jar 100% 2487 2.4KB/s 00:00

[root@hadoop-db security]# scp *.jar ws1dn2:/opt/java/jre/lib/security/ The authenticity of host 'ws1dn2 (192.168.1.62)' can't be established. RSA key fingerprint is 45:47:8e:56:08:66:2b:23:a6:a6:f1:09:14:8a:64:91. Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'ws1dn2,192.168.1.62' (RSA) to the list of known hosts.

local_policy.jar 100% 2500 2.4KB/s 00:00

US_export_policy.jar 100% 2487 2.4KB/s 00:00

[root@hadoop-db security]# scp *.jar ws1dn3:/opt/java/jre/lib/security/ The authenticity of host 'ws1dn3 (192.168.1.63)' can't be established. RSA key fingerprint is 02:5c:3b:ce:20:f1:27:f9:6e:ca:f9:95:f6:66:84:6f. Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'ws1dn3,192.168.1.63' (RSA) to the list of known hosts.

local_policy.jar 100% 2500 2.4KB/s 00:00 US_export_policy.jar

7.10. 重启Ambari-Server 7.11. 开始安装向导

输入以上创建的KDC信息

点击下一步

测试

下一步，不修改任何东西

8. 安装问题FAQ

7.12. 安装挂载CentOS 6.4 DVD

安装挂载CentOS 6.4 DVD到各个节点上，不然安装过程会失败（所有节点操作一遍） mount -t iso9660 -o loop /dev/cdrom /mnt/cdrom 9. Hortonworks Hadoop 2.6组件服务全家福

本文来源：https://www.bwwdw.com/article/amq.html