LVS+keepalived负载均衡（FULLNAT模式）

LVS FULLNAT模式安装

By 清风徐来 605612253@qq.com

1 部署规划

依照淘宝开源的FULLNAT模式LVS，规划使用版本信息： Linux内核：2.6.32-220.23.1.el6 LVS版本：version 1.2.1 Keepalived版本：v1.2.2 序号 主机IP 域名 作用 备注 1 10.142.67.121 TEST-DEV-121 Master 编译内核+LVS+keepalived 2 10.142.67.122 TEST-DEV-122 Backup 编译内核+LVS+keepalived 3 10.142.78.74 TEST-BDD-074 Hiveserver2 编译内核+hiveserver2 4 10.142.78.76 TEST-BDD-076 Hiveserver2 编译内核+hiveserver2 2 LVS安装

2.1 内核编译

内核编译需要在master和backup节点都执行，以下以master节点为例 安装脚本：

compilekerna-LVSmaster.sh

由于不能上外网，所以提前把对应的安装包下好 下载的暂时放在家目录 [op@TEST-DEV-121 ~]$ ls

compilekerna-LVSmaster.sh kernel-2.6.32-220.23.1.el6.src.rpm Lvs-fullnat-synproxy.tar.gz

安装

此处由于是编译内核，使用root用户来执行，以免遇到各种权限问题 [op@TEST-DEV-121 ~]$ sudo chmod +x compilekerna-LVSmaster.sh

[op@TEST-DEV-121 ~]$ sudo su – [root@TEST-DEV-121 ~]# cd /home/op/ [root@TEST-DEV-121 op]# uname -r 2.6.32-431.el6.x86_64 # 此处是编译前看一下内核版本号

[root@TEST-DEV-121 op]# ./compilekerna-LVSmaster.sh 安装依赖包的时候，发现默认yum源没有这些东西 Error Downloading Packages:

newt-devel-0.52.11-3.el6.x86_64: failure:

Packages/newt-devel-0.52.11-3.el6.x86_64.rpm from base: [Errno 256] No more mirrors to try.

slang-devel-2.2.1-1.el6.x86_64: failure:

Packages/slang-devel-2.2.1-1.el6.x86_64.rpm from base: [Errno 256] No more mirrors to try.

asciidoc-8.4.5-4.1.el6.noarch: failure: Packages/asciidoc-8.4.5-4.1.el6.noarch.rpm from base: [Errno 256] No more mirrors to try. 所以，要自己下

newt-devel-0.52.11-3.el6.x86_64.rpmslang-devel-2.2.1-1.el6.x86_64.rpmasciidoc-8.4.5-4.1.el6.noarch.rpm

[root@TEST-DEV-121 op]# rpm -ivh slang-devel-2.2.1-1.el6.x86_64.rpm

warning: slang-devel-2.2.1-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 192a7d7d: NOKEY

Preparing...

########################################### [100%] 1:slang-devel

########################################### [100%] [root@TEST-DEV-121 op]# rpm -ivh

newt-devel-0.52.11-3.el6.x86_64.rpm

warning: newt-devel-0.52.11-3.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY

Preparing...

########################################### [100%] 1:newt-devel

########################################### [100%] [root@TEST-DEV-121 op]# rpm -ivh asciidoc-8.4.5-4.1.el6.noarch.rpm

warning: asciidoc-8.4.5-4.1.el6.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 192a7d7d: NOKEY

Preparing...

########################################### [100%] 1:asciidoc

########################################### [100%] 再次运行即可 编译结束后： DEPMOD 2.6.32

sh

/home/op/rpms/BUILD/kernel-2.6.32-220.23.1.el6/linux-2.6.32-220.23.1.el6.x86_64/arch/x86/boot/install.sh 2.6.32 arch/x86/boot/bzImage \\

System.map \

ERROR: modinfo: could not find module xen_procfs ERROR: modinfo: could not find module xen_scsifront ERROR: modinfo: could not find module xen_hcall

感觉像是虚拟机是xen做的底层支持。但是kernel里面没有xen支持模块，所以会报错。所以应该在时提及上没问题。 之后修改启动顺序

[root@TEST-DEV-121 ~]# vim /boot/grub/grub.conf default=0

之后重启查看是否内核变成了2.6

[root@TEST-DEV-121 ~]# sudo reboot 之后

[op@TEST-DEV-121 ~]$ uname -r 2.6.32

2.2 Keepalived安装

注意，一定要先安装keepalived服务，再安装LVS服务，否则会报错；同时两个节点都需要安装。

2.2.1 安装包

http://kb.linuxvirtualserver.org/images/a/a5/Lvs-fullnat-synproxy.tar.gz 里面有个tools压缩包就是了

2.2.2 部署安装

解压

[op@TEST-DEV-121 ~]$ tar -zxvf lvs-tools.tar.gz

安装依赖

[op@TEST-DEV-121 keepalived]$ sudo yum install -y openssl openssl-devel popt-devel kernel-devel popt*

报错，发现没有这个包

Error Downloading Packages:

popt-static-1.13-7.el6.x86_64: failure: Packages/popt-static-1.13-7.el6.x86_64.rpm from base: [Errno 256] No more mirrors to try. 下载对应的包即可

popt-static-1.13-7.el6.x86_64.rpm

[op@TEST-DEV-121 ~]$ sudo rpm -ivh popt-static-1.13-7.el6.x86_64.rpm

warning: popt-static-1.13-7.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID c105b9de: NOKEY

Preparing...

########################################### [100%]

1:popt-static

########################################### [100%] 编译安装keepalived

[op@TEST-DEV-121 ~]$ cd tools/

[op@TEST-DEV-121 tools]$ cd keepalived/ [op@TEST-DEV-121 keepalived]$ ./configure

--with-kernel-dir=\检查结果

Keepalived configuration ------------------------

Keepalived version : 1.2.2 Compiler : gcc Compiler flags : -g -O2

Extra Lib : -lpopt -lssl -lcrypto Use IPVS Framework : Yes IPVS sync daemon support : Yes IPVS use libnl : No Use VRRP Framework : Yes Use Debug flags : No 三个YES即可。

[op@TEST-DEV-121 keepalived]$ make

[op@TEST-DEV-121 keepalived]$ sudo make install 添加系统管理

[op@TEST-DEV-121 keepalived]$ sudo cp

/usr/local/etc/rc.d/init.d/keepalived /etc/init.d/ [op@TEST-DEV-121 keepalived]$ sudo cp

/usr/local/etc/sysconfig/keepalived /etc/sysconfig/ [op@TEST-DEV-121 keepalived]$ sudo mkdir /etc/keepalived [op@TEST-DEV-121 keepalived]$ sudo cp

/usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

[op@TEST-DEV-121 keepalived]$ sudo cp /usr/local/sbin/keepalived /usr/sbin/

由于默认keepalived的日志是在message里面，现在指定文件接受keepalived日志

[op@TEST-DEV-121 keepalived]$ sudo vim /etc/sysconfig/keepalived #KEEPALIVED_OPTIONS=\KEEPALIVED_OPTIONS=\

然后在rsyslog中添加一行

[op@TEST-DEV-121 keepalived]$ sudo vim /etc/rsyslog.conf #keepalived log

local0.* /var/log/keepalived.log

修改配置文件后启动

[op@TEST-DEV-121 ~]$ sudo /etc/init.d/rsyslog restart 关闭系统日志记录器： [确定] 启动系统日志记录器： [确定] 启动

[op@TEST-DEV-121 keepalived]$ sudo service keepalived start

2.3 Lvs安装

安装包同上，都在tools目录下 编译安装

[op@TEST-DEV-121 ~]$ cd tools/ipvsadm/ [op@TEST-DEV-121 ipvsadm]$ sudo make

[op@TEST-DEV-121 ipvsadm]$ sudo make install 验证

[op@TEST-DEV-121 ipvsadm]$ sudo ipvsadm IP Virtual Server version 1.2.1 (size=4194304)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

[op@TEST-DEV-121 ~]$ lsmod | grep ip_vs ip_vs 161155 0 ipv6 323428 283 ip_vs

3 RealServer端安装

3.1 源码编译

下载地址：

http://kb.linuxvirtualserver.org/images/3/34/Linux-2.6.32-220.23.1.el6.x86_64.rs.src.tar.gz 解压

[op@TEST-BDD-064 ding]$ ls

Linux-2.6.32-220.23.1.el6.x86_64.rs.src.tar.gz [op@TEST-BDD-064 ding]$ tar -zxvf

Linux-2.6.32-220.23.1.el6.x86_64.rs.src.tar.gz 修改.config文件 安装几个依赖包

yum install -y openssl openssl-devel popt-devel kernel-devel popt* [op@TEST-BDD-074 ~]$ cd ding/

[op@TEST-BDD-074 ding]$ cd linux-2.6.32-220.23.1.el6.x86_64.rs/ [op@TEST-BDD-074 linux-2.6.32-220.23.1.el6.x86_64.rs]$ [op@TEST-BDD-074

linux-2.6.32-220.23.1.el6.x86_64.rs]$ vim .config 将CONFIG_IPV6=M改成CONFIG_IPV6=y

编辑makefile，可以加一下后缀，这样uname –r的时候可以显示出来，当然不加也可以。 我修改了一下

[op@TEST-BDD-074 linux-2.6.32-220.23.1.el6.x86_64.rs]$ vim Makefile

EXTRAVERSION =-220.el6.x86_64 之后开始编译

[op@TEST-BDD-074 linux-2.6.32-220.23.1.el6.x86_64.rs]$ make 此后会出现新功能增加的提示

The private TCP option for support Taobao LVS full-NAT feature (TOA) [M/n/y/?] (NEW) y 输入y即可 结束时输出

[op@TEST-BDD-074 linux-2.6.32-220.23.1.el6.x86_64.rs]$ sudo make modules_install

[op@TEST-BDD-074 linux-2.6.32-220.23.1.el6.x86_64.rs]$ sudo make install

会有几个报错：

sh

/usr/op/ding/linux-2.6.32-220.23.1.el6.x86_64.rs/arch/x86/boot/install.sh 2.6.32-220.el6.x86_64 arch/x86/boot/bzImage \\ System.map \

ERROR: modinfo: could not find module xen_hcall ERROR: modinfo: could not find module xen_procfs ERROR: modinfo: could not find module ipv6

ERROR: modinfo: could not find module xen_scsifront ERROR: modinfo: could not find module xen_balloon 报错原因：

因为虚拟机底层使用了xen作为虚拟机引擎，默认kernel没有包含

对于ipv6的报错，是因为之前编译的时候修改了参数，最终没有发现，是因为包里面没有。 更改内核启动顺序：

[op@TEST-BDD-074 ~]$ sudo vim /boot/grub/grub.conf default=0 之后重启即可

本文来源：https://www.bwwdw.com/article/mjyw.html