服务器之家：专注于服务器技术及软件下载分享
分类导航

Mysql|Sql Server|Oracle|Redis|MongoDB|PostgreSQL|Sqlite|DB2|mariadb|Access|数据库技术|

服务器之家 - 数据库 - Mysql - Keepalived+HAProxy实现MySQL高可用负载均衡的配置

Keepalived+HAProxy实现MySQL高可用负载均衡的配置

2020-06-01 17:23乐晨 Mysql

这篇文章主要介绍了keepalived+haproxy实现MySQL高可用负载均衡的配置方法,通过这两个软件可以有效地使MySQL脱离故障及进行健康检测,需要的朋友可以参考下

 Keepalived

由于在生产环境使用了mysqlcluster,需要实现高可用负载均衡,这里提供了keepalived+haproxy来实现.

      keepalived主要功能是实现真实机器的故障隔离及负载均衡器间的失败切换.可在第3,4,5层交换.它通过VRRPv2(Virtual Router Redundancy Protocol) stack实现的.

      Layer3:Keepalived会定期向服务器群中的服务器.发送一个ICMP的数据包（既我们平时用的Ping程序）,如果发现某台服务的IP地址没有激活，Keepalived便报告这台服务器失效，并将它从服务器群中剔除，这种情况的典型例子是某台服务器被非法关机。Layer3的方式是以服务器的IP地址是否有效作为服务器工作正常与否的标准。

     Layer4:主要以TCP端口的状态来决定服务器工作正常与否。如web server的服务端口一般是80，如果Keepalived检测到80端口没有启动，则Keepalived将把这台服务器从服务器群中剔除。

     Layer5:在网络上占用的带宽也要大一些。Keepalived将根据用户的设定检查服务器程序的运行是否正常，如果与用户的设定不相符，则Keepalived将把服务器从服务器群中剔除。

Software Design

Keepalived+HAProxy实现MySQL高可用负载均衡的配置

keepalived启动后会有单个进程

?
1
2
3
8352 ?    Ss   0:00 /usr/sbin/keepalived
8353 ?    S   0:00 _ /usr/sbin/keepalived
8356 ?    S   0:01 _ /usr/sbin/keepalived

父进程:内存管理,子进程管理等等

子进程:VRRP子进程

子进程:Healthchecking 子进程
实例

2台mysqlcluster 10.1.6.203 master  10.1.6.205 backup

vip 10.1.6.173

目的访问10.1.6.173 3366端口 分别轮询通过haproxy转发到10.1.6.203 3306 和10.1.6.205 3306

mysqlcluster搭建参照之前博客,这里在2台机上安装keepalived

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
root@10.1.6.203:~# apt-get install keepalived
root@10.1.6.203:~# cat /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
    script "killall -0 haproxy"  # verify the pid existance
    interval 2          # check every 2 seconds
    weight -2          # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
    interface eth1        # interface to monitor
    state MASTER        
    virtual_router_id 51     # Assign one ID for this route
    priority 101         # 101 on master, 100 on backup
    nopreempt
    debug
 
    virtual_ipaddress {
        10.1.6.173
    }
 
    track_script {    #注意大括号空格
        chk_haproxy
    }
 
    notify_master /etc/keepalived/scripts/start_haproxy.sh #表示当切换到master状态时,要执行的脚本
    notify_fault /etc/keepalived/scripts/stop_keepalived.sh #故障时执行的脚本
    notify_stop  /etc/keepalived/scripts/stop_haproxy.sh #keepalived停止运行前运行notify_stop指定的脚本 }

VRRPD配置包括三个类:

  • VRRP同步组(synchroization group)
  • VRRP实例(VRRP Instance)
  • VRRP脚本

这里使用了 VRRP实例, VRRP脚本

注意配置选项:

stat:指定instance(Initial)的初始状态，就是说在配置好后，这台服务器的初始状态就是这里指定的，但这里指定的不算，还是得要通过竞选通过优先级来确定，里如果这里设置为master，但如若他的优先级不及另外一台，那么这台在发送通告时，会发送自己的优先级，另外一台发现优先级不如自己的高，那么他会就回抢占为master

  • interface：实例绑定的网卡，因为在配置虚拟IP的时候必须是在已有的网卡上添加的
  • priority 101：设置本节点的优先级，优先级高的为master
  • debug：debug级别
  • nopreempt:设置为不抢占
?
1
2
3
4
5
vrrp_script chk_haproxy {
    script "killall -0 haproxy"  # verify the pid existance
    interval 2          # check every 2 seconds 脚本执行间隔
    weight -2          # add 2 points of prio if OK 脚本结果导致的优先级变更：2表示优先级+2；-2则表示优先级-2
}

然后在实例(vrrp_instance)里面引用，有点类似脚本里面的函数引用一样：先定义，后引用函数名

?
1
2
3
track_script {
    chk_haproxy
}

注意：VRRP脚本(vrrp_script)和VRRP实例(vrrp_instance)属于同一个级别

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
root@10.1.6.203:scripts# cat start_haproxy.sh
#!/bin/bash
 
sleep 5
get=`ip addr |grep 10.1.6.173 |wc -l`
echo $get >> /etc/keepalived/scripts/start_ha.log
 
if [ $get -eq 1 ]
then
    echo "`date +%c` success to get vip" >> /etc/keepalived/scripts/start_ha.log
    /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg
else
    echo "`date +%c` can not get vip" >> /etc/keepalived/scripts/start_ha.log
fi
root@10.1.6.203:scripts# cat stop_keepalived.sh
#!/bin/bash
 
pid=`pidof keepalived`
if [ $pid == "" ]
then
  echo "`date +%c` no keepalived process id" >> /etc/keepalived/scripts/stop_keep.log
else
  echo "`date +%c` will stop keepalived " >> /etc/keepalived/scripts/stop_keep.log
  /etc/init.d/keepalived stop
fi
 
/etc/init.d/keepalived stop
 
root@10.1.6.203:scripts# cat stop_haproxy.sh
#!/bin/bash
 
pid=`pidof haproxy`
echo "`date +%c` stop haproxy" >> /etc/keepalived/scripts/stop_ha.log
kill -9 $pid

同理配置10.1.6.205

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
root@10.1.6.205:~# cat /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
  script "killall -0 haproxy"  # verify the pid existance
  interval 2          # check every 2 seconds
  weight 2           # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
  interface eth1        # interface to monitor
  state BACKUP
  virtual_router_id 51     # Assign one ID for this route
  priority 100         # 101 on master, 100 on backup
  virtual_ipaddress {
    10.1.6.173
  }
 
  track_script {
    chk_haproxy
  }
 
notify_master /etc/keepalived/scripts/start_haproxy.sh
notify_fault /etc/keepalived/scripts/stop_keepalived.sh
notify_stop /etc/keepalived/scripts/stop_haproxy.sh
 
}

 HAProxy

下面再介绍下haproxy

       HAProxy是一款基于TCP(第四层)和HTTP(第七层)应用的代理软件,它也可作为负载均衡器.可以支持数以万计的并发连接.同时可以保护服务器不暴露到网络上,通过端口映射.它还自带监控服务器状态的页面.

      安装haproxy

?
1
2
3
4
5
wget -O/tmp/haproxy-1.4.22.tar.gz http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.22.tar.gz
tar xvfz /tmp/haproxy-1.4.22.tar.gz -C /tmp/
cd /tmp/haproxy-1.4.22
make TARGET=linux26
make install

      haproxy需要对每一个mysqlcluster服务器进行健康检查

1.在2台主机分别配置haproxy.cfg

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
root@10.1.6.203:scripts# cat /etc/haproxy/haproxy.cfg
global
    maxconn 51200 #默认最大连接数
    #uid 99
    #gid 99
    daemon    #以后台形式运行haproxy
    #quiet
    nbproc 1   #进程数量(可以设置多个进程提高性能)
    pidfile /etc/haproxy/haproxy.pid #haproxy的pid存放路径,启动进程的用户必须有权限访问此文件
 
defaults
    mode tcp      #所处理的类别 (#7层 http;4层tcp )
    option redispatch  #serverId对应的服务器挂掉后,强制定向到其他健康的服务器
    option abortonclose #当服务器负载很高的时候，自动结束掉当前队列处理比较久的连接
    timeout connect 5000s  #连接超时
    timeout client 50000s #客户端超时
    timeout server 50000s  #服务器超时
    log 127.0.0.1 local0  #错误日志记录
    balance roundrobin  #默认的负载均衡的方式,轮询方式
 
listen proxy
    bind 10.1.6.173:3366  #监听端口
    mode tcp        #http的7层模式
    option httpchk    #心跳检测的文件
    server db1 10.1.6.203:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3   #服务器定义,check inter 12000是检测心跳频率 rise 3是3次正确认为服务器可用， fall 3是3次失败认为服务器不可用，weight代表权重
    server db2 10.1.6.205:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3
 
listen haproxy_stats
    mode http
    bind 10.1.6.173:8888
    option httplog
    stats refresh 5s 
    stats uri /status #网站健康检测URL，用来检测HAProxy管理的网站是否可以用，正常返回200，不正常返回503
    stats realm Haproxy Manager
    stats auth admin:p@a1SZs24 #账号密码
 
root@10.1.6.205:~$ cat /etc/haproxy/haproxy.cfg
global
    maxconn 51200
    #uid 99
    #gid 99
    daemon
    #quiet
    nbproc 1
    pidfile /etc/haproxy/haproxy.pid
 
defaults
    mode tcp
    option redispatch 
    option abortonclose
    timeout connect 5000s
    timeout client 50000s
    timeout server 50000s
    log 127.0.0.1 local0
    balance roundrobin
 
listen proxy
    bind 10.1.6.173:3366
    mode tcp
    option httpchk
    server db1 10.1.6.203:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3
    server db2 10.1.6.205:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3
 
listen haproxy_stats
    mode http
    bind 10.1.6.173:8888
    option httplog
    stats refresh 5s 
    stats uri /status
    stats realm Haproxy Manager
    stats auth admin:p@a1SZs24

2.安装xinetd

?
1
root@10.1.6.203:~# apt-get install xinetd

3.在每个节点添加xinetd服务脚本和mysqlchk端口号

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
root@10.1.6.203:~# vim /etc/xinetd.d/mysqlchk
# default: on
# description: mysqlchk
service mysqlchk         #需要在servive定义
{
    flags      = REUSE
    socket_type   = stream
    port      = 9222
    wait      = no
    user      = nobody
    server     = /opt/mysqlchk
    log_on_failure += USERID
    disable     = no
    per_source   = UNLIMITED
    bind      = 10.1.6.173
}
 
root@10.1.6.203:~# vim /etc/services
mysqlchk    9222/tcp            # mysqlchk

4.编写mysqlchk监控服务脚本

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
root@10.1.6.203:~# ls -l /opt/mysqlchk
-rwxr--r-- 1 nobody root 1994 2013-09-17 11:27 /opt/mysqlchk
root@10.1.6.203:~# cat /opt/mysqlchk
#!/bin/bash
#
# This script checks if a mysql server is healthy running on localhost. It will
# return:
# "HTTP/1.x 200 OK " (if mysql is running smoothly)
# - OR -
# "HTTP/1.x 500 Internal Server Error " (else)
#
# The purpose of this script is make haproxy capable of monitoring mysql properly
#
 
MYSQL_HOST="localhost"
MYSQL_SOCKET="/var/run/mysqld/mysqld.sock"
MYSQL_USERNAME="mysqlchkusr"   #该账户密码需要在mysql里添加
MYSQL_PASSWORD="secret"
MYSQL_OPTS="-N -q -A"
TMP_FILE="/dev/shm/mysqlchk.$$.out"
ERR_FILE="/dev/shm/mysqlchk.$$.err"
FORCE_FAIL="/dev/shm/proxyoff"
MYSQL_BIN="/opt/mysqlcluster/mysql-cluster-gpl-7.2.6-linux2.6-x86_64/bin/mysql"
CHECK_QUERY="select 1"
 
preflight_check()
{
  for I in "$TMP_FILE" "$ERR_FILE"; do
    if [ -f "$I" ]; then
      if [ ! -w $I ]; then
        echo -e "HTTP/1.1 503 Service Unavailable "
        echo -e "Content-Type: Content-Type: text/plain "
        echo -e " "
        echo -e "Cannot write to $I "
        echo -e " "
        exit 1
      fi
    fi
  done
}
 
return_ok()
{
  echo -e "HTTP/1.1 200 OK "
  echo -e "Content-Type: text/html "
  echo -e "Content-Length: 43 "
  echo -e " "
  echo -e "<html><body>MySQL is running.</body></html> "
  echo -e " "
  rm $ERR_FILE $TMP_FILE
  exit 0
}
return_fail()
{
  echo -e "HTTP/1.1 503 Service Unavailable "
  echo -e "Content-Type: text/html "
  echo -e "Content-Length: 42 "
  echo -e " "
  echo -e "<html><body>MySQL is *down*.</body></html> "
  sed -e 's/ $/ /' $ERR_FILE
  echo -e " "
  rm $ERR_FILE $TMP_FILE
  exit 1
}
preflight_check
if [ -f "$FORCE_FAIL" ]; then
    echo "$FORCE_FAIL found" > $ERR_FILE
    return_fail;
fi
$MYSQL_BIN $MYSQL_OPTS --host=$MYSQL_HOST --socket=$MYSQL_SOCKET --user=$MYSQL_USERNAME --password=$MYSQL_PASSWORD -e "$CHECK_QUERY" > $TMP_FILE 2> $ERR_FILE
if [ $? -ne 0 ]; then
    return_fail;
fi
return_ok;

测试

2个节点开启keepalived(主节点会获得vip,自动拉起haproxy),xinetd

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@10.1.6.203:~# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
  link/ether 00:26:b9:36:0f:81 brd ff:ff:ff:ff:ff:ff
  inet 211.151.105.186/26 brd 211.151.105.191 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
  link/ether 00:26:b9:36:0f:83 brd ff:ff:ff:ff:ff:ff
  inet 10.1.6.203/24 brd 10.1.6.255 scope global eth1
  inet 10.1.6.173/32 scope global eth1
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
  link/ether 00:26:b9:36:0f:85 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
  link/ether 00:26:b9:36:0f:87 brd ff:ff:ff:ff:ff:ff
root@10.1.6.203:~# netstat -tunlp | grep ha
tcp    0   0 10.1.6.173:3366   0.0.0.0:*        LISTEN   1042/haproxy
tcp    0   0 10.1.6.203:8888   0.0.0.0:*        LISTEN   1042/haproxy
udp    0   0 0.0.0.0:56562      0.0.0.0:*              1042/haproxy
root@10.1.6.203:~# netstat -tunlp | grep xine
tcp    0   0 10.1.6.203:9222   0.0.0.0:*        LISTEN   30897/xinetd
root@10.1.6.203:~# ps -ef | grep haproxy
root   1042   1 0 Sep17 ?    00:00:00 /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg

测试:

通过vip10.1.6.173 3366访问cluster数据库(注意账户dave权限需要加3个ip10.1.6.203,10.1.6.205,10.1.6.173)

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
root@10.1.6.203:mgm# mysql -udave -p -h 10.1.6.173 -P 3366
Enter password:
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 1344316
Server version: 5.5.22-ndb-7.2.6-gpl-log MySQL Cluster Community Server (GPL)
 
Type 'help;' or 'h' for help. Type 'c' to clear the buffer.
 
mysql> show databases;
+--------------------+
| Database      |
+--------------------+
| information_schema |
| dave       |
| test        |
+--------------------+
3 rows in set (0.01 sec)
 
mysql>

手动分别使keepalive,haproxy,数据库挂掉.vip10.1.6.173会自动漂到10.1.6.205从上，并不影响vip的访问

通过vip,haproxy查看各节点状态

http://10.1.6.173:8888/status

Keepalived+HAProxy实现MySQL高可用负载均衡的配置

  • mysql
  • 负载均衡
  • Keepalived
  • HAProxy

    • 延伸 · 阅读

    精彩推荐
    最近更新
    编辑推荐
    阅读排行
    热门标签
    150 　 1044 　 1062 　 1449 　 1053 　 1045 　 1290 　 1819 　 2013 　 1406 　 1005 　 1862 　 1864 　 1146 　 2003 　 1175 　 1215 　 1615 　 1054 　 2006 　 1130 　 1366 　 10048 　 28000 　 1786 　 1135 　 1577 　 1665 　 1067 　 1093 　
    © 2019-2024 服务器之家(www.zzvips.com) 版权所有关于我们联系我们版权申明网站地图