Physical setup view :
For above setup we will use IPs as below;
Server 1 : 10.1.1.2
Server 2 : 10.1.1.3
VIP : 10.1.1.4
- Nginx stays at the top and it load balances the connections to tomcats using private IPs (10.1.1.2, 10.1.1.3).
- VIP configured using Keepalived service and it runs in both servers.
- Tomcat runs in both servers. In Tomcat, Redis host IP configured to be the VIP (10.1.1.4).
Initially, VIP will be owned by the server which starts first or the server where keepalived service starts first. We will call this the master server.
- When the master goes down, VIP will be taken over by the secondary server. When this happens, we use keepalived's notify script to make the redis as master and the failed server's redis will become the slave.
- If Nginx service or the Redis service goes down, then also the VIP will be moved to the secondary server. For this, we use keepalived's tracking script to float the VIP based on service status.
- Redis service init script also changed to check the VIP status at startup. It will check whether the specific server owns the VIP. If it has the VIP, Redis will change it's role to master.
Configuration of each service as a summary
Tomcat session replication with redis :
- To enable tomcat-redis session replication, you need to add redis session manager jar files to tomcat’s lib directory.
Note : copy “tomcat-redis-session-manager-1.1.jar” if java6 is used. Copy “tomcat-redis-session-manager-1.2-tomcat-7-java-7.jar” if java7 is used. Other two libs are same for both versions.
make sure to add commons-pool-1.6.jar and jedis-2.0.0.jar also to the lib directory.
- Edit “/$CATALINA_HOME/conf/context.xml” and add following under context
<Valve className="com.radiadesign.catalina.session.RedisSessionHandlerValve" />
<Manager className="com.radiadesign.catalina.session.RedisSessionManager"
host="10.1.1.4"
port="6379"
database="0"
maxInactiveInterval="60" />
* make sure redis listens on the server IP instead of lookback IP.
Sample nginx upstream configuration :
upstream tomcat {
server 10.1.1.2:8080;
server 10.1.1.3:8080;
check interval=3000 rise=2 fall=5 timeout=3000;
keepalive 16;
}
* Above will monitor the service port only. You can use an URI also to check the service status.
Keepalived configuration :
vrrp_script chk_myscript {
script "/etc/keepalived/scripts/track.sh"
interval 5
fall 2
rise 2
}
vrrp_instance vip_testweb {
state BACKUP
interface eth0
virtual_router_id 61
priority 101
advert_int 1
nopreempt
notify "/etc/keepalived/scripts/notify.sh"
track_script {
chk_myscript
}
virtual_ipaddress {
10.1.1.4/24
}
}
--------------------------------------------------------------------------------------------------------------------------
track.sh -- this would trigger the VIP float based on nginx and redis service status.
#!/bin/bash
#### This script will check the redis by adding a key with current time as value. And will check the nginx service status also.
#### If both are running prpoerly script should return "0" as the result code.
ngxstat=`/usr/bin/pgrep nginx | wc -l`
time=`date +%T`
redstat=`/usr/bin/redis-cli set health_check $time`
cmdstat=`echo $?`
if [ $cmdstat -ne 0 ];
then
redstat=err:$cmdstat
else
redstat="OK"
fi
echo redstat $redstat
echo ngxstat $ngxstat
if [ $redstat == "OK" ] && [ $ngxstat != 0 ];
then
echo services OK
exit 0
else
echo services ERROR
exit 123
fi
------------------------------------------------------------------------------------------------------------------------------------------------------
notify.sh -- this will trigger the script with the keepalived status (Master/Backup/Fault) as parameter. Based on keepalived VRRP instance status, redis role will be changed. In the script put the proper peer IP and port.
#!/bin/bash
TYPE=$1
NAME=$2
STATE=$3
PEER_IP="10.1.1.3"
PEER_PORT="6379"
echo $TYPE $NAME $STATE
case $STATE in
"MASTER") /usr/bin/redis-cli slaveof no one
exit 0
;;
"BACKUP") /usr/bin/redis-cli slaveof $PEER_IP $PEER_PORT
exit 0
;;
"FAULT") /usr/bin/redis-cli slaveof no one
exit 0
;;
*) echo "unknown state"
exit 1
;;
esac
-------------------------------------------------------------------------------------------------------------------------
Redis init script modification :
Add below lines to redis init script. This way, when redis starts it will decide to become master or slave. In the script put the proper peer IP and port.
VIP="10.1.1.4"
PEER_IP="10.1.1.3"
PEER_PORT="6379"
vipcheck() {
vip=`ip add sh | grep $VIP | wc -l`
if [ $vip -eq 1 ]; then
echo
echo $"VRRP is MASTER"
/usr/bin/redis-cli slaveof no one
else
echo
echo $"VRRP is BACKUP"
/usr/bin/redis-cli slaveof $PEER_IP $PEER_PORT
fi
}
- Call the vipcheck() function at start()
start() {
[ -x $redis ] || exit 5
[ -f $REDIS_CONF_FILE ] || exit 6
echo -n $"Starting $prog: "
daemon $redis $REDIS_CONF_FILE
retval=$?
vipcheck
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
After applying above configuration in both servers, you can test the failover by stopping keepalived service or nginx service or even redis service.
All keepalived events will be logged at "/var/log/messages"
VIP can be seen by typing the command, "ip add sh"
Even though configurations and steps are not in detail, I hope this would help you to understand how I have achieved redis level high availability.
No comments:
Post a Comment