DNSMASQ as internal DNS server with a PHP mangement page (GUI)

Environment

• Apache as the web server. (In my server apache set to run in port 9000)

          PHP and SQLite DB files located at "/var/www/html/dnsmasq" and the same path set as the root directory.

         service accessed via http://server1/dnsmasq/login.php

• login.php
• admin.php - add/remove/modify host entries and update the DB
• admin.db - user logins
• db.php - DB connections
• hosts.db - host entry DB
• update.sh - updates the local hosts file when invoked from the PHP file
• dnsmasq.conf - replace dnsmasq initial file with this

* make sure to grant dnsmasq service reload access to apache user in visudo

   # visudo

   -- add below entry

    apache  ALL = NOPASSWD: /etc/init.d/dnsmasq reload

* current user in admin.db is "admin", password is "password"

   to change the login or to add a user use below sql commands;

   

   change admin user password ---  sqlite3 admin.db "update logins set password='123456' where username='admin'"

   add a user --- sqlite3 admin.db "insert into logins (username,password) values ('admin','password')"

https://github.com/lahiruprasad/dnsmasq-php/blob/gh-pages/dnsmasq.zip
https://github.com/lahiruprasad/dnsmasq-php/blob/gh-pages/dnsmasq.conf

Configure HornetQ in JBoss EAP 6 to work with NAT (Eucalyptus Cloud network environment)

You need to apply below changes in "server\default\deploy\hornetq\hornetq-configuration.xml" file.

<connectors>
      <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="192.168.1.5 – this is the NATTED IP"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </connector>
     
      <connector name="netty-throughput">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="192.168.1.5 – this is the NATTED IP "/>
         <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
         <param key="batch-delay" value="50"/>
      </connector>

      <connector name="in-vm">
         <factory-class>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</factory-class>
         <param key="server-id" value="${hornetq.server-id:0}"/>
      </connector>

   </connectors>

   <acceptors>  
      <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host"  value="10.1.2.9 – this is the server private IP"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </acceptor>
     
      <acceptor name="netty-throughput">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host"  value="10.1.2.9 – this is the server private IP "/>
         <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
         <param key="batch-delay" value="50"/>
         <param key="direct-deliver" value="false"/>
      </acceptor>

      <acceptor name="in-vm">
        <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>
        <param key="server-id" value="0"/>
      </acceptor>

   </acceptors>
Change the "run.conf" file as below;

-Djboss.bind.address=10.1.2.9  -Djava.rmi.server.hostname=192.168.1.5

Few important Windows Resource Kit tools

Following are some tools from the Windows Resource Kit, which I think are very important.

You can download the Windows Resource Kit from here.

These will work at any location of the command prompt when copied to "c:\windows\system32"

• consume

          From this you can run a simple stress test on your PC's memory, CPU and disks.

          This is similar to the Linux "stress" command.

          CAUTION !!! use with care if you are going run this in a production server.

          C:\Users\lahiru>consume /?
          Universal Resource Consumer - Just an innocent stress program, v 0.1.0
          Copyright (c) 1998, 1999, Microsoft Corporation

          consume RESOURCE [-time SECONDS]

          RESOURCE can be one of the following:

          -physical-memory
          -page-file
          -disk-space
          -cpu-time
          -kernel-pool

• createfil

          Use this command if you need to create a big dummy file, which is equal to Linux "dd" command.

          C:\Users\lahiru>creatfil /?
 
          Usage: creatfil FileName [FileSize]
       
          -? :  This message
          -FileName -- name of the new file
          -FileSize -- size of file in KBytes, default is 1024 KBytes

• tail

        Simple tail command which is similar to Linux "tail" command.

          C:\Users\lahiru>tail /?

          usage: TAIL [switches] [filename]*
          switches: [-?] display this message
          [-n] display last n lines of each file (default 10)
          [-f filename] keep checking filename for new lines 
        

• robocopy

        This is more like Linux "rsync'.

          Refer "robocopy /?" or "robocopy.doc" for command usage/parameters.

• qgrep

          This command is useful if you need to find a string from set of files.

• PortQry 

        portqry.exe does not come in resource kit. You can download it from here.
        
        portqry is useful if you want to test a port status and use it in a batch file. Because telnet can't be used to get port status to a script.

           

LVS DR mode - Loopback interface creation in real servers

  

Linux servers

           -- create a loopback adaptor with the relevant VIP

        vi /etc/sysconfig/network-scripts/ifcfg-lo:vip1
        enter relevant values as below;

        DEVICE=lo:vip1
        IPADDR=10.1.31.25
        NETMASK=255.255.255.255
        ONBOOT=yes   
        NAME=loopback-lvs-vip
   

          -- disable ARP responses on loopback

               
        vi /etc/sysctl.conf
        append below lines to the file

        net.ipv4.conf.lo.arp_ignore = 1
        net.ipv4.conf.lo.arp_announce = 2
        net.ipv4.conf.all.arp_ignore = 1
        net.ipv4.conf.all.arp_announce = 2

         -- update sysctl values

        sysctl –p

         -- start the loopback adaptor

        ifup lo:vip1

    Windows servers

            -- add a new loopback adaptor via add remove hardware wizard

               
               
         -- first disable the newly created adapter and configure the VIP with the
            netmask 255.255.0.0 temporary. We need to change the netmask to
            255.255.255.255 which can’t be done via interface properties. We need 
            to  update the registry for this.

        

 

          

            -- configure the relevant VIP on loopback interface

                   Now before enabling the loopback device, go to registry and modify
                   the subnet mask.

              

                     Change the highlighted SubnetMask value to 255.255.255.255
                     Close the registry and enable the loopback interface.


                

Tomcat + Redis high availability with Keepalived

Physical setup view :

For above setup we will use IPs as below;

Server 1 : 10.1.1.2

Server 2 : 10.1.1.3

VIP        : 10.1.1.4

• Nginx stays at the top and it load balances the connections to tomcats using private IPs (10.1.1.2, 10.1.1.3).
• VIP configured using Keepalived service and it runs in both servers.
• Tomcat runs in both servers. In Tomcat, Redis host IP configured to be the VIP (10.1.1.4).

Initially, VIP will be owned by the server which starts first or the server where keepalived service starts first. We will call this the master server.

1. When the master goes down, VIP will be taken over by the secondary server. When this happens, we use keepalived's notify script to make the redis as master and the failed server's redis will become the slave.
2. If Nginx service or the Redis service goes down, then also the VIP will be moved to the secondary server. For this, we use keepalived's tracking script to float the VIP based on service status.
3. Redis service init script also changed to check the VIP status at startup. It will check whether the specific server owns the VIP. If it has the VIP, Redis will change it's role to master.

Configuration of each service as a summary

Tomcat session replication with redis :

• To enable tomcat-redis session replication, you need to add redis session manager jar files to tomcat’s lib directory.

https://github.com/jcoleman/tomcat-redis-session-manager/downloads

Note : copy “tomcat-redis-session-manager-1.1.jar” if java6 is used. Copy “tomcat-redis-session-manager-1.2-tomcat-7-java-7.jar” if java7 is used. Other two libs are same for both versions.

make sure to add commons-pool-1.6.jar and jedis-2.0.0.jar also to the lib directory.

https://commons.apache.org/proper/commons-pool/download_pool.cgi

http://mvnrepository.com/artifact/redis.clients/jedis/2.0.0

• Edit “/$CATALINA_HOME/conf/context.xml” and add following under context

<Valve className="com.radiadesign.catalina.session.RedisSessionHandlerValve" />

<Manager className="com.radiadesign.catalina.session.RedisSessionManager"

 host="10.1.1.4"

 port="6379"

 database="0"

 maxInactiveInterval="60" />

* make sure redis listens on the server IP instead of lookback IP.

Sample nginx upstream configuration :

upstream tomcat {

        server 10.1.1.2:8080;

        server 10.1.1.3:8080;

        check interval=3000 rise=2 fall=5 timeout=3000;

        keepalive 16;

                }

* Above will monitor the service port only. You can use an URI also to check the service status.

Keepalived configuration :

vrrp_script chk_myscript {

  script       "/etc/keepalived/scripts/track.sh"

  interval 5

  fall 2

  rise 2

}

vrrp_instance vip_testweb {

    state BACKUP

    interface eth0

    virtual_router_id 61

    priority 101

    advert_int 1

    nopreempt

    notify "/etc/keepalived/scripts/notify.sh"

        track_script {

                chk_myscript

                }

        virtual_ipaddress {

                10.1.1.4/24

                }

}

--------------------------------------------------------------------------------------------------------------------------

track.sh -- this would trigger the VIP float based on nginx and redis service status.

#!/bin/bash

#### This script will check the redis by adding a key with current time as value. And will check the nginx service status also.

#### If both are running prpoerly script should return "0" as the result code.

ngxstat=`/usr/bin/pgrep nginx | wc -l`

time=`date +%T`

redstat=`/usr/bin/redis-cli set health_check $time`

cmdstat=`echo $?`

if [ $cmdstat -ne 0 ];

then

redstat=err:$cmdstat

else

redstat="OK"

fi

echo  redstat $redstat

echo ngxstat $ngxstat

if [ $redstat == "OK" ]  && [ $ngxstat != 0 ];

then

echo services OK

exit 0

else

echo services ERROR

exit 123

fi

------------------------------------------------------------------------------------------------------------------------------------------------------

notify.sh -- this will trigger the script with the keepalived status (Master/Backup/Fault) as parameter. Based on keepalived VRRP instance status, redis role will be changed. In the script put the proper peer IP and port. 

                 

                 

#!/bin/bash

TYPE=$1

NAME=$2

STATE=$3

PEER_IP="10.1.1.3"

PEER_PORT="6379"

echo $TYPE $NAME $STATE

case $STATE in

        "MASTER") /usr/bin/redis-cli slaveof no one

                  exit 0

                  ;;

        "BACKUP") /usr/bin/redis-cli slaveof $PEER_IP $PEER_PORT

                  exit 0

                  ;;

        "FAULT") /usr/bin/redis-cli slaveof no one

                  exit 0

                  ;;

        *)       echo "unknown state"

                 exit 1

                  ;;

esac

-------------------------------------------------------------------------------------------------------------------------

Redis init script modification :

Add below lines to redis init script. This way, when redis starts it will decide to become master or slave. In the script put the proper peer IP and port.

VIP="10.1.1.4"

PEER_IP="10.1.1.3"

PEER_PORT="6379"

vipcheck() {

       vip=`ip add sh | grep $VIP | wc -l`

        if [ $vip -eq 1 ]; then

        echo

        echo $"VRRP is MASTER"

        /usr/bin/redis-cli slaveof no one

        else

        echo

        echo $"VRRP is BACKUP"

        /usr/bin/redis-cli slaveof $PEER_IP $PEER_PORT

        fi

}

• Call the vipcheck() function at start()

start() {

    [ -x $redis ] || exit 5

    [ -f $REDIS_CONF_FILE ] || exit 6

    echo -n $"Starting $prog: "

    daemon $redis $REDIS_CONF_FILE

    retval=$?

    vipcheck

    echo

    [ $retval -eq 0 ] && touch $lockfile

    return $retval

}

After applying above configuration in both servers, you can test the failover by stopping keepalived service or nginx service or even redis service.

 All keepalived events will be logged at "/var/log/messages"

VIP can be seen by typing the command, "ip add sh"

Even though configurations and steps are not in detail, I hope this would help you to understand how I have achieved redis level high availability.