H-Sphere Sysadmin Guide

Adding Load Balanced NetApp Servers To H-Sphere
 

Related Docs:  

• Updating H-Sphere On NetApp Servers

In H-Sphere 2.3 and up, it is possible to add load balanced Web and mail clusters to H-Sphere on the basis of NetApp Filer. NetApp Filers are building blocks for open storage networks, allowing companies to simplify, share and scale their storage networking and content delivery infrastructures.

NetApp file server solution requires 4 or more physical servers:

  • NetApp (file) server for managing resources. Web and mail servers can jointly use one NetApp server or have their own NetApp servers - one for web and one for mail.
  • At least two boxes (master and slave) for web/mail servers, further referred to as Web1 (Web master), Web2 (Web slave), Mail1 (mail master), Mail2 (mail slave);
  • Load Balancer server for storage and load balancing across the web/mail servers. For example, Load Balancer directs traffic to another server if the first one is currently overloaded.

To create Web/mail load balanced clusters integrated into H-Sphere:

  1. Prepare the System to Work With NetApp Filer
  2. Install and Configure NetApp Server
  3. Install H-Sphere on Web/Mail Servers
  4. Configure NetApp Web Servers
    - Dedicated IPs On NetApp Web Servers
  5. Configure NetApp Mail Servers
  6. Configure CP Server To Implement NetApp Support
  7. Add Web/Mail Servers to LoadBalanced H-Sphere Clusters

 

Step 1. Prepare the System to Work With NetApp Filer

  1. Grant rsh access to the NetApp server from the CP box to root and cpanel user.
  2. Grant nfs access to the /etc directory for the CP box in rw mode.
  3. Grant nfs access to the home directory on the storage partition (/vol/vol0/home) for the CP box in rw mode with root privileges (-access=192.168.0.9:192.168.0.10, root=192.168.0.8:192.168.0.9:192.168.0.10).
 

Step 2. Install and Configure NetApp Server

Buy NetApp Filer directly from www.netapp.com. The instructions about installing and configuring the filer itself can be found in the NetApp Documentation.

Do the following steps to configure your NetApp server (read the NetApp Manual for commands):

  1. Telnet to the NetApp server:

    telnet <NetApp_IP>

    Here, <NetApp_IP> is the NetApp server IP.
  2. Get the list of NetApp partitions with the qtree command:

    qtree

  3. To enable disk quota management, export the /etc directory on the NetApp server and allow to mount it only from the CP box:

    exportfs -o access=<CP_IP>,root=<CP_IP>,rw=<CP_IP> /etc

    Here, <CP_IP> is the CP server IP.
  4. To enable user disk space management on the web/mail servers, export the user storage directory on the NetApp server allow to mount it from the physical web/mail boxes:

    exportfs -o access=<Web/Mail1_IP>:<Web/Mail2_IP>,root=<Web/Mail1_IP>:<Web/Mail2_IP>,rw=<Web/Mail1_IP>:<Web/Mail2_IP> /vol/hsphere/home

    Here, <Web/Mail1_IP>, <Web/Mail2_IP> are web/mail server IPs, /vol/hspere/home is the user storage directory.
  5. Exit telnet session on the NetApps server.
 

Step 3. Install H-Sphere on Web/Mail Servers

Make sure to install on these servers the same version of H-Sphere as is running on your cp server.

  1. Download current installer
  2. Collect your current H-Sphere configuration:
    # ./import
  3. Launch the configuration wizard to enter installation data:
    # ./configure

    In the configuration menu:

    1. Add a master/slave server with the 'a' command. You will be prompted to add the server's IP and mask. After that, it returns to the main configuration menu and you will see the server's IP in the list of physical servers;
    2. Choose the newly added server by typing in its number in the list and pressing Enter. You will see the list of services that can be enabled/disabled for this server.
    3. To enable a service for the newly added server, type in the number that corresponds to WEB/MAIL service.
    4. When you type '-' and return to the main menu, you will see the service enabled in front of the new server's IP in the list of your servers, for example:

      |-[2] 1.2.3.4 [Web]

    5. Repeat steps 1-4 above to add a slave web server in the same manner.
    6. Save configuration and exit with the 's' and 'q' commands from the main menu.
  4. Upload install scripts to the new servers and configure SSH keys to access them:
    make add-server-<ID>
    where <ID> is the number of the new server you have saved.
  5. Login to the new server as root, then install and configure the new server:
    cd /hsphere/install
    make install
 

Step 4. Configure Master and Slave Web Servers

  1. On each web box, mount the NetApp storage partition to the /mnt/NAS directory:

    mkdir /mnt/NAS
    mount -t nfs <NetApp_IP>:/vol/hsphere/home /mnt/NAS

  2. Copy the following directories to the mountpoint directory on the NetApp server:

    cp -prf /usr/local/frontpage /mnt/NAS/linuxweb1-001/
    cp -prf /hsphere/local/config/httpd/ssl.shared /mnt/NAS/linuxweb1-001/

    On the master (Web1) web box:

    cp -prf /hsphere/* /mnt/NAS/linuxweb1-001/

    On the the slave (Web2) web box:

    cp -prf /hsphere/* /mnt/NAS/linuxweb2-001/

  3. On both Web servers, create the /hsphere and /hsphere2 directories if you don't have them:

    mkdir /hsphere
    mkdir /hsphere2

  4. On the master web server, /hsphere directory should point to the previously copied linuxweb1-001 master web server directory on the NetApp server /hsphere2 to the slave server directory (linuxweb2-001). Similarly, on the slave web server, /hsphere should point to the slave web server directory, /hsphere2 to the master web server directory.

    Thus, the two corresponding mountpoints for the /hsphere and /hsphere2 directories should be added in the /etc/fstab file on both web servers.

    For the master web server there should be the following lines:

    <NetApp_IP>:/vol/hsphere/home/linuxweb1-001 /hsphere  nfs  defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/hsphere/home/linuxweb2-001 /hsphere2 nfs  defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/hsphere/home/linuxweb1-001/frontpage /usr/local/frontpage nfs  defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/hsphere/home/linuxweb1-001/ssl.shared /hsphere/local/config/httpd/ssl.shared nfs  defaults,nfsvers=3  0 0

  5. For the slave server, the /etc/fstab file should also contain mountpoints to other directories previously copied to the NetApp server.

    <NetApp_IP>:/vol/hsphere/home/linuxweb2-001 /hsphere nfs defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/home/linuxweb1-001 /hsphere2 nfs defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/hsphere/home/linuxweb1-001/local/home /hsphere/local/home nfs defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/hsphere/home/linuxweb1-001/local/var/statistic /hsphere/local/var/statistic nfs defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/hsphere/home/linuxweb1-001/local/var/httpd/logs /hsphere/local/var/httpd/logs nfs defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/hsphere/home/linuxweb1-001/frontpage /usr/local/frontpage nfs defaults,nfsvers=3  0 0
<NetApp_IP>:/vol/hsphere/home/linuxweb1-001/ssl.shared /hsphere/local/config/httpd/ssl.shared nfs defaults,nfsvers=3  0 0

  6. Check if the crontab contains scripts for synchronizing information between the master server Web1 and the slave server Web2.

    Important: Starting from H-Sphere 2.4.1, synchronization scripts are installed to the /hsphere/shared/scrips/load-ballancing directory on both the master and slave servers. You should copy them to the /hsphere/shared/scrips/cron directory where your Web server cron scripts are stored.

    a) Master web server: Crontab on the master web server should contain the following lines:

    1 */4 * * * /usr/sbin/ntpdate ntps1-{0,1,2}.uni-erlangen.de # time syncronization script
    0 1 * * * nice -15 /hsphere/shared/scripts/cron/cron_analyze.pl
    0 2 * * * nice -15 /hsphere/shared/scripts/cron/cron_rotate.pl
    0 3 * * * nice -15 /hsphere/shared/scripts/cron/ftp_anlz.pl
    0 4 * * * nice -15 /hsphere/shared/scripts/cron/ftp_anlz_user.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/apache-confsynch.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/apache-restart.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-confsynch.pl
    */3 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-restart.pl
    0 5 * * * nice -15 /hsphere/shared/scripts/cron/db_usage.pl

    • /hsphere/shared/scripts/cron/apache-confsynch.pl - synchronize master and slave NetApp Web servers
    • /hsphere/shared/scripts/cron/ftp-confsynch.pl - synchronize master and slave NetApp FTP servers

    7. Note: For correct synchronization between servers, it is essential to have time sychronization script in crontab on both servers!

    Crontab on the master server should also contain the following scripts:

    On RedHat Linux:

    */1 * * * * rsync -e ssh /etc/passwd 10.10.2.12:/etc/passwd
    */1 * * * * rsync -e ssh /etc/shadow 10.10.2.12:/etc/shadow
    */1 * * * * rsync -e ssh /etc/group 10.10.2.12:/etc/group

    On FreeBSD:

    */1 * * * * rsync -e ssh /etc/passwd 209.132.97.68:/etc/passwd
    */1 * * * * rsync -e ssh /etc/group 209.132.97.68:/etc/group
    */1 * * * * rsync -e ssh /etc/master.passwd 209.132.97.68:/etc/master.passwd
    */1 * * * * rsync -e ssh /etc/pwd.db 209.132.97.68:/etc/pwd.db
    */1 * * * * rsync -e ssh /etc/spwd.db 209.132.97.68:/etc/spwd.db

    b) Slave web server: Crontab on the master and the slave server should contain the following lines:

    1 */4 * * * /usr/sbin/ntpdate ntps1-{0,1,2}.uni-erlangen.de
    0 1 * * * nice -15 /hsphere/shared/scripts/cron/cron_analyze.pl
    0 2 * * * nice -15 /hsphere/shared/scripts/cron/cron_rotate.pl
    0 3 * * * nice -15 /hsphere/shared/scripts/cron/ftp_anlz.pl
    0 4 * * * nice -15 /hsphere/shared/scripts/cron/ftp_anlz_user.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/apache-need-restart.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/apache-restart.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-need-restart.pl
    */3 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-restart.pl
    0 5 * * * nice -15 /hsphere/shared/scripts/cron/db_usage.pl

    • /hsphere/shared/scripts/cron/apache-need-restart.pl - control Web server restart after synchronizing master and slave NetApp Web servers
    • /hsphere/shared/scripts/cron/ftp-need-restart.pl - control FTP server restart after synchronizing master and slave NetApp FTP servers

    • On the master Web server, create IP map file:

      vi /hsphere2/local/config/httpd/sites/map_table.txt

      Insert one line of the following format:

      <Master_IP>|<Slave_IP>

    • To synchronize the NetApp Web servers on dedicated IPs, after you have configured the IP map file in the previous step, add the corresponding H-Sphere scripts to the crontab on the master Web server and the slave Web server.

      For the master Web server:

      */4 * * * * /hsphere/shared/scripts/cron/master-ipsynch.pl

      For the slave Web server:

      */4 * * * * /hsphere/shared/scripts/cron/slave-ipupdate.pl

    • On the master Web server, configure separate pid and log files for the master and the slave Web servers.

      1. Pid files (httpd.pid) for the master and the slave will be located in separate directories, logs1 for the master, and logs2 for the slave. Create these directories on the master server:

         mkdir /hsphere/local/var/httpd/logs/logs1
 mkdir /hsphere/local/var/httpd/logs/logs2

      2. Run the following commands to reconfigure Apache to write to separate log and pid files for the master and slave servers.

        a) On the master Web server:

        For Linux:

        perl -pi -e
's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs1/httpd.pid:g' /etc/rc.d/init.d/httpd
/hsphere/shared/apache/bin/apachectl /hsphere/local/config/httpd/httpd.conf
perl -pi -e 's:/hsphere/local/var/httpd/logs/access_log
\s:/hsphere/local/var/httpd/logs/access_log.1:g' /hsphere/local/config/httpd/httpd.conf
perl -pi -e 's:/hsphere/local/var/httpd/logs/error_log
\s:/hsphere/local/var/httpd/logs/error_log.1:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
perl -pi -e 's:/hsphere/local/var/httpd/logs/ssl_engine_log
\s:/hsphere/local/var/httpd/logs/ssl_engine_log.1:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache

        For FreeBSD:

        perl -pi -e
's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs1/httpd.pid:g' /hsphere/shared/apache/bin/apachectl
/hsphere/local/config/httpd/httpd.conf
perl -pi -e 's:/hsphere/local/var/httpd/logs/access_log
\s:/hsphere/local/var/httpd/logs/access_log.1:
g'/hsphere/local/config/httpd/httpd.conf
perl -pi -e 's:/hsphere/local/var/httpd/logs/error_log
\s:/hsphere/local/var/httpd/logs/error_log.1:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
perl -pi -e 's:/hsphere/local/var/httpd/logs/ssl_engine_log
\s:/hsphere/local/var/httpd/logs/ssl_engine_log.1:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache

        b) On the slave Web server:

        For Linux:

        perl -pi -e
's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs2/httpd.pid:g' /etc/rc.d/init.d/httpd
/hsphere/shared/apache/bin/apachectl /hsphere/local/config/httpd/httpd.conf
perl -pi -e 's:/hsphere/local/var/httpd/logs/access_log
\s:/hsphere/local/var/httpd/logs/access_log.2:g' /hsphere/local/config/httpd/httpd.conf 
perl -pi -e 's:/hsphere/local/var/httpd/logs/error_log
\s:/hsphere/local/var/httpd/logs/error_log.2:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
perl -pi -e 's:/hsphere/local/var/httpd/logs/ssl_engine_log
\s:/hsphere/local/var/httpd/logs/ssl_engine_log.2:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache

        For FreeBSD:

        perl -pi -e
's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs2/httpd.pid:g' /hsphere/shared/apache/bin/apachectl
/hsphere/local/config/httpd/httpd.conf
perl -pi -e 's:/hsphere/local/var/httpd/logs/access_log
\s:/hsphere/local/var/httpd/logs/access_log.2:g' /hsphere/local/config/httpd/httpd.conf
perl -pi -e 's:/hsphere/local/var/httpd/logs/error_log
\s:/hsphere/local/var/httpd/logs/error_log.2:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
perl -pi -e 's:/hsphere/local/var/httpd/logs/ssl_engine_log 
\s:/hsphere/local/var/httpd/logs/ssl_engine_log.2:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache

      3. Restart Apache on both boxes and check if pid and log files exist on the master server:

        /hsphere/local/var/httpd/logs/logs1/httpd.pid - master pid file
        /hsphere/local/var/httpd/logs/logs2/httpd.pid - slave pid file
        /hsphere/local/var/httpd/logs/access_log.1 - master access log file
        /hsphere/local/var/httpd/logs/access_log.2 - slave access log file
        /hsphere/local/var/httpd/logs/error_log.1 - master error log file
        /hsphere/local/var/httpd/logs/error_log.2 - slave error log file
        /hsphere/local/var/httpd/logs/ssl_engine_log.1 - master ssl engine log file
        /hsphere/local/var/httpd/logs/ssl_engine_log.2 - slave ssl engine log file

 

Step 5. Configure Master and Slave Mail Servers

  1. On each mail box, mount the mail storage partition to the /mnt/NAS directory:

    mkdir /mnt/NAS
    mount -t nfs <NetApp_IP>:/vol/hsphere/mail /mnt/NAS

  2. Copy the following directories to the mountpoint directory on the NetApp server:

    cp -prv /hsphere/local/var/vpopmail /mnt/NAS/
    cp -prv /var/qmail/control /mnt/NAS/
    cp -prv /var/qmail/users /mnt/NAS/

  3. Configure /etc/fstab for mail servers:

    vi /etc/fstab

    On both the master and the slave mail servers /etc/fstab should contain the following lines: 
    <NetApp_IP>:/vol/hsphere/mail/vpopmail   /hsphere/local/var/vpopmail  nfs  rw  0 0
<NetApp_IP>:/vol/hsphere/mail/control   /var/qmail/control  nfs  rw  0 0
<NetApp_IP>:/vol/hsphere/mail/users   /var/qmail/users  nfs  rw  0 0
  4. On the master mail server, crontab contains all necessary H-Sphere scripts.
  5. On the slave mail server, crontab SHOULD NOT contain any H-Sphere scripts!
 

Step 6. Configure CP Server To Implement NetApp Support

On the H-Sphere CP server:

  1. Install suidperl
    - for Linux, it could be installed, for example, like this:

    rpm -ivh perl-suidperl-5.6.1-34.99.6.i386.rpm

    - for FreeBSD, it is already installed into the system.
  2. Set permissions for the script fileserver-quota.pl:

    chmod +s /hsphere/shared/scripts/fileserver-quota.pl

  3. Set SUPPORT_NET_APP property in the hsphere.properties file:

    SUPPORT_NET_APP=TRUE

  4. Mount /etc/ directory on the netapp fileserver to the /hsphere/YOUR_NETAPP_SERVER_IP/etc directory on CP server.

    $NA="YOUR_FILER_IP", (e.g: $NA="192.168.1.1")

    mkdir /hsphere/$NA/etc
    mount $NA:/etc /hsphere/$NA/etc

  5. In logical Web server properties, set File Server and File Path, where:
    - File Server is a filer name or IP-address and also a qtree-name;
    - File Path is a path to the mounted filer storage directory.
    6. For example:

    file_server=YOUR_FILER_IP:/YOUR_QTREE (e.g: 192.168.1.1:/vol/vol0)
    file_server_path=YOUR_FILESERVER_PATH (e.g: file_server_path=/web0.msp0/local)

  6. Check the /hsphere/<NetApp_IP>/etc/quotas. There should be a line like this:

    * user@/vol/hsphere/home 20000M 160K

  7. Check that rsh and quota are enabled:

    rsh <NetApp_IP> quota report

  8. Add physical servers (master and slave) with their IPs to the system.
  9. Add master logical servers to the newly added physical servers (there's no need to add slave logical servers).
 

Step 7. Add Web/Mail Servers to LoadBalanced H-Sphere Clusters

For web clusters, each master server can have only one slave server. This means that when adding web servers to H-Sphere LB cluster, you have to set them up in pairs. Multiple web servers are possible in H-Sphere 2.5 and up.

For mail clusters, multiple slave servers are possible.

To add web/mail servers to load balanced H-Sphere clusters:

  1. Install H-Sphere on Web/Mail Servers. Make sure to install on these servers the same version of H-Sphere as is running on your cp server.
  2. Configure master and slave web servers.
  3. Configure master and slave mail servers.
  4. Add physical servers (master and slave) with their IPs to the system via the interface.
  5. Add master logical servers to the newly added physical servers (there's no need to add slave logical servers). All management is run on masters only but there are scripts to replicate changes for slaves.

    6.  

Related Docs:  

• Updating H-Sphere On NetApp Servers



© Copyright 1998-2006. Positive Software Corporation.
All rights reserved.