Monday, June 23, 2014

Fronting WSO2 AS worker/manager cluster with HAProxy

This blog describes how to configure HAProxy as a load balancer with WSO2 Application Server cluster

HAProxy 1.5
WSO2 Application Server 5.2.1

Setup WSO2 AS Cluster




This cluster consist of 3 WSO2 Application Server instances, as 3 worker nodes and 1 manager node, where 1 node work as both worker and manager. HAProxy performs load balancing with this cluster by distributing incoming requests to the worker nodes via HTTP/S on port 80/443.

All admin requests can be sent to the manager node directly via HTTPs on port 9444 or through the HAProxy via 443 port depending on how we configure the manager node.

Click here to see how to setup WSO2 Application Server cluster






Setup HAProxy load balancer


* Install HAProxy
$ sudo add-apt-repository ppa:vbernat/haproxy-1.5
$ sudo apt-get update
$ sudo apt-get install haproxy

Need version 1.5 since native SSL support was implemented in 1.5

* Set ENABLED to 1 if you want the init script to start haproxy

$sudo vi /etc/default/haproxy

* Edit the /etc/haproxy/haproxy.cfg file and add the following

# load balancing among the worker nodes - HTTP
frontend ft_wrk
      bind as.wso2.com:80
      default_backend bk_wrk

backend bk_wrk
      balance roundrobin
    server node1 as.wso2.com:9764
      server node2 as.wso2.com:9765
      server node3 as.wso2.com:9766

# load balancing among the worker nodes - HTTPS
# access the management console via HTTPS
frontend https-in
      bind *:443 ssl crt /etc/haproxy/ssl/haproxy.pem
      acl is_mgt hdr_beg(host) -m beg mgt.as.wso2.com
      acl is_wrk hdr_beg(host) -m beg as.wso2.com

      use_backend mgt_as_wso2_com if is_mgt
      use_backend as_wso2_com if is_wrk
      default_backend as_wso2_com

backend as_wso2_com
      balance roundrobin
      server node1 as.wso2.com:9444 check ssl
      server node2 as.wso2.com:9445 check ssl
      server node3 as.wso2.com:9446 check ssl

backend mgt_as_wso2_com
      server server1 mgt.as.wso2.com:9444 check ssl

NOTE: load balancer can receive HTTPS requests via 443 port either to management console https://mgt.as.wso2.com/carbon or worker nodes https://as.wso2.com/
frontend https-in block handles HTTPS requests come to the load balancer via 443 port,
bind *:443 ssl crt /etc/haproxy/ssl/haproxy.pem provide valid certificate to HAProxy.

acl is_mgt hdr_beg(host) -m beg mgt.as.wso2.com
acl is_wrk hdr_beg(host) -m beg as.wso2.com
acl properties filter manager and worker requests

Server verification is enabled by default in HAProxy, so need to specify the ca-file as follows
server node1 as.wso2.com:9444 check ssl ca-file /ca-file/path

To disable the server verifications need to specify ssl verify none as follows or specify ssl-server-verify none in global section
server node1 as.wso2.com:9444 check ssl verify none

* Mapping the host names to the IP
Update the “/etc/hosts” file

<IP-of-worker>    as.wso2.com
<IP-of-manager>    mgt.as.wso2.com


* Restart the HAProxy

$sudo  /etc/init.d/haproxy restart

Fronting WSO2 AS worker/manager cluster with Nginx

This Blog describes how to configure Nginx as a load balancer with WSO2 Application Server cluster

Nginx 1.4.1
WSO2 Application Server 5.2.1

Setup WSO2 AS Cluster




This cluster consist of 3 WSO2 Application Server instances, as 3 worker nodes and 1 manager node, where 1 node work as both worker and manager. Nginx performs load balancing with this cluster by distributing incoming requests to the worker nodes via HTTP/S on port 80/443.

All admin requests can be sent to the manager node directly via HTTPs on port 9444 or through the Nginx via 443 port depending on how we configure the manager node.
Click here to see how to setup WSO2 Application Server cluster


Setup Nginx load balancer



* Install Nginx


$sudo apt-get install nginx


* Locate the http{} block of the /etc/nginx/nginx.conf file and add the following


# load balancing among the worker nodes - HTTP
upstream wso2.as.com {
            server as.wso2.com:9764;
          server as.wso2.com:9765;
          server as.wso2.com:9766;
}

server {
          listen 80;
          server_name as.wso2.com;
          location / {
                      proxy_pass http://wso2.as.com;
          }
}

# load balancing among the worker nodes - HTTPS
# To configure load balancing for HTTPS instead of HTTP, just use “https” as the protocol.
upstream ssl.wso2.as.com {
          server as.wso2.com:9444;
          server as.wso2.com:9445;
          server as.wso2.com:9446;
}

server {
          listen 443;
          server_name as.wso2.com;
          ssl on;
          ssl_certificate /etc/nginx/ssl/server.crt;
          ssl_certificate_key /etc/nginx/ssl/server.key;
          location / {
                      proxy_pass https://ssl.wso2.as.com;
          }
}

# access the management console via HTTPS
server {
          listen 443;
          server_name mgt.as.wso2.com;
          ssl on;
          ssl_certificate /etc/nginx/ssl/server.crt;
          ssl_certificate_key /etc/nginx/ssl/server.key;
          location /carbon {
              proxy_pass https://mgt.as.wso2.com:9444;
          }
}


* Mapping the host names to the IP
Update the “/etc/hosts” file


<IP-of-worker>    as.wso2.com
<IP-of-manager>    mgt.as.wso2.com


* Restart the Nginx


$sudo /etc/init.d/nginx restart

Thursday, June 19, 2014

WSO2 AS worker/manager cluster without WSO2 ELB

This blog describes how to setup WSO2 Application Server worker/manager cluster without using WSO2 Elastic Load Balancer and later we can use this cluster for configure different 3rd party load balances

WSO2 Application Server - 5.2.1

1. Use Cases

Each use case consist of 3 WSO2 AS instances form worker/manager cluster fronting with load balancer.

Use Case 1:
Requests direct to worker node cluster through LB
No access to admin console through LB


 Use Case 2:
Requests direct to worker node cluster through LB.
Allows to access admin console through LB.




Use Case 3:
Requests direct to worker node cluster through LB.
Allows to access admin console through LB.
Additionally manager node also serve requests. (Here manger node work as both worker and manager)






Manager/Worker3
Worker1
Worker2
offset
1
2
3
localMemberPort
4100
4200
4300
http
9764
9765
9766
https
9444
9445
9446


2. Tips to configure 3rd party load balancer

In order to configure the load balancer, need to consider the following
  1. Load balancer ports are http 80 and https 443
  2. Direct the http requests to the worker nodes with load balancing as http://as.wso2.com/<service> via http 80 port
  3. Direct the https requests to the worker nodes with load balancing as https://as.wso2.com/<service> via https 443 port
  4. Access the management console as https://mgt.as.wso2.com/carbon via https 443 port

In WSO2 AS cluster worker nodes server requests on http 9763 and https 9443 ports and can access the management console https 9443 port.



3. Configure the Manager node

1. Unzip the WSO2 AS

2. Setting up the cluster configurations
Edit “<AS_HOME>/repository/conf/axis2/axis2.xml” file as follows

*** Enable clustering for this node:

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.as.domain</parameter>
<parameter name="localMemberHost">mgt.as.wso2.com</parameter>
<parameter name="localMemberPort">4100</parameter>

*** Specify the well known member
Here, the well known member is a worker node

<members>
  <member>
        <hostName>as.wso2.com</hostName>
        <port>4200</port>        
  </member>
</members>

3. Configuring the port offset and host name
Edit “<AS_HOME>/repository/conf/carbon.xml” file as follows

<Offset>1</Offset>
<HostName>as.wso2.com</HostName>
<MgtHostName>mgt.as.wso2.com</MgtHostName>


4. Mapping the host names to the IP
Update the “/etc/hosts” file

127.0.0.1    as.wso2.com
127.0.0.1    mgt.as.wso2.com


5. Allow access the management console only through LB

Configure the HTTP/HTTPS proxy ports to communicate through the load balancer
Edit “<AS_HOME>/repository/conf/tomcat/catalina-server.xml” file as follows

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
              port="9763"
              proxyPort="80"

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
              port="9443"
              proxyPort="443"


4. Configure the worker node


1. Unzip the WSO2 AS

2. Setting up the cluster configurations
Edit “<AS_HOME>/repository/conf/axis2/axis2.xml” file as follows

*** Enable clustering for this node:

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.as.domain</parameter>
<parameter name="localMemberHost">as.wso2.com</parameter>
<parameter name="localMemberPort">4200</parameter>



*** Specify the well known member
Here, the well known member is manager node

<members>
  <member>
        <hostName>mgt.as.wso2.com</hostName>
        <port>4100</port>        
  </member>
</members>



3. Configuring the port offset and host name
Edit “<AS_HOME>/repository/conf/carbon.xml” file as follows

<Offset>2</Offset>
<HostName>as.wso2.com</HostName>


4. Configure the HTTP/HTTPS proxy ports to communicate through the load balancer
Edit “<AS_HOME>/repository/conf/tomcat/catalina-server.xml” file as follows

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
              port="9763"
              proxyPort="80"

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
              port="9443"
              proxyPort="443"


5. Create the worker2 node
Get a copy of worker1 node and change the following in axis2.xml and carbon.xml

<parameter name="localMemberPort">4300</parameter>

<Offset>3</Offset>

5. Testing the cluster


1. Restart the configured load balancer

2. Start the manager node
sudo <AS_HOME>/bin/wso2server.sh

3. Start the worker1 and worker2 nodes
sudo <AS_HOME>/bin/wso2server.sh -DworkerNode=true

4. Check member joined messages in all consoles

5. Access management console https://mgt.as.wso2.com:9444/carbon

6. Access management console through LB https://mgt.as.wso2.com/carbon

7. Testing load distribution - http://as.wso2.com/sample or https://as.wso2.com/sample


WSO2 AS worker/manager cluster with WSO2 ELB

This blog describes how to setup WSO2 Application Server worker/manager cluster with WSO2 Elastic Load Balancer.

WSO2 Elastic Load Balancer - 2.1.0
WSO2 Application Server - 5.2.1

1. Deployment Diagram


2. Configure the load balancer


1. Unzip the WSO2 ELB

2. Setting up the load balancing configuration
Edit “<ELB_HOME>/repository/conf/loadbalancer.conf” file


appserver {
       domains   {          
           wso2.as.domain {
               tenant_range    *;
group_mgt_port 4500;
      worker {
          hosts as.wso2.com;
      }   
mgt {
    hosts mgt.as.wso2.com;
}   
           }
       }
}


3. Setting up the cluster configurations
Edit “<ELB_HOME>/repository/conf/axis2/axis2.xml” file as follows


<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.carbon.lb.domain</parameter>
<parameter name="localMemberHost">127.0.0.1</parameter>
<parameter name="localMemberPort">4000</parameter>

4. Configuring the ELB to listen on default ports
Edit “<ELB_HOME>/repository/conf/axis2/axis2.xml” file as follows


<transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">
     <parameter name="port">80</parameter>
</transportReceiver>
<transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">
       <parameter name="port" locked="false">443</parameter>
</transportReceiver>


5. Mapping the host names to the IP
Update the “/etc/hosts” file

<IP-of-worker>    as.wso2.com
<IP-of-manager>    mgt.as.wso2.com


3. Configure the manager node


1. Unzip the WSO2 AS

2. Setting up the cluster configurations
Edit “<AS_HOME>/repository/conf/axis2/axis2.xml” file as follows


<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.as.domain</parameter>
<parameter name="localMemberHost">mgt.as.wso2.com</parameter>
<parameter name="localMemberPort">4100</parameter>

*** Specify this node belongs to the management sub domain
<parameter name="properties">
<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
          <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
          <property name="subDomain" value="mgt"/>
</parameter>

*** Specify the well known member
<members>
   <member>
         <hostName>127.0.0.1</hostName>
         <port>4500</port>        
   </member>
</members>

3. Configuring the port offset and host name
Edit “<AS_HOME>/repository/conf/carbon.xml” file as follows

<Offset>1</Offset>
<HostName>as.wso2.com</HostName>
<MgtHostName>mgt.as.wso2.org</MgtHostName>


4. Configure the HTTP/HTTPS proxy ports to communicate through the load balancer
Edit “<AS_HOME>/repository/conf/tomcat/catalina-server.xml” file as follows


<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
               port="9763"
               proxyPort="80"

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
               port="9443"
               proxyPort="443"


4. Configure the worker node


1. Unzip the WSO2 AS

2. Setting up the cluster configurations
Edit “<AS_HOME>/repository/conf/axis2/axis2.xml” file as follows


<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.as.domain</parameter>
<parameter name="localMemberHost">as.wso2.com</parameter>
<parameter name="localMemberPort">4200</parameter>
*** Specify this node belongs to the management sub domain
<parameter name="properties">
<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
          <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
          <property name="subDomain" value="worker"/>
</parameter>

*** Specify the well known member
<members>
   <member>
         <hostName>127.0.0.1</hostName>
         <port>4500</port>        
   </member>
</members>

3. Configuring the port offset and host name
Edit “<AS_HOME>/repository/conf/carbon.xml” file as follows

<Offset>2</Offset>
<HostName>as.wso2.com</HostName>

4. Configure the HTTP/HTTPS proxy ports to communicate through the load balancer
Edit “<AS_HOME>/repository/conf/tomcat/catalina-server.xml” file as follows

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
               port="9763"
               proxyPort="80"

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
               port="9443"
               proxyPort="443"


5. Create the worker2 node
Get a copy of worker1 node and change the following

<parameter name="localMemberPort">4300</parameter>
<Offset>3</Offset>


5. Testing the cluster


1. Start the ELB
sudo <ELB_HOME>/bin/wso2server.sh

2. Start the manager node
sudo <AS_HOME>/bin/wso2server.sh

3. Start the worker1 and worker2 nodes
sudo <AS_HOME>/bin/wso2server.sh -DworkerNode=true

4. Check member joined messages in all consoles

5. Access management console https://mgt.as.wso2.com/carbon

6. Test load distribution - create jaggery app to log a message - http://as.wso2.com/sample