Tuesday, December 1, 2015

Configure WSO2 DSS with Cassandra

Here are the steps to setup DSS and Cassandra integration

Setup 1: download and install Cassandra
WSO2 DSS 3.5.0 is based on WSO2 Carbon 4.4.2 and Cassandra version 2.0 is recommended
WSO2 DSS 3.2.2 is based on WSO2 Carbon 4.2.0 and Cassandra version 2.0 is recommended

Step 2: download and install WSO2DSS

Step 3: configure following operations in DSS
Create Key space - UserKS
CREATE KEYSPACE UserKS WITH replication = {'class':'SimpleStrategy', 'replication_factor': 1}
Create column family - User (id, name, country)
CREATE TABLE UserKS.User (id text, name text, country text, PRIMARY KEY (id))
INSERT INTO UserKS.User (id, name, country) values (?,?,?)
UPDATE UserKS.User SET country = ? WHERE id =?

You can try the above command using cqlsh tool resides in cassandra/bin

Option1: Going through the WSO2DSS Management Console Wizard and configure
Option2: Use Developer Studio, create a CAR file and deploy it on WSO2DSS.


Note: DSS generates a .dbs file which contains all the configurations, which you can find out in wso2dss-3.2.2\repository\deployment\server\dataservices location

Step 4: execute Cassandra queries from Soap UI
Get the WSDL from the WSO2DSS Management console, add it to the Soap UI and invoke

Contents of .dbs file
<data name="UserDS" transports="http https local">
    <config id="UserDataSource">
        <property name="cassandraServers">localhost</property>
    <query id="createKeySpace" useConfig="UserDataSource">
        <expression>CREATE KEYSPACE UserKS WITH replication = {'class':'SimpleStrategy', 'replication_factor': 1}</expression>
    <query id="dropKeySpace" useConfig="UserDataSource">
        <expression>DROP KEYSPACE UserKS</expression>
    <query id="createUserTable" useConfig="UserDataSource">
        <expression>CREATE TABLE UserKS.User (id text, name text, country text, PRIMARY KEY (id))</expression>
    <query id="addUser" useConfig="UserDataSource">
        <expression>INSERT INTO UserKS.User (id, name, country) values (?,?,?)</expression>
        <param name="id" ordinal="1" sqlType="STRING"/>
        <param name="name" ordinal="2" sqlType="STRING"/>
        <param name="country" ordinal="3" sqlType="STRING"/>
    <query id="getUsers" useConfig="UserDataSource">
        <expression>SELECT * FROM UserKS.User</expression>
        <result element="Users" rowName="User">
            <element column="id" name="id" xsdType="string"/>
            <element column="name" name="name" xsdType="string"/>
            <element column="country" name="country" xsdType="string"/>
    <query id="updateUser" useConfig="UserDataSource">
        <expression>UPDATE UserKS.User SET country = ? WHERE id =?</expression>
        <param name="country" sqlType="STRING"/>
        <param name="id" sqlType="STRING"/>
    <query id="deleteUser" useConfig="UserDataSource">
        <expression>DELETE FROM UserKS.User WHERE id = ?</expression>
        <param name="id" sqlType="STRING"/>
    <query id="getUserById" useConfig="UserDataSource">
        <expression>SELECT * FROM UserKS.User where id = ?</expression>
        <result element="Users" rowName="User">
            <element column="id" name="id" xsdType="string"/>
            <element column="name" name="name" xsdType="string"/>
            <element column="country" name="country" xsdType="string"/>
        <param name="id" sqlType="STRING"/>
    <operation name="createKeySpace">
        <call-query href="createKeySpace"/>
    <operation name="dropKeySpace">
        <call-query href="dropKeySpace"/>
    <operation name="createUserTable">
        <call-query href="createUserTable"/>
    <operation name="addUser">
        <call-query href="addUser">
            <with-param name="id" query-param="id"/>
            <with-param name="name" query-param="name"/>
            <with-param name="country" query-param="country"/>
    <operation name="getUsers">
        <call-query href="getUsers"/>
    <operation name="updateUser">
        <call-query href="updateUser">
            <with-param name="country" query-param="country"/>
            <with-param name="id" query-param="id"/>
    <operation name="deleteUser">
        <call-query href="deleteUser">
            <with-param name="id" query-param="id"/>
    <operation name="getUserById">
        <call-query href="getUserById">
            <with-param name="id" query-param="id"/>
    <resource method="GET" path="user/get">
        <call-query href="getUsers"/>
    <resource method="GET" path="user/get/{id}">
        <call-query href="getUserById">
            <with-param name="id" query-param="id"/>
    <resource method="PUT" path="user/add/{id}/{name}/{country}">
        <call-query href="addUser">
            <with-param name="id" query-param="id"/>
            <with-param name="name" query-param="name"/>
            <with-param name="country" query-param="country"/>
    <resource method="POST" path="user/update/{country}/{id}">
        <call-query href="updateUser">
            <with-param name="country" query-param="country"/>
            <with-param name="id" query-param="id"/>
    <resource method="DELETE" path="user/delete/{id}">
        <call-query href="deleteUser">
            <with-param name="id" query-param="id"/>

Wednesday, July 1, 2015

ActiveMQ does not store more than 1 message when the message size is Big

When performing some tests on a new proxy service, I observed that in some cases a message store was not accepting more than one message at a time.

Here's a brief description of what I was trying to do and what I observed:

I have a proxy service that takes messages from a queue, perform an xslt translation on it, then put it in a message store. There is a Message processor that picks up the message from the store and calls a remote web service. Once the the web service returns, I enrich the main message with the result and send it to its final destination.

What happened is when messages reached certain size (about 4MB), the web service call was taking about 40 seconds to return results. While the message processor was waiting for the response, nothing else was getting in the message store (see example below).

Example: I have 3 messages in the input queue that need to be processed. (A proxy service listens on this queue) Here's what happens. 

Step 1: The proxy picks up Message_1, translates it and puts it in the message store to be processed by the remote web service call
Step 2: The message processor picks up Message 1 from the store and sends it to the remote web service for processing. At the same time, the proxy picks Message 2, translates it and tries to put it in the message store.
Step 2.5: the message processor is still waiting for the remote service to return. Message 1 seems to be still in the store, Message 2 is in the ESB memory, and Message 3 is not being picked up by the proxy because the proxy still has Message 2.
Step 3: The remote call returns. The message processor handles control of Message 1 to the next step on the ESB (enrich message with response and send to final destination). Message 1 is removed from the message store. Message 2 finally gets in the message store, and Message 3 is picked up by the proxy.
Step 4: Repeat until no more messages exist in the input queue.

In theory, message store is acting like a queue, while message processor processing messages from the message store. So, Message 2 should have been successfully placed in the message store and waited there for Message 1 to finish processing. But for some reason it doesn't.

Here's the sample synapse-config I used with WSO2 ESB 4.8.1 and ActiveMQ 5.4.3 used as the queue.

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
   <proxy name="FrontQueue"
          transports="https http jms"
            <property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
            <property name="OUT_ONLY" value="true"/>
            <store messageStore="MyTest"/>
    <proxy name="FrontProxy" transports="http" startOnLoad="true">
            <address uri="jms:/FrontQueue?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&amp;java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&amp;java.naming.provider.url=tcp://localhost:61616"/>
            <property name="OUT_ONLY" value="true"/>
   <endpoint name="SimpleStockQuoteService">
      <address uri=""/>
    <messageStore class="org.apache.synapse.message.store.impl.jms.JmsStore"
      <parameter name="java.naming.factory.initial">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
      <parameter name="store.jms.password">admin</parameter>
      <parameter name="java.naming.provider.url">tcp://localhost:61616</parameter>
      <parameter name="store.jms.username">admin</parameter>
      <parameter name="store.jms.JMSSpecVersion">1.1</parameter>
   <messageProcessor class="org.apache.synapse.message.processor.impl.forwarder.ScheduledMessageForwardingProcessor"
      <parameter name="client.retry.interval">1000</parameter>
      <parameter name="interval">1000</parameter>
      <parameter name="is.active">true</parameter>

Here my only guess is message size. because this reproduced only for the big messages like 4MB. 

1. When I debug the  MessageStoreMediator, messages passes to the queue without waiting, but in ActiveMQ shows there's only 1 message in the queue at given time. (once the first message finish processing, second message come to the queue)

2. When I setup the scenario with WSO2 MB 2.0.0, all the small and big messages were placed in queue correctly regardless of the message size.

3. When I send big messages in different frequency, I could reproduce the same behavior via JMS Client too.
Here are the results when I send different size of messages with different delays via JMS Client 

1MB 10 messages
with 5000ms delay - all 10 messages queued
with 10000ms delay - all 10 messages queued
with 30000ms delay - all 10 messages queued

2MB 10 messages
with 5000ms delay - only 6 messages queued
with 10000ms delay - only 3 messages queued
with 30000ms delay - only 1 messages queued

4MB 10 messages
with 5000ms delay - only 6 messages queued
with 10000ms delay - only 3 messages queued
with 30000ms delay - only 1 messages queued

So it seems there's no issue with WSO2 ESB message store and this is a limitation of ActiveMQ; it takes message by message to the queue when the message size is big.

Further investigating I noticed the following log message in ActiveMQ log at the time when message which does not placed in queue.

2014-12-08 09:25:05,922 | INFO | Usage Manager Memory Limit (1048576) reached on queue://SAMPLEQUEUE4000. Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. See http://activemq.apache.org/producer-flow-control.html for more info | org.apache.activemq.broker.region.Queue | ActiveMQ Transport: tcp:///

"Producer Flow Control occurs when you hit one of the size limits (memory, disk, etc.) for either the entire broker or a single destination. Once you hit it, producers on that destination (or the whole broker, depending on which limit you hit) are unable to send more messages until enough space frees up to hold the next one. So your one-out-one-in mental model is the right one (though if the messages are of different sizes, then it might not be truly one-for-one). This will continue for as long as the limit continues to be hit because producers are faster than consumers; it's not time-based, and it's not forever, just until consumers start catching up and you're not running into any limits." 

So when I changed either the producerFlowControl=false or memoryLimit to higher value, I could successfully send all big messages to the queue without waiting until the previous messages finish processing.


<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">   

WSO2 ESB http://wso2.com/products/enterprise-service-bus/
WSO2 MB http://wso2.com/products/message-broker/ 

Wednesday, April 29, 2015

Deploy InfoRecoverySample in WSO2 IS

WSO2IS 5.0.0

1. Download the source code from

2. Configure the following in web.xml file
a. Specify the "carbonServerUrl" with the URL of the Identity Server. eg: https://localhost:9443
b. Specify the credentials to access Identity Server, "accessUsername" as admin and "accessPassword" as admin.
c. Specify the trustStore absolute resource path for "trustStorePath", eg: <IS_HOME>

3. Update the following entry in sso.properties file

4. Exclude the slf4j-api jar from pom.xml file.


4. Build the source mvn clean install

5. Copy the InfoRecoverySample.war to the <IS_HOME>/repository/deployment/server/webapps 

6. Configure the email confirmation links.
a. To configure for password reset call back in Identity Server (in <IS_HOME>/repository/conf/email/email-admin-config.xml with email type as type="passwordReset")
according to the sample as follows. This will be the confirmation handler path of the user's confirmation.


<configuration type="passwordReset">
<subject>WSO2 Carbon - Password Reset</subject>
Hi {first-name}
We received a request to change the password on the {user-name} account associated with this e-mail address. If you made this request, please click the link below to securely change your password:


If clicking the link doesn't seem to work, you can copy and paste the link into your browser's address window.
If you did not request to have your {user-name} password reset, simply disregard this email and no changes to your account will be made.
Best Regards,
WSO2 Carbon Team

Please refer the Identity Server documentation for sample email template in https://docs.wso2.com/display/IS500/Recover+with+Notification

b. To configure for account confirmation in Self sign up provide the configuration in email-admin-config.xml
with email type as "accountConfirmation". Following is the call back handler for the sample.

7. Configure the identity-mgt.properties file attributes as follows in the Identity Server (in <IS_HOME>/repository/conf/security/identity-mgt.properties)


8. Edit the axis.xml file with the following configuration. This file is found in the <IS_HOME>/repository/conf/axis2/directory. Uncomment the following in the file and provide the necessary email settings.

<transportSender name="mailto"

    <parameter name="mail.smtp.from">sampleemail@gmail.com</parameter>
    <parameter name="mail.smtp.user">sampleemail</parameter>
    <parameter name="mail.smtp.password">password</parameter>
    <parameter name="mail.smtp.host">smtp.gmail.com</parameter>
    <parameter name="mail.smtp.port">587</parameter>
    <parameter name="mail.smtp.starttls.enable">true</parameter>
    <parameter name="mail.smtp.auth">true</parameter>

9. Now restart the server and register new SP as shown in the image below
Assertion Consumer URL = https://localhost:9443/InfoRecoverySample/home.jsp
Enable Single Logout = https://localhost:9443/InfoRecoverySample/home.jsp

10: Access the sample with https://localhost:9443/InfoRecoverySample


Configure Active Directory as a Secondary User Store


Install and configure Active directory with WSO2 IS 5.0.0

Steps to configure secondary user store:

1. Login as admin/admin
2. Go to Configure -> User Store Management, Add secondary user store, give the information as shown in the image, click update

Refer https://docs.wso2.com/display/IS500/Configuring+Secondary+User+Stores


1. The secondary user store configuration should be found at the following location

2. Create a new user in secondary user store and check that user can login to the management console.
Go to configure -> User and Roles -> Users -> Add New User (ex: aduser)
Select the domain ADSUS and give the user details and Finish
Go to Roles and give the login permission to the Internal/Everyone role.
Note: Internal/Everyone role automatically assign to any user

Now logout and try to login with aduser

Saturday, April 25, 2015

VFS to SFTP cannot access absolute paths

Learn WSO2 ESB VFS Transport

When sending or receiving files with VFS transport to a SFTP folder, the specified path is relative to the user home, not a absolute path.

Eg:  If the URL is vfs:sftp://user:password@ftp.server.com/myPath, the file will be stored at /home/user/myPath not in /myPath

This is how SFTP protocol works. SFTP uses SSH protocol to communicate with the server, so it always log in to the users home directory and then it takes the path relative to the user home directory. That is why this behavior occurs.

If you need to access the different location instead of user home

Solution 1:

This is not supported in WSO2 ESB 4.8.1.

So as a workaroud you can mount /myPath to the /home/user/myPath location and then make sure user have access to the /myPath.

mount --bind /myPath /home/user/myPath

Solution 2:

This will be available in next version WSO2 ESB 4.9.0.

"sftpPathFromRoot=true" needs to be added to the URL if absolute paths are used

<parameter name="transport.vfs.FileURI">vfs:sftp://user:password@ftp.server.com/myPath?sftpPathFromRoot=true</parameter>


Run multiple samples without restarting the WSO2 ESB

WSO2 ESB includes working examples that demonstrate its features and capabilities.

When executing ESB samples, each time we need to restart the ESB server as:
Please note: if a given sample required to change axis2.xml file or add some libraries to the ESB, then need to restart the ESB.

./wso2esb-samples.sh -sn 0

How to run multiple samples without restarting the WSO2 ESB:

Solution 1:

The simplest way is just copy the configuration from the ESB docs and paste it on ESB source view and update: then it will deploy the new synapse config to the ESB_HOME/repository/deployment/server/synapse-configs/default

Solution 2:

Create a CAR file and deploy it to the ESB.

We can use the WSO2 Developer Studio for create CAR files

VFS access SFTP with special character password

Learn WSO2 ESB VFS Transport

When we need to access the FTP server using SFTP, VFS connection-specific URL need to be given as :

<parameter name="transport.vfs.FileURI">vfs:sftp://username:p@ssword@ftp.server.com/filePath?vfs.passive=true</parameter>

When the password contains a special characters (eg: p@ssword), it gives the following error.

2015-03-27 13:06:03,766 [-] [PassThroughMessageProcessor-5] ERROR VFSTransportSender cannot resolve replyFile
org.apache.commons.vfs2.FileSystemException: Invalid absolute URI "sftp://username:***@ftp.server.com/filePath?vfs.passive=true".

Solution 1:

Replace the special characters with the respective hex representation.

<parameter name="transport.vfs.FileURI">vfs:sftp://username:p%40ssword@ftp.server.com/filePath?vfs.passive=true</parameter>

    Char     Hex Code
    -------  --------
    [space]    %20
       "       %22
       #       %23
       $       %24
       %       %25
       &       %26
       '       %27
       (       %28
       )       %29
       *       %2A
       +       %2B
       ,       %2C
       -       %2D
       .       %2E
       /       %2F
       :       %3A
       ;       %3B
       <       %3C
       =       %3D
       >       %3E
       ?       %3F
       @       %40
       \       %5C
       ^       %5E
       `       %60
       |       %7C

Solution 2:

Change the password to plain text password without any special characters :)

Wednesday, April 22, 2015

Send email via WSO2 ESB with several attachments

WSO2 ESB Mail transport allows you to send a email only with 1 attachment. If you want to send number of attachments here's the solution.

Scenario: Here I want to extract some content from the input message and attach to the email as two different attachment.

Solution: You can achieve this requirement using Gmail connector.


  1. Add the Gmail connector to the ESB and enable it.

  1. Setup the scenario: split the input message and send it with an email using two different attachments.
    1. Input message I used
       <subject>This is the subject of the mail</subject>
       <textContent>This is the text content of the mail</textContent>      
       <part1>Content of the Part 1</part1>
       <part2>Content of the Part 2</part2>      

  1. proxy configuration
<proxy xmlns="http://ws.apache.org/ns/synapse"
        <property name="username" expression="//username/text()"/>
        <property name="password" expression="//password/text()"/>
        <property name="subject" expression="//subject/text()"/>
        <property name="toRecipients" expression="//toRecipients/text()"/>
        <property name="textContent" expression="//textContent/text()"/>
        <class name="org.test.mediator.SimpleClassMediator"/>
        <property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
        <log level="custom">
           <property name="login" value="success"/>

  1. Develop a custom class mediator to split the message parts and add it to the synapse MessageContext as attachments. 
          Refer https://docs.wso2.com/display/ESB481/Writing+a+WSO2+ESB+Mediator
          Here is the custom class mediator I wrote for this scenario 

package org.test.mediator;

import org.apache.axiom.om.OMElement;
import org.apache.axiom.om.xpath.AXIOMXPath;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.synapse.MessageContext;
import org.apache.synapse.core.axis2.Axis2MessageContext;
import org.apache.synapse.mediators.AbstractMediator;
import org.jaxen.JaxenException;

import javax.activation.DataHandler;
import javax.mail.util.ByteArrayDataSource;

public class SimpleClassMediator extends AbstractMediator {

  private static final Log log = LogFactory.getLog(SimpleClassMediator.class);

  @Override public boolean mediate(MessageContext messageContext) {
      OMElement bodyContent = messageContext.getEnvelope().getBody().getFirstElement();
      try {
          AXIOMXPath xPathPart1 = new AXIOMXPath("//part1");
          String part1 = ((OMElement) xPathPart1.selectSingleNode(bodyContent)).getText();

          AXIOMXPath xPathPart2 = new AXIOMXPath("//part2");
          String part2 = ((OMElement) xPathPart2.selectSingleNode(bodyContent)).getText();

          org.apache.axis2.context.MessageContext axis2MsgCtx =
              ((Axis2MessageContext) messageContext).getAxis2MessageContext();

          javax.activation.DataHandler handler1 = new DataHandler(new ByteArrayDataSource(part1.getBytes(), "text/plain"));
          axis2MsgCtx.addAttachment("part1.txt", handler1 );

          javax.activation.DataHandler handler2 = new DataHandler(new ByteArrayDataSource(part2.getBytes(), "text/plain"));
          axis2MsgCtx.addAttachment("part2.txt", handler2);

      } catch (JaxenException e) {
          log.error("Error while adding attachments...", e);

      return true;

Now the setup is done, restart the ESB and send the request.
(Versions: ESB 4.8.1, Gmail Connector 1.0.0)

More reading: http://wso2.com/library/1148/