Huge Message Processing with WSO2 ESB Smooks Mediator

Smooks is a powerful framework for processing, manipulating and transforming XML and non XML data. WSO2 ESB supports executing Smooks features through 'Smooks Mediator'. 

One of the main features introduced in Smooks v1.0 is the ability to process huge messages (Gbs in size) [1]. Now with the WSO2 ESB 4.5.0 release (and later), Huge Message Processing feature is supported through Smooks Mediator!

Smooks supports three types of processing for huge messages which are,

1. one-to-one transformation

2. splitting and routing

3. persistence

This post shows how to process large input messages using Splitting and routing approach. 

Step 1: Create sample Huge Input file. 

This post assumes the input message is in the following format.

    

        Joe
    
    
        
            Pen
            8.80
        
        
            Book
            8.80
        
        
            Bottle
            8.80
        
        
            Note Book
            8.80
        
    

You can write a simple java program to generate a file with large number of entries. 

FileWriter fw = new FileWriter("input-message.txt");
PrintWriter pw = new PrintWriter(fw);
       
        /*XML */
        pw.print("\n 
\n   Joe\n 
\n \n");
        for(int i=0;i<=2000000;i++){
        pw.print("\t\n\t\tPen\n\t\t8.80\n\t\n");
                 
        }
        pw.write(" \n");

Step 2: Smooks Configuration 

Let's write the Smooks configuration to split and route the above message. When we are processing huge messages with Smooks, we should make sure to use the SAX filter.

The basic steps of this Smooks process are, 

1. Java Binding - Bind the input message to java beans

2. Templating - Apply a template which represents split message on input message elements

3. Routing - Route each split message

So for doing each of the above steps we need to use the relevant Smooks cartridges.

1. Java Binding

The Smooks JavaBean Cartridge allows you to create and populate Java objects from your message data [2]. We can map input message elements to real java objects by writing bean classes or to virtual objects which are Maps and Lists. Here we will be binding to virtual objects. In that way we can build complete object model without writing our own business classes.

Let's assume that we are going to split the input message such that one split message contains a single order item information (item-id, product, quantity, price) with the order information (order-id, customer-id, customer-name).

So we can define two beans in our smooks configuration;  order and orderItem.

2. Templating

Smooks Templating allows fragment-level templating using different templating solutions. Smooks supported templating technologies are FreeMarker and XSL templating. In here we are going to use FreeMarker templating solution.

Configuring FreeMarker templates in Smooks is done through the http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd configuration namespace. We can refer the message content in template definition through the java beans which we have defined in the above step.

There are two methods of FreeMarker template definitions. They are In line and External Template Reference. In this example let's use in-line templating.

First we need to decide the format of a single split message. Since we are going to split the input message such that one split message contains a single order-item information (item-id, product, quantity, price) with the order information (order-id, customer-id, customer-name), it will look as follows.

The java object model we had populated above is been used in template definition.

         
 
    
           ${order.customerName}
           ${order.customerNumber?c}
    
       

           ${order.orderItem.product}
           ${order.orderItem.quantity}
           ${order.orderItem.price}
       
 

Let's add the templating configuration to our smooks configuration.

Please note that using <ftl:outputto>, you can direct Smooks to write the templating result directly to an OutputStreamResource.

 3. Routing

So far we have defined the bean model of the message, then defined the template of a single split message. Now we have to continue smooks configuration to route each message fragment to an endpoint. These endpoints can be file, database or JMS endpoints.

In this sample let's route the message fragments to file locations. As in the above step we defined the outputTo element to write to orderItemSplitStream resource, lets add outputStream named orderItemSplitStream to our smooks configuration.

We need to define following attributes when defining the outputStream

fileNamePattern

Can be composed by referring java object model we created. The composing name should be a unique name for each message fragment.

destinationDirectoryPattern

Destination where files should be created.

highWaterMark

Maximum number of files that can be created in the directory. This should be increased according to the input message size.

    
 

 
 

     
     
     
     
     

 

     
     
     
     

      


  
  
  
       
       
  




     order-${order.orderId}-${order.orderItem.itemId}.xml
     
          /home/lakmali/dev/test/smooks/orders
     
     

 

Step 3: Process with WSO2 ESB Smooks Mediator

Now we have finished writing the smooks configuration which will split and route an incoming message. So now we need to get this executed against our Huge Message. WSO2 ESB Smooks Mediator is a solution for this which integrates Smooks features with WSO2 ESB.

So our next step is writing a synapse configuration to fetch the file containing the incoming message through VFS transport and  mediate through the Smooks Mediator to get our task done.

Here is the synpase Configuration

   
      
         
            
               
               

            
         
      
      MOVE
      5
      file:///home/lakmali/dev/test/smooks/original
      file:///home/lakmali/dev/test/smooks/in
      file:///home/lakmali/dev/test/smooks/original
      .*\.xml
      application/xml
      MOVE
   
   
   
      
         
         
         
      
      
   
   
      
      
   

Make sure to Change the VFS Transport Configuration Parameters.


transport.vfs.MoveAfterProcess - Move the input file to this location after processing
transport.vfs.FileURI - Input File location
transport.vfs.MoveAfterFailure - Move the input file to this location after a failure

Create a proxy service with the given synpase configuration. There is an available ESB sample with this configuration which you can run by executing the following command.

Go to ESB_HOME/bin
And run
./wso2esb-samples.sh -sn 658

Now drop the sample Huge Input file to transport.vfs.FileURI location.

Now check the destinationDirectoryPattern location where you can find the split file results of the huge file.


[1] http://www.smooks.org/mediawiki/index.php?title=V1.5:Smooks_v1.5_User_Guide#Processing_Huge_Messages_.28GBs.29
[2] http://www.smooks.org/mediawiki/index.php?title=V1.5:Smooks_v1.5_User_Guide#Java_Binding

Google Analytics Tracking for WSO2 API Manager

WSO2 API Manager is a platform for creating, managing, consuming and monitoring APIs. It employs proven SOA best practices to solve a wide range of API management challenges such as API provisioning, API governance, API security and API monitoring.


You can download API Manager latest version from:

http://wso2.com/products/api-manager

and find the online documentation from:

http://docs.wso2.org/wiki/display/AM130/WSO2+API+Manager+Documentation


With WSO2 API Manager 1.3.0 release you can track your API invocations through Google Analytics!

First let me explain you what is google analytics. Google analytics is a service offered by Google which tracks vists to a web site and generates detailed statistics based on those visits.
Now WSO2 API Manager latest 1.3.0 release fascilitates you to integrate with Google Analytics to track run time statistics for API invocations.

Here is the step by step guide to do this.

1. Setup a Google analytics account.

If you already have a google account, Sign-In. Else Create a New Google Account and Sign-In.

Then go to http://www.google.com/analytics/ and Sign-In

Click on the Admin Icon on top right

Then Click Create New Account as highlighted in below image.

Now select 'Web Site' for what would you like to Track and give information about the account as follows. Please note that since we are actually not tracking a site, give any site URL for 'Web Site URL'.

Now Click Get 'Tracking ID' bottom of the page.

Accept the Google Analytics Terms of Service and you will directed to a page with Tracking information.

Now you have sucessfully created your Tracking account and you will need the obtained Tracking-ID for further steps.

2. Enable Google Analytics Tracking in API Manager.

Open API_MANAGER_HOME/repository/conf/api-manager.xml. Locate the and set Enabled to true and enter the Tracking-ID obtained in as follows.

Save the changes to API_MANAGER_HOME/repository/conf/api-manager.xml and restart your API Manager running instance.

Now we have successfully integrated WSO2 API Manager with Google Analytics. Now lets see how we can see the statistics.

3. View Google Analytics statistics.

Real Time Statistics

First create and publish an API. Then through the API store subscribe to that API.

Now go to Google Analytics [http://www.google.com/analytics/] and select the above created 'APISTORESTAT' account from the list of Accounts.

On the appearing page go to 'Home' tab.

Now you will see 'Real-Time' icon in left menu. Click on that and select 'Overview'.

Then invoke/call the above created API through embedded RESTClient or Curl.

Now you will be able to see a hit on 'PageViews' Per second graph and also 'Right Now' users as 1.

Reporting Statistics

Google analytics reporting statistics population takes more than 24hours from the invocation. Therefore to view the statistics on the Dashboard you need to wait!

Here is a sample Dashboard with populated statistics.

Here as you can see there are Widgets with statistics related to Audience, Traffic, Page Content, Visit Duration etc. You can add any widgets for your preference through 'Add Widget'.

How to enable remote access to mysql database server

Enable remote access to mySQL database server

1. Open the MySQL server configuration file my.cnf.

sudo gedit /etc/mysql/my.cnf

2. Change the bind-address value to your MySQL server ip. If the MySQL server IP=10.100.2.200, change it as follows, save and close my.cnf file.

bind-address=10.100.2.200

3. Now Grant access to remote IP.
If the remote IP=10.100.2.180 

Grant permission to a specific database (ex: testbd)

GRANT ALL ON testdb.* TO testuser@'10.100.2.180' IDENTIFIED BY 'test123';

Grant all permissions

GRANT ALL PRIVILEGES ON *.* TO testuser@'10.100.2.180' IDENTIFIED BY 'test123' WITH GRANT OPTION;

4. Open port 3306, if firewalls are enabled. Following is a sample iptables rule to open Linux iptables firewall.

/sbin/iptables -A INPUT -i eth0 -p tcp --destination-port 3306 -j ACCEPT

5. logout of mysql.

6. Connect from remote host

mysql -u root -h 10.100.2.200 -p

Smooks Entity Persistence Frameworks - Hibernate

With the new Smooks Persistence cartridge in Smooks 1.2, we can directly use several entity persistence frameworks from within Smooks (Hibernate, JPA etc). So in this blog post let's try out the Hibernate sample specified in here.

Since smooks have given the basic configurations only, this blog post is going to assist the peolple who need full implementation.

What we are going to do is process the following XML message and store the product information to a bean and persist the order information.

input.xml
------------

    1
    123456
    
        
            11
            2
        
        
            22
            7
        
    

First we have to Specify our Entity classes (Order, OrderLine, Product)

//Order class
package example.entity;

import java.util.ArrayList;
import java.util.List;

import javax.persistence.*;

@Entity
@Table(name="orders")
public class Order {
 
    @Id
    private Integer ordernumber;
 
    @Basic
    private String customerId;
 
    @OneToMany(mappedBy = "order", cascade = CascadeType.ALL)
    private List orderItems = new ArrayList();
 
    public void addOrderLine(OrderLine orderLine) {
        orderItems.add(orderLine);
    }
    
    public Integer getOrdernumber() {
 return ordernumber;
 }

   public void setOrdernumber(Integer ordernumber) {
 this.ordernumber = ordernumber;
 }

   public String getCustomerId() {
 return customerId;
 }

   public void setCustomerId(String customerId) {
 this.customerId = customerId;
 }

   public List getOrderItems() {
 return orderItems;
 }

   public void setOrderItems(List orderItems) {
 this.orderItems = orderItems;
 }

}

//OrderLine class 
package example.entity;

import javax.persistence.*;

@Entity
@Table(name = "orderlines")
public class OrderLine {

 @Id
 @GeneratedValue(strategy = GenerationType.IDENTITY)
 private Integer id;

 @ManyToOne
 @JoinColumn(name = "orderid")
 private Order order;
 
 @Basic
 private Integer quantity;

 @ManyToOne
 @JoinColumn(name = "productid")
 private Product product;

 public Integer getId() {
  return id;
 }

 public void setId(Integer id) {
  this.id = id;
 }

 public Order getOrder() {
  return order;
 }

 public void setOrder(Order order) {
  this.order = order;
 }

 public Integer getQuantity() {
  return quantity;
 }

 public void setQuantity(Integer quantity) {
  this.quantity = quantity;
 }

 public Product getProduct() {
  return product;
 }

 public void setProduct(Product product) {
  this.product = product;
 }

}

//Product Class
package example.entity;

import javax.persistence.*;

@Entity
@Table(name = "products")
public class Product {
 
    @Id
    private Integer id;
   
 @Basic
    private String name;

 public Integer getId() {
  return id;
 }

 public void setId(Integer id) {
  this.id = id;
 }

 public String getName() {
  return name;
 }

 public void setName(String name) {
  this.name = name;
 }

}

Here is the Smooks configuration to bind xml message to java beans and persist the order entity. 
smooks-hbt-config.xml
-----------------------------------

     
    
        
        
        
    
 
     
    
        
        
        
    
 
    
    
        from Product p where p.id = :id
        
            
        
    
 
    
    
     

So now you would wonder where we store the datasource configuration. If we just use Hibernate we can configure datasource information inside hibernate.cfg.xml

  
    jdbc:mysql://localhost:3306/dbname
    uname
    pwd
    com.mysql.jdbc.Driver
    org.hibernate.dialect.MySQLDialect
 
    false
 
    true
    update
 
    
    1
    thread
  
  
  
  
 

The database represented in the connection must have 4 tables.


customers (id int(11), name varchar(30))
products (id int(11), name varchar(30))
orders (ordernumber int(11), customerId varchar(30))
orderlines (id int(11), quantity int(11), orderid int(11), productid int(11))

Insert the following sample data to customers and procucts tables.

insert into customers (id, name) values (123456, 'Devin Snow');
insert into customers (id, name) values (789101, 'Homer Simpson');
insert into customers (id, name) values (999999, 'John Doe');

insert into products  (id, name) values (11, 'Cheese cake');
insert into products  (id, name) values (22, 'Chocolate 300gr');
insert into products  (id, name) values (33, 'Beer');
insert into products  (id, name) values (44, 'Smooks in Action');
insert into products  (id, name) values (55, 'Ultimate guide to Smooks');

Let's execute Smooks now. 

package example.main;

import java.io.File;
import java.io.IOException;

import javax.xml.transform.Source;
import javax.xml.transform.stream.StreamSource;

import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.Transaction;
import org.hibernate.cfg.AnnotationConfiguration;
import org.milyn.Smooks;
import org.milyn.container.ExecutionContext;
import org.milyn.persistence.util.PersistenceUtil;
import org.milyn.scribe.adapter.hibernate.SessionRegister;
import org.milyn.scribe.register.DaoRegister;
import org.xml.sax.SAXException;

public class Main {

 public static void main(String args[]) throws IOException, SAXException {
  Smooks smooks = new Smooks("smooks-hbt-config.xml");

  ExecutionContext executionContext = smooks.createExecutionContext();

  SessionFactory sessionFactory = new AnnotationConfiguration().configure().buildSessionFactory();
   
  Session session = sessionFactory.openSession();
  DaoRegister register = new SessionRegister(session);
   
  // This sets the DAO Register in the executionContext for Smooks //to access it. 
  PersistenceUtil.setDAORegister(executionContext,register);
   
  Transaction transaction = session.beginTransaction();
  
  Source source = new StreamSource(new File("input.xml"));
   
  smooks.filterSource(executionContext, source);
   
  transaction.commit();

 }