Difference between revisions of "Stardust/Knowledge Base/Integration/Camel/Json File To Structured Data"

Latest revision as of 04:07, 18 February 2014

Requirements

1. Need to write a process which should be triggered when a file arrives at a particular folder.
2. It should pick the file content (which happens to contain JSON Strings).
3. Need to convert the JSON string into a structured data and present it to users for editing.
4. Once edited, need to convert the structured data back to JSON string.

Steps (Reading Single Record from JSON File):

1. Create a RAD project and create a model like shown below.
2. Create a class as shown in the DataToJSON class Code section.
3. Download jackson-core-asl-1.9.10.jar and jackson-mapper-asl-1.9.10.jar and put it in WEB-INF/lib of the RAD project.
4. Build the project, deploy the model.
5. Place a file with itxt as extension in the designated folder which is being monitored by Camel Trigger (in this case its C:\temp). You can define the trigger as per your need and can change the file extension to monitor of folder to monitor.

File Content is:

{"Name":"Tanmoy Roy","EmpID":"1002345","DoJ":1376332200000,"Salary":54433.0,
"Phones":[{"Home":"11111111","Mobile":"111111111","Office":"33333333333"},
{"Home":"2222222","Mobile":"2222222","Office":"34444444"}],
"dependents":[{"Name":"T1","Relation":"TR1","Age":21},
{"Name":"T2","Relation":"TR2","Age":43}]}

To achieve this we will be using the Camel Trigger to monitor the arrival of the file. We will also use Camel Application Type to route the request to the required beans (in this case dataToJSON).

The aim to use Camel Application Type is to provide flexibility to the solution (currently we are routing to beans, but using Camel Aplication Type we can route to other supported endpoints as per need).

Define Receive Incoming File Camel Trigger as shown in the image below.

Define 2 Camel Producer Application (FileTOObject and convertToJSON in this case) using that we will convert JSON to Object and back. Associate these to activities Convert File into Strcutured Data and Convert Data To JSON.

From Applications, choose Camel Producer Application and in its General Tab define the bean like:

<bean id="dataToJSON" class="com.sungard.cm.invo.ipp.training.DataToJSON"/>

and it is Producer Route Tab define route like (for Convert File into Strcutured Data)

<to uri="bean:dataToJSON?method=readBack"/>

This will call the readBack() method of the bean dataToJSON, which is reading from JSON string and converting it into an Strcutured data type (Map) called Employee. and for (Convert Data To JSON)

<to uri="bean:dataToJSON?method=enrich"/>

This will call the enrich() method of the bean dataToJSON, which is reading from Strcutured Data called Employee and converting it to JSON string.

package com.sungard.cm.invo.ipp.training;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.ListIterator;
import java.util.Map;
 
import org.codehaus.jackson.JsonGenerationException;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.type.TypeReference;
 
import com.google.gson.Gson;
 
//import com.google.gson.Gson;
public class DataToJSON<E> {
 
	public String enrich(Map data) {
		System.out.println("Incoming Data value: "+data);
		//Gson gson = new Gson();
		ObjectMapper objectMapper = new ObjectMapper();
 
		String returnData = null;
		try {
			returnData = objectMapper.writeValueAsString(data);
		} catch (JsonGenerationException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (JsonMappingException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		System.out.println("JSON value: "+returnData );
		return returnData;
	}
 
	public Map<String, Object> readBack(String data) {
 
            System.out.println("Incoming JSON Data value: "+data);
            //Gson gson = new Gson();
            ObjectMapper objectMapper = new ObjectMapper();
 
            Map <String, Object>employee = new HashMap<String, Object>();
            try {
                  employee = objectMapper.readValue(data, new TypeReference<Map<String, Object>>() {});
            } catch (JsonGenerationException e) {
                  // TODO Auto-generated catch block
                  e.printStackTrace();
            } catch (JsonMappingException e) {
                  // TODO Auto-generated catch block
                  e.printStackTrace();
            } catch (IOException e) {
                  // TODO Auto-generated catch block
                  e.printStackTrace();
            }
            System.out.println("MAP value: "+employee );
            return employee;
    }
}

Results (Reading Single Record from JSON File):

As soon as the file is placed the process gets started and You will see it as below after completing Display File Contents activity:

Next activity screen will present the data for read/write, make changes to it and complete the activity.

You will get JSON String out of edited data as shown below:

{"Phones":[{"Home":"11111111","Mobile":"111111111","Office":"33333333333"},
{"Home":"2222222","Mobile":"2222222","Office":"34444444"},
{"Home":"6666666","Mobile":"6666666","Office":"66666666"}],
"Name":"Tanmoy Roy","dependents":[{"Name":"T1","Relation":"TR1","Age":21},
{"Name":"T2","Relation":"TR2","Age":43},
{"Name":"T3","Relation":"TR3","Age":56}],
"EmpID":"1002345","DoJ":1376332200000,"Salary":54433.0}

Steps (Reading Multiple Record from JSON File):

Edit the model to define Customer and Customers Structure Data as shown below:

Add a method to the DataTOJSON class like:

public Map readMultiple(String data) {
 
     Map outData = new HashMap(); 
     List listOfCustomers = new ArrayList();
	System.out.println("Incoming JSON Data value: "+data);
 
	ObjectMapper objectMapper = new ObjectMapper();
 
	//Map <String, Object>employee = new HashMap<String, Object>();
	List<Map<String,Object>> customers = new ArrayList<Map <String, Object>>();
	try {
		customers = objectMapper.readValue(data, new TypeReference< List<Map <String, Object>>>() {});
	} catch (JsonGenerationException e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	} catch (JsonMappingException e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	} catch (IOException e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	}
	System.out.println("List value: "+customers );
	System.out.println("Maps in list: "+ customers.get(0) + "and" + customers.get(1));
 
	ListIterator<Map<String,Object>> iter = customers.listIterator();
 
	for (Map<String, Object> map : customers) {
		listOfCustomers.add(map);
	}
	outData.put("Customer", listOfCustomers);
	System.out.println("List value: "+listOfCustomers + "and outData: "+ outData );
	return outData;
}

<to uri="bean:dataToJSON?method=readMultiple"/>

Your incoming source file is looking like this now:

[
{"name":"Tanmoy","email":"a@a.in","address":"Pune","zip":"321100"},
{"name":"Tanmoy","email":"bb@gmail.com","address":"Pune","zip":"321100"}
]

Results (Reading Multiple Record from JSON File):

When in response to the arrived file the process starts and you go to the activity Display Strcutured Data, you will get this view:

After editing the list of Customers you get this JSON (assuming you changed the address as Mumbai of one customer and address as Delhi and zip as 11023 of the other customer:

[
{"name":"Tanmoy","email":"a@a.in","address":"Mumbai","zip":"321100"},
{"name":"Tanmoy","email":"bb@gmail.com","address":"Delhi","zip":"110023"}
]

Steps (Waiting for File arrival at FTP folder):

Define a process like shown below (Ensure you check Process Supports Attachments - in our example we intend to attach the incoming file to started Process Instance):

• 

  Process Model

Have Camel Route and additional bean of Camel Trigger defined as below - additional bean is needed to applying filter about the file extensions that you are interested in monitoring):

Camel Route - 
<from  uri="ftp://motu@localhost/FtpFeed?password=motu&amp;filter=%23myFilter&amp;stepwise=false&amp;delay=5000&amp;delete=true"/>
<to uri="ipp:process:start?processId={SendMailProcess}CamelFTPProcess&amp;data=FileContents::${bodyAsString(String)}"/>
<to uri="ipp:process:attach?ippAttachmentFileName=incomingFile.txt&amp;ippAttachmentFileContent=${bodyAsString(String)}"/>
 
 
Additional Bean - 
<bean id="myFilter" class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter">
      <property name="includes" value="*.txt,*.csv, *.xml"/>
  </bean>

Save and deploy the model.

Please add these 2 jars to WEB-INF/lib of your project (required for Camel FTP)

1. camel-ftp-2.9.2.jar
2. commons-net-3.1.jar

Results (Waiting for File arrival at FTP folder):

Place files in the FtpFeed folder of the home directory of ftp user (motu in this case). The file will be consumed and deleted and a process named CamelFTPProcess will be started.

• 

  Activity showing incoming file as Strcutured Data

• 

  Showing incoming file as Process Attachment

The model can be found here

Artifacts:

Please get the required artifacts from here