Mule Tutorial Series: File Endpoint, Data Mapper, Java Component

In my previous blog post, CSV to Maps component was used to convert the CSV input to Java Maps. The CSV to Maps component loads the entire CSV content in memory before transforming into Maps. In case the file is large, this approach would lead to large memory consumption. Mule provides a memory efficient solution using the DataMapper component.

The flow depicted below is similar to previous blog example; the only difference is that the CSV to Maps component is replaced with DataMapper.

File-DataMapper

The File endpoint is configured to pick up files from specific location and on completion of processing move it to a pre-defined location. The developer is free to choose location as per his/her convenience.

My configuration for properties within the ‘General’ tab of the endpoint is below:

Path: C:\MuleStudio\File\In
Move to Directory: C:\MuleStudio\File\Out

Use the DataMapper component’s graphical mapping editor, select CSV as the input and Collection of POJO as output. The graphical editor should show the mapping between the two formats as per the screen shot below.

File-DataMapper-UI

The individual field level mapping is automatically generated by the tool. This is sometimes a time consuming process. Do remember to select a Collection and not POJO on the right side mapping. To view the actual generated transformation logic, click on ‘Script’ in the editor. In case you need further customization, edit the script and make changes.

File-DataMapper-Script

In the present mapping, the date of birth mapping’s date format was changed to align with the expected CSV format.

The generated mapping file is saved and stored in the mapping folder of the source code. For the current example, the csv_to_pojo.grf file is the generated mapping file. For detailed explanation around configuring DataMapper mapping please refer mule documentation.

The final component of the flow is the Java component. Its configuration is as below.

In the ‘General’ tab the following configuration is set up.

Class Name: com.vinraj.integration.file.PersonPrinter

The source code of PersonPrinter and Person class is shown below:

package com.vinraj.integration.file;

import java.util.List;

public class PersonPrinter {
	
	public void process(List<Person> persons) {
		System.out.println("Size: " + persons.size()); 
		System.out.println("Person list: " + persons);
	}

}
package com.vinraj.integration.file;

public class Person {
	
	private String firstName = null;
	private String lastName = null;
	private java.util.Date dob = null;
	
	//Standard getters and setters for each attribute
	
	@Override
	public String toString() {
		return "Person [firstName=" + firstName + ", 
			lastName=" + lastName
			+ ", dob=" + dob + "]";
	}
}

Running the flow as Mule application generates the following log output in the console.

INFO  2013-12-27 13:42:28,872 [[integration].connector.file.mule.default.receiver.01] 
org.mule.transport.file.FileMessageReceiver: Lock obtained on file: C:\MuleStudio\File\In\persons.csv

Size: 3

Person list: [Person [firstName=Bill, lastName=Kent, dob=Fri Feb 11 00:00:00 GMT+05:30 1977], 
Person [firstName=Mark, lastName=Tully, dob=Tue Dec 01 00:00:00 GMT+05:30 1970], 
Person [firstName=Daisy, lastName=Smith, dob=Wed Dec 01 00:00:00 GMT+05:30 1982]]

The input CSV file content is as shown below:

Bill,Kent,11/02/1977
Mark,Tully,1/12/1970
Daisy,Smith,1/12/1982

The source code of the flow XML is shown below

<?xml version="1.0" encoding="UTF-8"?>

<mule>
    <data-mapper:config name="csv_to_pojo" transformationGraphPath="csv_to_pojo.grf" 
	doc:name="csv_to_pojo"/>
    <flow name="file_dm_processingFlow1" doc:name="file_dm_processingFlow1">
        <file:inbound-endpoint path="C:\MuleStudio\File\In" 
		moveToDirectory="C:\MuleStudio\File\Out" responseTimeout="10000" 		
		doc:name="File" fileAge="1500" pollingFrequency="2000"/>
            <data-mapper:transform config-ref="csv_to_pojo" doc:name="CSV To Pojo"/>
            <component class="com.vinraj.integration.file.PersonPrinter" doc:name="Java"/>
    </flow>
</mule>

Note: the XML namespaces added within the mule element tag has been removed for code legibility.

The DataMapper component supports streaming input and output data to enable efficient processing of large data sets. The example used DataMapper without streaming feature. This example illustrates the use of DataMapper with streaming feature.

The graphical Mule flow depiction is as below:

File-DataMapper-Streaming

The configuration of the flow is exactly the same as previous example except for minor changes mentioned below.

The file endpoint’s ‘General’ tab configuration is as below:

Path: C:\MuleStudio\File\Streaming-In
Move to Directory: C:\MuleStudio\File\Streaming-Out

The DataMapper component configuration remains unchanged except for the following change.
Click on the down arrow icon in the Mule properties editor pane.

File-DataMapper-Streaming-Config1

On the properties tab of the DataMapper component, check the Streaming checkbox.

File-DataMapper-Streaming-Config2

The last component of the flow is Java. Its configuration is as shown:

In the General tab, set the Class Name property to ‘com.vinraj.integration.file.PersonStreamingPrinter’

The source code of the PersonStreamingPrinter is shown below:

package com.vinraj.integration.file;

import java.io.BufferedReader;
import java.io.DataInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.Iterator;

import org.mule.api.MuleEventContext;
import org.mule.api.lifecycle.Callable;

public class PersonStreamingPrinter implements Callable {

	@Override
	public Object onCall(MuleEventContext eventContext) throws Exception {
	
		Iterator itr = (Iterator) eventContext.getMessage().getPayload();
		
		System.out.println("Start--------------");
		while(itr.hasNext()) {
			Object obj = itr.next();
			System.out.println(obj);
		}
		System.out.println("End--------------");
		
        return null;	
	}
}

Note the PersonStreamingPrinter class uses a slightly different approach to retrieve the parsed object. This is needed due to the streaming feature.

Running the Mule flow generates the following output in the console.

INFO  2013-12-27 14:13:22,163 [[integration].connector.file.mule.default.receiver.02] 
org.mule.transport.file.FileMessageReceiver: Lock obtained on file: C:\MuleStudio\File\Streaming-In\persons.csv
Start--------------
WARN  2013-12-27 14:13:27,165 [Finalizer] org.mule.transport.file.ReceiverFileInputStream: 
Failed to move file from C:\MuleStudio\File\In\persons.csv to C:\MuleStudio\File\Out\persons.csv

WARN  2013-12-27 14:13:27,166 [Finalizer] org.mule.transport.file.FileMessageReceiver: 
Failure trying to remove file C:\MuleStudio\File\In\persons.csv from list of files under processing

Person [firstName=Bill, lastName=Kent, dob=Fri Feb 11 00:00:00 GMT+05:30 1977]
Person [firstName=Mark, lastName=Tully, dob=Tue Dec 01 00:00:00 GMT+05:30 1970]
Person [firstName=Daisy, lastName=Smith, dob=Wed Dec 01 00:00:00 GMT+05:30 1982]
End--------------

The flow’s XML source code is as below:

<mule>
    <data-mapper:config name="csv_to_pojo_1" transformationGraphPath="csv_to_pojo_1.grf" 
	doc:name="csv_to_pojo_1"/>
    <flow name="file_dm_streaming_processingFlow1" doc:name="file_dm_streaming_processingFlow1">
        <file:inbound-endpoint path="C:\MuleStudio\File\Streaming-In" 
		moveToDirectory="C:\MuleStudio\File\Streaming-Out" 
		responseTimeout="10000" doc:name="File"/>
        <data-mapper:transform config-ref="csv_to_pojo_1" stream="true" 
		pipeSize="204" doc:name="CSV To Pojo"/>
        <component class="com.vinraj.integration.file.PersonStreamingPrinter" 		
		doc:name="Java"/>
    </flow>
</mule>

Note: the XML namespaces added within the mule element tag has been removed for code legibility.

Advertisements

5 thoughts on “Mule Tutorial Series: File Endpoint, Data Mapper, Java Component

  1. Hi
    Thank you for the tutorial.

    May I ask the following questions please.

    1. Does Datamapper works only for the enterprise edition? because I have trouble mapping. as explained below.
    2. I have tried to do this datamapper example /tutorial you explained – ” Mule Tutorial Series: File Endpoint, Data Mapper, Java Component”
    however
    I have done exactly as you mentioned as –

    Select CSV as the input and Collection of POJO as output. The graphical editor should show the mapping between the two formats as per the screen shot below.

    For input csv – i used userdefined – and added three fields. – is this correct?
    However output section – it is showing list Collection
    but it is not showing any fields underneath. or any mapping .
    Could you please help me with this – I have spent almost 6 hours and could not able to figure it out.
    Any help much appreciated.
    What am I doing wrong here?

    Thank you and regards
    Nivi

  2. Very good explanation with source code. As part of 3.7 dataweave is like advance of data transformer and very easy to use.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s