For the past few months I have been working on an application which has a lot of integration points; @20 internal as well as external applications/systems. As considerable amount of data flows between the interfacing systems, the data transfer is a batch feed; normally fixed length file. The file transfer pattern is probably the most earliest EAI pattern used.
Spending that much time on the file transfer pattern reminiscing the benefits and limitations of this integration pattern.
Isolation from programming language/platform idiosyncrasies
From the integrating systems point of view the integrating system’s area of concern is limited to a single file feed. As long as the format is as per mutually agreed rules and the file content integrity(no information loss during transmission) maintained, the receiving system does not need to know or care if the sending system is a mainframe, client server application or web application. File Transfer provides a simple but powerful method to achieve Enterprise Application Integration.
File processing ubiquity/standardization
All programming languages provide ready-to-use API to open, interogate and process files. No special devices/software is required to retrieve file content data. Note the statement made above applies in case the file uses fixed length or comma separated value(CSV) as its communication format. Selection of more modern formats like XML or JSON etc may require additional software components to process the feed.
File Transfer protocol standardization
The file transfer process is facilitated by the FTP or SFTP protocol. The standardization of the protocol ensures interoperability and availability of standard tooling to implement file transfers across programming languages and platforms.
So what are the trade offs of using file transfer for EAI?
Not real time
File transfers are scheduled periodic jobs which will function at regular intervals during the day, or as an end of the day/week/month/year cycle. This will make the receiving system’s information accurate until the last snapshot of the file transfer. Greater that duration between two successive file transfers, greater the possibility of receiving system dealing with stale data.
Contains data no meta-data
File transfer is implemented using the following data communication formats, namely fixed length, comma separated values and XML. Fixed length is the most popular data communication format(largely because legacy systems support it). Only XML format allows meta data to flow along with data, in case of fixed length or CSV only data follows. It is the responsibility of receiving system to understand the meta data represented based on the order of placement of data with the file.
Multiple System updating shared data not supported
File transfer does not provide a mechanism to support consistent multiple updates to shared data. A separate mechanism needs to be put in place to support this. Generally by enterprise design data manipulation is done by a single system and downstream system purely read the data. Else they have a standardized defined data flow to ensure consistency.