Migration of SAP PI File Channels with Content Converter

One of the challenges with migration is how to handle the File Content Converter when you are working on a migration to SAP Integration Suite.

The solution is not only for Figaf customers, you can use it as a standalone.

Background

When XI 3.0 was introduced, there was significant file processing activity. Communication often relied on TXT or CSV files that required parsing.

In the File communication channel, there was an option to directly create a content conversion. This feature allowed for the conversion of CSV/TXT files into a predictable XML structure, which could then be processed via routing and message mapping.

Later, this functionality was also made available as a module for use by other users.

Migration Challenges

In SAP Cloud Integration, the CSV to XML converter operates differently, utilizing an XSD to facilitate migration. This means that if a file needs to be migrated, it must be converted into a different structure, possibly necessitating updates to your mappings.

Converting from XML to CSV/TXT is somewhat simpler, as you can readily use an XSLT or a Groovy script for the conversion.

However, if you wish to reuse your existing message mappings, the input structure must remain the same. Creating these mappings in Groovy can be complex, depending on your team’s expertise.

Solution

Our goal was to simplify the migration process and assist customers in migrating without the need for a Groovy developer to handle all migration scenarios. We have developed an open-source project converter that enables the migration of existing SAP PI channels without modifying the sources.

You can explore the Content-converter project here. We built it from the ground up, based on test cases we devised. It may not be fully compatible with what you have on SAP PI/PO. However, it can be improved by creating new unit tests for it.

For Figaf customers, please submit a case at the cost of one ICO migration. Otherwise, submit it via GitHub, ensuring not to use real sensitive data, as it will become part of our test repository.

How it Works?

We have created a straightforward setup. The converter project is saved as a JAR file in your iFlow.

Then, you can call it with a Groovy script as follows:

import com.sap.gateway.ip.core.customdev.util.Message
import com.figaf.content.converter.ConversionConfig
import com.figaf.content.converter.ContentConversionException
import com.figaf.content.converter.xml.FlatToXmlContentConverter

def Message processData(Message message) {
    def conversionConfig = createConversionConfig()
    def inputStream = message.getBody()
    byte[] inputByteArray = inputStream.bytes
    def contentConverter = new FlatToXmlContentConverter()
    try {
        byte[] xmlConvertedFile = contentConverter.convert(inputByteArray, conversionConfig)
        message.setBody(xmlConvertedFile)
    } catch (ContentConversionException ex) {
        def messageLog = messageLogFactory.getMessageLog(message)
        messageLog.addAttachmentAsString("OriginalPayload", message.getBody(String), "text/plain")
        messageLog.addAttachmentAsString("ContentConversionException", ex.getMessage(), "text/plain")
        throw ex
    }
    return message
}

def static ConversionConfig createConversionConfig() {
    return new ConversionConfig(
        recordsetStructure: 'HR,1,PR,*,LI,*,KK,*',
        documentName: 'CPIListFixed',
        documentNamespace: 'https://figaf.com/CPIListFixed',
        ignoreRecordsetName: false,
        sectionParameters: [
           'KK': new ConversionConfig.SectionParameters(
                        fieldFixedLengths: null,
                        fieldNames: 'KEY,StartTime,Duration,IFlow,MessageID,CorrelationID,ApplicationMessageID,ApplicationMessageType,Status,Sender,Receiver,DOC,ORDERNR,Vendor',
                        keyFieldValue: 'KK',
                        fieldSeparator: ';'
                    ),
           'PR': new ConversionConfig.SectionParameters(
                        fieldFixedLengths: '2,2,5,6',
                        fieldNames: 'KEY,TYPE,ID,NAME',
                        keyFieldValue: 'PR',
                        fieldSeparator: null
                    ),
           'HR': new ConversionConfig.SectionParameters(
                        fieldFixedLengths: '2,3,4',
                        fieldNames: 'KEY,ID1,Doc',
                        keyFieldValue: 'HD',
                        fieldSeparator: null
                    ),
           'LI': new ConversionConfig.SectionParameters(
                        fieldFixedLengths: '2,2,6',
                        fieldNames: 'KEY,LINE,AMOUNT',
                        keyFieldValue: 'LI',
                        fieldSeparator: null
                    )
        ]
    )
}

Then it is just a normal flow with the Groovy script.

Also, the same kind of script for Receiver conversion where you can convert from XML to Text/CSV.

Automated Migration

If you are using Figaf migration tool you really do not need to consider anything for it. You will of cause need to select the template you have to convert your file channel to SFTP/FTPS system. The addition of the Content converter happens automatically.

Testing

As with all other things, you need to be performing tests. We have found that on file communication channels, you could add the Additional Parameter called messageLog to true and then it would be possible to see the payload in the monitor.

Then you can create a Test case with Figaf to enable you to test with the original payload.

Additionally, then find the check mark on the testing page to enable recording of the original payloads. This way you get to check that everything works as expected.

Also, yes Figaf can test messages starting form SFTP/FTPS it just create a new iFlow with same settings and run the test on it. Clean and simple. 

Modules

There is also some of the conversion that has happened in an adapter module. We have not analyzed the setup to see if it will be possible with the current module or with a small extension. So it may be possible but we will need to see some use case for it before we start implementing it.

Demonstration

You can see the full end to end demonstration including testing and migration using Figaf for both Sender and Receiver channel.

Try it out for Free!

It is straightforward to get started with Figaf and use it in your migration project. You can install and run Figaf in less than 1 hour and then perform your first migration then.  

Just sign up for the free Migration Edition.