I asked how people were monitoring SAP Cloud Integration, especially how you are storing messages or payloads to understand what is going on with an iFlow.
It did spark a good debate and discussion, with a lot of comments.
Initial thought. SAP Cloud Integration is not really designed for this there is just a 32 GB limit of storage of payloads. For a system that can process millions of messages then it is not sufficient to make it possible to store it all.
It generally means you need to understand your need and how much data you will be delivering.
MPL attachments
This is the most common approach to handling monitoring with 54% of the responses. Probably because it is easy to view payloads in the monitor aspect. MPL attachments have been pretty easy to use. From a Groovy script, you can save some messages as an attachment.
There are also some problems with it as outlined in the note. 2593825 – MPL Attachment is not writing because Circuit Breaker is open in SAP Cloud Integration, and contains the following points.
- The threshold for Circuit Breaker is capped to 1GB of MPL attachments per 24 hours.
- Without this protection, Database of SAP Cloud Integration might become overloaded, which in turn causes more severe and unexpected errors.
There are also different people that have burned their hands with the approach, so be careful about what you are logging.
Persist Messages
I thought this was the best approach for creating a way to handle the flow. You simply just add a new step to save data in a persist message. Then you are not limited to the daily 1GB of data.
The manual does specify that there are only 32 GB of space for this in combination with Data Store.
So it is not something you can use for saving payloads for every integration.
I’m not sure how it works if it just deletes the first payloads once you reach 32 GB or it will stop working.
Another problem with persisted messages is that you can only get access to it via the API so there is no UI components for it.
There are some options for it.
- Figaf Suite gives you access to the monitor CPI Iflows and views the payloads from both MPL and Persisted messages. And you can then give access to different business users.
- It seems like CPI helper also has an option to view the payloads directly. You just need to use the plugin while looking at a payload.
Data Store
I had not thought that Data Store could also be used as a monitoring option. I thought it was more for using once you had correlations you wanted to perform.
It is using the same datastore as for the persisted messages, which probably means that they can be interlinked.
One benefit is that you can set expiration days on the payloads, so even successful messages will only be saved for 2 days and failed for 5 days. This gives you a better way to manage the workflow.
One challenge is still that you need a good way to access the payloads. Datastore does have a UI that enables you to find the messages.
In the short term, it would be nice to add a feature to the Figaf Tool to simplify monitoring iFlows with Data Store objects so you can see them just as with Persisted messages.
JMS/Queues
If you need processing and have processing you will need to add messages to a queue and then start processing it. It will add a little extra complexity and some cost but will allow you to send messages multiply times and have a retry.
External location
If you want to be sure you can access and save payloads it may be an option to save important payloads in an external system like SAP Document Management, AWS S3 or a logging platform (Splunk) may also be a possibility. Or I have seen a number of cases a HANA database has been used for the staging.
It will require an external call out to the system to save the messages but then you have a good way to persist them.
How do you handle logging?
Please share if you have a better way to handle logging here.
Figaf Suite Monitoring
In the Figaf Tool we have a number of components that makes it much easier for you to handle your monitoring. This will simplify the way you are building iflows. The tools have the following options
- Improved iFlow monitor so you can see both MPL but also Persisted Messages in one go. With this option, you have a better option to search for real payloads and find them
- Alerting rules. If an iflow fails for some reason it is possible to notify the right person about the problem. This simplifies the monitoring aspect of the setup
- Persistence of message. In the Figaf Tool you also have the option to specify that you want to save messages for longer than 30 days with the EDI Monitor. This allows you to download all messages and then save them into the data store.
It is pretty easy to get started with the Figaf Tool we have a guide to install Figaf in just 10 minutes to monitor CPI