Why consider the Pipeline Concept for your migrations?
If you are running a migration project from SAP PI/PO you have a pretty standard way of processing messages. It was hell to work with but the standardized way of handling the flows made it easier to deliver reliable messages.
The Pipeline Concept is something that you should consider for your migration. There are many alternatives built by consultants or in-house. But it solves the fundamental problem from SAP about how you ensure reliable messaging for your SAP Integration. In a way that can be scales without using too many JMS Queues.
You can create more creative solutions in Cloud Integration to handle your flows. And you can consider redesigning your integration. Please do. But know it does require a lot of effort on the technical and business side to change.
If you are trying to migrate as with your existing document structures then you should consider how the Pipeline Concept can help you. It will speed up your migration journey and ensure that you can have a good architecture using your existing artifacts.
As I write this in September 2024, I don’t think you will be able to operate the pipeline concept without usage of Figaf. You can explore it without, but it is cumbersome.
The reason that I find the concept nice:
- Simplifies the logic needed to build in to each iflow. If you need to make a change to the processing just do it once and it will affect all the other areas where it is used.
- One way of error handling your flows that allows you to reprocess failed messages. It makes it much easier to monitor.
- SAP will be continues going to improve the setup and have been listing for the feedback and improved it. In 1.0.6 you can see some of this new like better us of the length of the PID field and the P2P flows if you dont have routing.
- If you follow the standard, you will get continuous improvements in your processing flow.
- In time if you get new developers on the team it will be much easier to onboard them.
- The ProcessDirect flow allows you to build your own logic. If you eventually want to use some other way of processing your logic, you can reuse the ProcessDirect flows with the mapping and business logic.
- Figaf is the only tool that currently supports this concept end to end, from migration, testing, transport, and production. We will continue to support it and add more features.
In a recent video I showed how you can use the Figaf tool to manage the Pipeline Concept end to end. Since this we have added the new Point to Point migration to streamline the flow and speedup processing.
Problems
There are lots of problems that hinder your usage of it. Some/many of this will be improved over time. So if you check back later it may be working easier.
Operating
By fair it is a huge solution that can be complex to operate, and there is room for improvement. SAP is listening to this and have been expanding the solution to support the usage better.
I would recommend that you will be exploring the solution together with Figaf. It is the best option to understand and operate the flows. We have built most of the things to make it possible to operate the pipelines. We do hope that you will continue using the Figaf tool, and hope SAP’s tools will mature to operate them.
No UI for the partner directory
It was created before the Partner Directory was really used with is causing some problems. It is quite cumbersome to edit via the API. You would need to use quite a scripting engine to handle this.
SAP will deliver a UI for the Partner Directory so you can easy use it.
If you want to explore it now, then Figaf have a fully function UI for the API. This will allow you to create all partner directory values including XSLT. If you get our free migration edition you get 2 months of DevOps trial, where you can use to explore how the pipeline concept works. Then hopefully SAP delivers the UI or you may find the Figaf tool gives you a lot of advantages.
Pipeline concept is complicated
It definitely contains a lot of flows and requires the usage of Partner Directory to run, so your integration becomes more fragmented. You can essentially have an SFTP adapter in one inbound flow and then mapping and connection in the outbound flow. This can make it a bit more difficult to see how messages connect, so you will need to move more inflows.
Monitoring
If you have more messages it will be difficult to see the End2End picture of them. So one messages will be processed in multiply flows making it a challenge to see what is going on.
You can always follow the correlation id to see how a message has been processed.
Restart specific messages
The JMS currently is not super ideal to this. You dont have an option just to reprocess one message from the failed state. You can only copy a full Deadletter queue and start the process from there. I would be expecting to see improvements in this with better API for the JMS queue.
Partner Directory on Edge Integration Cell
Partner Directory is not in EIC which mean you cannot use the solution there yet. We did a PoC for a customer that replicated the Partner Directory to a value mapping. Then they upgraded the flows with the option to use value mapping (routing purpose).
This will allow them a seamless transition once the Partner Directory is released on EIC. That way you can explore how the tool works and them speed up later.
Conclusion
There are many more challenges.
But I do think it deserves your consideration if it can help you with your migration. The simpler you can build it the simple it will be to operate your integration landscape.