SOAP Message Payload search in ABAP using HANA Fulltext index
I was initially planning to use Elasticsearch for this but, as it turned out, using a fulltext index on HANA was quicker to implement and simpler and, besides, introducing another service in the landscape at the site at which I am working could prove difficult or impossible.
At my current client, the consumers are calling our Enterprise services on the ABAP SOAP runtime (SRT) directly and not going via PI. This leaves us with a bit of a challenge, as we have no way to monitor SOAP messages, nor can we locate problems easily.
The main advantage of this solution (as would have been the case with an Elasticsearch one) is the ability to do a full text search through the SOAP message payloads, which is arguably its biggest selling point.
I developed the solution on HANA but it would conceivably work, with some tweaks, on any other database system that supports full text indexes on fields, which would probably just mean having to change the SQL statements shown below that create the full text indexes so suit the syntax of the target DB platform.
The solution comprises two aspects: the first is a way to record the SOAP messages in a table, and the second is a program that allows you to search through them. (Well, there are three aspects if you also consider adding a program to delete old messages, which I haven’t finished yet).
First, let’s look at the definition of the table that stores the recorded messages:
Note that all the fields, beside the key, are contained in a structure which are included in the table definition.
The service name, namespace and operations, which are passed as type STRING in the SRT, I am storing with a max of 132 characters. This is only because, when I compile a list of all the services and operations for selection further down the line, it is not possible to do a GROUP BY on what are essentially LOB fields (i.e. if I was storing them as type STRING in the database).
Moving on, we now need to record the incoming SOAP messages. To do this, I created an implicit enhancement at the end of method HANDLE_REQUEST in class CL_SOAP_HTTP_EXTENSION, which is the handler for a few ICF nodes under the ICF node “/sap/bc/srt/”, that handle SOAP messages, including “/sap/bc/srt/xip”, which is the ICF node that handles SOAP requests for XI messages.
The method with the enhancement looks as follows:
 To avoid any possible interference with the current LUW when calling a SOAP service, I am using an alternative connection for saving the record to the database, in this case the built-in connection “r/3*_bal_trace”, which is used for saving application logs (think SLG1). I guess one could have called the function in aRFC or qRFC or with a delayed update in the current LUW (all to avoid interfering with the processing of the service call), but anyway, this works as well. (I will probably change this later, see below). The source for the function module which is called in the above enhancement is given here:
Now, of course the data that is stored in the table will be (more) useful only if a full text index is created on the necessary fields which, in this case, are the request and response payload fields (REQUEST_PAYLOAD and RESPONSE_PAYLOAD). I have made the creation of the fulltext indexes part of the search program which is a hidden function accessible via the command field, where one must enter ‘CRFTI’ as the command, which will create the indexes (which you will see in the source of the program).
The listing of the search program is shown below. A link to the Github Gist with all the relevant code and information is at the end of this post. Note, however…
It’s not finished yet!
The solution as it stands now can hardly be deployed in a production system. The volumes of service calls that will be recorded are very high and the table will grow very quickly. What I am going to build next is a way to filter the requests that should be stored for long-term inspection (mainly those that do updates; reads are of lesser interest). For this, I think it will be better to change the update function to be called in some variant of asynchronous RFC so that it does not add unnecessary processing time to the service call. In addition, there needs to be a program to delete all recorded messages by age. In fact, I am working right now on adding a way to configure which requests and/or their respective payloads are stored.
Here is a screenshot of the search program in action. I am almost more proud of how I placed the GUI controls directly on the report selection screen (which is the culmination of other investigation into GUI controls) than I am of the full text index solution itself.
As promised, here is the listing of the search program, although you can just visit the Gist (link at the end) to see all the necessary parts of the solution.
https://gist.github.com/mydoghasworms/faeaf787cb08ded856e5bf75f986aac7
Leave a Reply