SAP PI message payload search using Elasticsearch
At our client we recently went live with a solution that incorporates SAP PI. At the time we did not have TREX set up to be able to perform searches on messages. (And at the time of writing we still don’t, but we should have it soon). Â So as a stopgap measure, I developed a solution that uses Elasticsearch to index payloads and allows you to search them.
NOTE: At the time of writing I remembered that in order to make this work, I had to make a fix to the Ruby nwrfc library for extending the buffer length for binary coded decimals, as otherwise it does not work. Make sure you are using the nwrfc-0.0.7 gem or later.
Elasticsearch is probably one of the coolest things I have seen in a while. I actually heard the name mentioned on our project, as it is used here for other things, but assumed it was some proprietary software. Basically it is an open source indexing and search server which communicates through a RESTful protocol. It uses a schema-less data store in which every field is indexed. There are dozens of supported clients, including one for Ruby.
Prior to this solution I actually developed one for indexing and searching ABAP code which replaces my previous efforts using Apache Solr. I have started disliking Solr more and more since in the last releases it requires increasing effort to get it up and running. Not so with Elasticsearch. It is really a zero-config solution. You plonk it down and run it; no further configuration required.
The solution I present here consists, on the one side, of nothing more than a ruby script that uses the nwrfc gem to retrieve PI messages using RFC and then submits the data to Elasticsearch for indexing by means of the elasticsearch-ruby gem. On the other side it has a search page which can be hosted as a plugin for Elasticsearch to search payloads and allows you to find matching requests or responses.
Now I should add a word of caution that I have not done much in the way of optimizing the retrieval script, e.g. for bulk indexing of requests which could probably speed up the process. Furthermore, it currently only allows you to set a start and end time for which to fetch payloads. The idea was to parameterize these options and run the script at regular intervals, but in the light of the fact that we will soon have the standard SAP solution in place, I stopped development thereof.
In addition, the search page is not very refined. I had initially had all these dreams about growing a fully-fledged application, which is why the user interface uses Webix, a superb Javascript UI library, which is admittedly an overkill for the search page, but could be used to build a great app.
There are other nice things that could be added. Currently the search page only supports searching through one index, for example. (So you have to make sure to specify the same index as in the retrieval script).
Anyway, it is a stopgap measure.
Someone at work pointed me to the Kibana plugin for Elasticsearch, which allows you to create smashing dashboards on the fly by specifying arbitrary queries and graphing them. This is very nice. I had first developed some graphs using webix, but that is rather cumbersome, although admittedly more flexible.
You can find the script that does the retrieval and indexing, as well as the search page in the following gist on GitHub:Â https://gist.github.com/mydoghasworms/19b9ba447c2fbf46c994
Leave a Reply