Handle erroneous Pub/Sub messages in SAP using ABAP SDK

[ajith-sap]

Systems Integration is a very common requirement for enterprises running business operations on SAP ERP. Because Pub/Sub allows systems and services to communicate asynchronously in near real time, it can be an ideal choice for SAP integration. Take a look at this series of blogs to get a general overview of how Pub/Sub and ABAP SDK for Google Cloud can be used to build SAP integration solutions.

Error handling for SAP Integrations

Error handling if often a mandatory requirement when you build an integration in SAP. Specifically, if you are pulling messages from a Pub/Sub subscription, your logic should handle cases where a message cannot be processed due to errors in the message (ex: missing mandatory attribute) or errors in SAP (ex: missing configuration or data in SAP).

• If you don’t acknowledge the message, the message will be included again next time messages are pulled from the subscription. However, if the error keeps happening, the list of such unprocessed messages can grow significantly in a very short period of time, causing performance issues in your integration and delaying processing of newer messages.
• If you acknowledge the message with error, the message is lost and cannot be re-processed again.

Both of the above are not desirable options for handling messages with errors. In this blog, I will explain how you can build your logic such a way that error scenarios can be handled efficiently without impacting the flow of messages that can be processed successfully. We will be using an in-built feature of Pub/Sub called dead-letter topic.

How dead-letter topic works

If a subscriber pulls a message from a Pub/Sub topic but cannot acknowledge it for any reason, Pub/Sub can forward the undelivered message to a dead-letter topic. You can define and set a dead-letter topic as a subscription property when creating a subscription for another topic.

Here’s a high level flow of how a dead-letter topic can be configured to work with a program that uses ABAP SDK for Google Cloud to pull messages from Pub/Sub:

1. A message is sent to MAIN_TOPIC from the Publisher.
2. An ABAP Program runs at regular intervals (ex: every 5 minutes), reading messages from subscription MAIN_TOPIC_SUBSCRIPTION of the topic MAIN_TOPIC using ABAP SDK. If a message cannot be processed, ABAP Program will not acknowledge the message.
3. If the ABAP program has not acknowledged the message after pulling it 5 (configurable) times from the subscription, the message is sent to topic DEAD_LETTER_TOPIC and deleted from subscription MAIN_TOPIC_SUBSCRIPTION.
4. ABAP Program can run in “Error Re-processing” mode at infrequent intervals (ex: once a day) or on Ad-hoc basis after the error is resolved. In this mode, messages are pulled from subscription DEAD_LETTER_TOPIC_SUBSCRIPTION of topic DEAD_LETTER_TOPIC.

Thus, with a few steps, we now have components that can efficiently handle error scenarios.

Benefits of dead-letter topic

Below are some benefits of this approach of using dead-letter topic

• Regular processing of “good” messages can continue without being impacted by erroneous messages
• No need to store the entire message in subscribing system (SAP in our case) or write logic to keep track of number of number of pulls and send the message to dead-letter topic.
• You can setup alerts using Cloud Monitoring when messages are sent to dead-letter topic.
• When alerted, a person can view content of the message to validate whether error is due to the message itself (ex: incorrect or missing attribute)
• If required, the publishing system can correct the errors and publish a new message

How to set up a dead-letter topic

You can run below gCloud commands to create topics and their subscriptions.

gcloud pubsub topics create MAIN_TOPIC
gcloud pubsub topics create DEAD_LETTER_TOPIC
gcloud pubsub subscriptions create MAIN_TOPIC_SUBSCRIPTION \
  --topic=MAIN_TOPIC \
  --dead-letter-topic=DEAD_LETTER_TOPIC \
  [--max-delivery-attempts=5] 
gcloud pubsub subscriptions create DEAD_LETTER_TOPIC_SUBSCRIPTION \
  --topic=DEAD_LETTER_TOPIC \

An important setting to highlight is when defining subscription MAIN_TOPIC_SUBSCRIPTION, we add the topic DEAD_LETTER_TOPIC as the dead-letter-topic property and we set 5 as the max-delivery-attempts. With a Pull delivery type for the subscription (default type set for MAIN_TOPIC_SUBSCRIPTION), 5 is the maximum number of times a message can be pulled from this subscription, after which the message is automatically forwarded to topic DEAD_LETTER_TOPIC.

To forward messages to a dead-letter topic, Pub/Sub must have permission to do the following:

• Publish messages to dead-letter topic
• Acknowledge the messages, which removes them from the subscription.

To grant Pub/Sub permission to publish messages to the dead-letter topic, run the following command after replacing 123456789012 with your Google Cloud Project number:

PUBSUB_SERVICE_ACCOUNT="service-123456789012@gcp-sa-pubsub.iam.gserviceaccount.com"

gcloud pubsub topics add-iam-policy-binding DEAD_LETTER_TOPIC \
  --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
  --role="roles/pubsub.publisher"

To grant Pub/Sub permission to acknowledge forwarded messages that have not been acknowledged, run the following command after replacing 123456789012 with your Google Cloud Project number:

PUBSUB_SERVICE_ACCOUNT="service-123456789012@gcp-sa-pubsub.iam.gserviceaccount.com"

gcloud pubsub subscriptions add-iam-policy-binding subscription-id \
  --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
  --role="roles/pubsub.subscriber"

ABAP Program Logic

Now that Pub/Sub resources are all wired up, here is a sample ABAP program showing how messages can be processed from different subscriptions based on processing mode.

REPORT zdead_letter_topic_sample

SELECTION-SCREEN BEGIN OF BLOCK b1 WITH FRAME TITLE TEXT-001.
  PARAMETERS:
    rb_main RADIOBUTTON GROUP g1 DEFAULT 'X',
    rb_err  RADIOBUTTON GROUP g1.
SELECTION-SCREEN END OF BLOCK b1.

CLASS lcl_process DEFINITION FINAL.
  PUBLIC SECTION.
    CLASS-METHODS:
      main
        IMPORTING iv_mode TYPE flag,
      process_message
        IMPORTING is_message        TYPE /goog/cl_pubsub_v1=>ty_029
        RETURNING VALUE(rv_success) TYPE flag.
ENDCLASS.

START-OF-SELECTION.
  lcl_process=>main( rb_main ).

CLASS lcl_process IMPLEMENTATION.
  METHOD main.

    TRY.

        "Open HTTP Connection
        DATA(lo_client) = 
          NEW /goog/cl_pubsub_v1( iv_key_name = 'MY_CLIENT_KEY' ).

        " Derive project id from the client object
        DATA(lv_p_projects_id) = CONV string( lo_client->gv_project_id ).
        " Name of the subscription from where we want to pull data
        DATA(lv_p_subscriptions_id) = 
           COND string( WHEN iv_mode = 'X' THEN 'MAIN_TOPIC_SUBSCRIPTION'
                         ELSE 'DEAD_LETTER_TOPIC_SUBSCRIPTION' ).

        "Call API method
        CALL METHOD lo_client->pull_subscriptions
          EXPORTING
            iv_p_projects_id      = lv_p_projects_id
            iv_p_subscriptions_id = lv_p_subscriptions_id
          IMPORTING
            es_output             = DATA(ls_output)
            ev_ret_code           = DATA(lv_ret_code)
            ev_err_text           = DATA(lv_err_text)
            es_err_resp           = DATA(ls_err_resp).

        IF /goog/cl_http_client=>is_success( lv_ret_code ) IS INITIAL.
          MESSAGE lv_err_text TYPE 'E'.
          RETURN.
        ENDIF.

        IF ls_output-received_messages IS INITIAL.
          MESSAGE 'No Messages were received!' TYPE 'S'.
          RETURN.
        ENDIF.

        LOOP AT ls_output-received_messages REFERENCE INTO DATA(lr_message).

          "Process the message
          IF process_message( lr_message->* ) IS NOT INITIAL.

            "If processing is successful -> Acknowlege the message.
            DATA(ls_input_ack) = 
               VALUE /goog/cl_pubsub_v1=>ty_001( 
                        ack_ids = VALUE #( ( lr_message->ack_id ) ) ).

            "Acknowledge the messages so it is not pulled again.
            CALL METHOD lo_client->acknowledge_subscriptions
              EXPORTING
                iv_p_projects_id      = lv_p_projects_id
                iv_p_subscriptions_id = lv_p_subscriptions_id
                is_input              = ls_input_ack
              IMPORTING
                es_output             = DATA(ls_output_ack)
                ev_ret_code           = lv_ret_code
                ev_err_text           = lv_err_text
                es_err_resp           = ls_err_resp.

            IF lo_client->is_success( lv_ret_code ) IS INITIAL.
              MESSAGE lv_err_text TYPE 'S' DISPLAY LIKE 'E'.
            ENDIF.

          ENDIF.

        ENDLOOP.

        lo_client->close( ).

      CATCH /goog/cx_sdk INTO DATA(lo_exception).
        MESSAGE lo_exception->get_text( ) TYPE 'E'.
    ENDTRY.

  ENDMETHOD.

  METHOD process_message.

    "Messages published to Pub/Sub is base-64 encoded
    "Decode to get the actual message
    DATA(lv_msg) = cl_http_utility=>decode_base64( is_message-message-data ).

    "TODO - 
    "Write business logic to process the message
    "If message is not processed successfull, log errors
    "Populate rv_success = 'X' if processing is successful.

  ENDMETHOD.

ENDCLASS.

When processing messages in “Main” mode, if a message is not acknowledged after 5 pulls from subscription MAIN_TOPIC_SUBSCRIPTION, Pub/Sub will automatically forward the message to topic DEAD_LETTER_TOPIC. These erroneous messages can be processed later in “Error Re-processing” mode without impacting “Main” mode.

Additionally, errors related to the processing of messages can be written to a resource of your choice: SAP Application Logs (BAL), Z table etc. A user can use this error log to take corrective action in SAP for errors due to incorrect configuration or data in SAP.

Building a robust integration using Pub/Sub and ABAP SDK for Google Cloud

As described above, you can build scalable system integration solutions with error handling logic in SAP by using Pub/Sub and ABAP SDK. Below are some key resources to learn more, ask questions and collaborate with us:

• Join our Google Cloud Community https://www.googlecloudcommunity.com/gc/Cloud-Product-Articles/tkb-p/cloud-articles/label-name/abap%...
• Subscribe to our YouTube channel https://www.youtube.com/@SAPOnGoogleCloud

Happy Learning and Innovating with Google Cloud !