Unable to read complete request/response payload

[cmarreddy]

Hi , 
I am trying to read the complete request payload using Apigee X policies such as Javascript and Assign Message policy and was not able to read the complete request payload. Instead, the payload is truncated. 

Additionally, when the Payload is truncated, I used the extract variable policy to extract a variable and was unable to extract since the request content is truncated.

Can you please let us know how to read the complete request/response payload in APIGEE X? If this functionality is currently not available  in APIGEE X, can you please let us know the alternative ways to resolve the issue.

Note: Tried the same logic in APIGEE Edge and was able to read the complete payload but not in APIGEE X.

Thanks

[dchiesa1]

There are a few reasons why you will not be able to "read the complete payload" in a JS policy, or in AssignMessage or ExtractVariables. 

1. the payload exceeds the stated platform limit of 10mb
2. the payload includes binary data

The first item is fairly obvious and I won't go into greater detail here about it. 

The second item is more curious - the problem is the JS policy treats content as a STRING, so if you have, for example, a multipart form in which some of the parts are binary streams (like a PDF or PNG) then the JS won't be able to parse or handle that payload effectively. The workaround is to use a callout that can handle non-string data, such as Python(Jython) or a Java callout. 

---

@cmarreddy wrote:

Instead, the payload is truncated. 

---

If you elaborate and explain what you mean by that, and how you observe that "the payload is truncated", I might be able to help further.

[cmarreddy]

My payload size is 78.2KB and my content type is application/XML and tried to execute the same in Apigee Edge and I am able to read the complete payload. 
"the payload is truncated" - I am able to read the partial payload instead of full payload.   

[dchiesa1]

If your payload size is 78kb then you are not reaching the payload size limit. 

I understand the meaning of the word "truncated".  What I don't understand is "I am able to read".  Exactly what are you doing to determine whether "you" can or cannot "read" the payload?  What are you observing that leads you to conclude that "you are not able to read" the payload?  

Is there a policy step you are executing?  Is there an XSL sheet you are executing? Are you drawing your conclusion based on an observation of Apigee trace?

What leads you to the conclusion?  What specifically does it mean "I am not able to read"? 

[cmarreddy]

I am seeing the truncated content as part of the Debug Session when tried to read into a variable using Javascript policy or Assign Message policy.print(content.getVariable("request.content"));

Additionally when tried to extract the variable which is part of truncated content using extract variable policy.

I am aware of the request body would be truncated in the debug session if it is exceeding certain limit and should be able to view the complete payload similar to the functionality in Apigee Edge when read variable into a variable. The Javascript policy is not able to parse the request content as it is truncated and not complete.

[dchiesa1]

I understand that in your first sentence, you are telling me, that in the debug UI, you cannot see the full payload.  That is as expected; the debug UI has changed in Hybrid and X, and it no longer shows the full payload. The payload is there, it's just that visually, you cannot inspect it in the debug ui.

The Javascript policy is not able to parse the request content as it is truncated and not complete.

That is a different statement entirely!  

Which is it?- you are not seeing the full content in the Debug UI?  Or the JavaScript step is not able to process (I will not use the word "see" here) the full length of the payload? 

If you are asserting that the JS step is not able to access the full payload, I want to verify that you have checked the length of the payload with logic like: 

var c = context.getVariable('request.content');
print('length of payload is: ' + c.length);

And you are telling me that the length is less than the ~78kb you expect?  If so, that's a bug.  But I am very skeptical because the JS step always has access to the full payload, if it is text. 

And your payload is plain XML?  It's not MTOM / XOP ?  There are no attachments?  

[cmarreddy]

"That is a different statement entirely!"  
I am not able to read the entire payload into a variable using Javascript policy, Since it is not able to read the complete payload in the variable it is not able to parse it. Both are interrelated. 

Content type is application/XML
Checked the length of the payload with the logic provided.
Length of payload is: 93797 which is around 94KB(which is less than 10MB or 3MB)