Handle huge Response Payload (18 MB) -JSON - Apigee Unable to consume it

[#1]

Team,

Use Case : Invoke multiple services(asynchronously) ,mashup data from those responses to produce a single output.

For this we are using node JS approach to make multiple calls.

Note : Our Services - can produce huge response of payload size of 10-18 MB.

Apigee document says -there is a limitation handling payload size -10 MB. To overcome this limitation we went ahead and tried streaming on response

Followed the below links ::

http://docs.apigee.com/api-services/content/enabling-streaming

https://community.apigee.com/questions/14240/https-streaming-api-using-apigee.html

We have added the required configuration on the Target Proxy..

<TargetEndpoint>
...
...

<ScriptTarget>
        <Properties>
            <Property name="keepalive.timeout.millis">60000</Property>
            <Property name="response.streaming.enabled">true</Property>
            <Property name="request.streaming.enabled">true</Property>
        </Properties>
        <ResourceURL>node://RetrieveProductDataFromDataSources.js</ResourceURL>
    </ScriptTarget>
</TargetEndpoint>

After this change - we are getting error for Empty JSON.. Please advice how to resolve this problem.

What we are missing here.?

Let me know if you need any further information

[#2]

Note:- we have tried JavaScript HTTP callout approach to make asynchronously calls - in this approach also we are unable to consume the huge payload.

[zubermohammad]

Hi @Murali,

I assume you are using streams in Nodejs, which is a bug in some OPDK installations, please refer support ticket APIRT-1827 or have a look at the question "Streaming response body to client in node".

If you could clarify on what you mean by "we are getting error for Empty JSON.".

Please give what exact error you are getting in trace tool or is it due to empty JSON body being passed something.

I would suggest to use Java Callout if you are having such big payload processing. In java you have direct access to streams and don't need to rely on file buffering which is one way to do in Nodejs.

[#4]

Hi @Mohammed Zuber,

Thanks for your response. We are using Javascript approach now for mashing up 2-3 responses as explained here

https://community.apigee.com/articles/2340/asynchronous-http-requests-in-an-api-proxy.html

Javascript1: To invoke the endpoint

var calloutResponse = httpClient.get('http://httpbin.org/get'); 
context.session['calloutResponse']= calloutResponse;

Javascript2: To get the response from session and use it for Mashup.

var exchange = context.session['calloutResponse'];
exchange.waitForComplete(1000);//timeout in millisecondsvar responsePayload = exchange.getResponse().content;

Above javascript approach works fine most of the times even when payload size is ~10 MB.

But when the payload size is ~18 MB then for some reason Javascript2 is failing saying "Javascript runtime error: \"TypeError: Cannot read property \"content\" from undefined" ". Response is undefined at that point and is it is because of any buffering issues?

Any ideas/suggestions are appreciated.

Thanks,

Pradeep

[#5]

Hi @Mohammed Zuber, I again tried edge cases with Javascript approach. Everything works fine until I reach 10 MB. After that it starts failing. Apigee documentation says that there is payload size limit of 10 MB for Edge Cloud. Looks like streaming as an option to increase the payload size limit but that option doesn't work for me as it by passes all policies (XSLT, JS etc). Let me know if you have any thoughts or experience in this.