How to handle streaming data together with normal request/response payload in a single API Proxy

[#1]

According to the Apigee Doc on Streaming data:

http://apigee.com/docs/api-services/content/enabling-streaming

When streaming is enabled, policies that require access to the request or response payload, such as XSLT transformations and XML to JSON policies cannot be run and are bypassed.

Our customer needs a solution of handling normal request/response payload data like JSON/XML along with steaming data like image or S3 like object data in a single API proxy with respect to its resource path, query param or Content-Type or whatever which we pass with the API call.

I'll really appreciate any hints from you on how we can realize it.

[#2 · Accepted solution]

Hi @Toshihiro Shibamoto,

Do you want it on the same base path ?

When you enable streaming, payloads more than the buffer size will be streamed else they will be considered as normal requests but if your payload > buffer size and if you have any policy which tries to operate on payload , you will see buffer overflow 413 .

In your case I feel you can enable streaming and have separate flows based on content-length header for normal else consider that as streaming request . (Never tried but theoretically should work )

if no , you can have 2 different endpoints.

cc @arghya das

[#3]

Hi @Maruti Chand,

Thank you so much for the answer to this question. As to the base path, yes we want to do this in the same API proxy, since the customer doesn't want it to be separated into multiple API proxies because of their use case. Could you advise me further on how we can enable streaming with conditional flow? Currently it seems to be bound to a ProxyEndpoint for enabling the flag. Then how can we handle it as to switch by separate conditional flow as you suggest? I'll really appreciate your advices on how to implement it with some proxy sample codes if available.

Best regards,

Toshi

[#4]

Hi @Toshihiro Shibamoto , You can have your proxy endpoint as below

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint name="default">
    <Description/>
    <PreFlow name="PreFlow">
        <Request/>
        <Response/>
    </PreFlow>
    <Flows>
     	<Flow name="Flow1">
   </Flow>
	<Flow name="Flow2">
   </Flow>
    </Flows>
    <PostFlow name="PostFlow">
        <Request/>
        <Response/>
    </PostFlow>
    <HTTPProxyConnection>
        <Properties>
            <Property name="request.streaming.enabled">true</Property>
            <Property name="response.streaming.enabled">true</Property>
        </Properties>
        <BasePath>/vsfmoivs</BasePath>
        <VirtualHost>default</VirtualHost>
    </HTTPProxyConnection>
    <RouteRule name="default">
        <TargetEndpoint>default</TargetEndpoint>
    </RouteRule>
</ProxyEndpoint>

You can have 2 flows with conditions and have policies that you want to modify the payload in flow1 and flow2 should not have any policies that can modify the payload like JSON2XML and if your payload is > 3MB (default) you will see the request streaming in flow2 .

PS : I have tried streaming but not the above scenario but I believe it should work . Pls let me know your test findings .

[#5]

Hi @Maruti Chand,

I'll try this and share it with customer.

I really appreciate your help.

Best regards,

Toshi

[jonesfloyd]

Thanks for this, Maruti! I've added a link to this post from the doc topic that Shibamoto-san referenced.

[ozanseymen]

Hi @Maruti Chand, @Toshihiro Shibamoto

I have my doubts on whether such a configuration will work the way it is documented here. @Toshihiro Shibamoto - is it possible to confirm this with your experiences with the customer please?

I am particularly interested in finding out whether:

1. "Requests below the buffer limit will not be streamed even if streaming is enabled in the proxy". The reason for my doubt is that in order for MP to calculate the message size, it should by definition buffer it. However Edge shouldn't buffer anything when streaming is enabled. Does Content-Length header play any part here? Bear in mind that this is not a mandatory header.
2. "Edge can dynamically choose a flow by looking at whether policies in it depends on request/response payload". Can anyone confirm if this is the case?

[#8]

Hi everyone,

I have a doubt on streaming.

I am passing an xml payload via javascript policy, iterating that payload based on my counter value and printing it. as soon as i am giving counter=50000 a message is coming "<CONTENT TOO LARGE TO DISPLAY>"

How to rectify this?

Thanks