Solved: Re: Streaming with ServiceCallout Response

harishav · 11-22-2022 10:09 PM

Hi Team,

We have a target server which returns the URL for downloading the file and we then use a ServiceCallout Policy to download the files from the URL and send that file data as a response from our proxy, and there are usecases where our file sizes can be greater than 10 MB.

I see we have an option to enable streaming to support payloads more than 10 MB, but that looks not applicable if the data is coming from a ServiceCallout response. Is there any other option to enable streaming for the payload coming from the Service Callout response ??

We are using Apigee X.

Thanks in Advance!

dchiesa1

We have a target server which returns the URL for downloading the file and we then use a ServiceCallout Policy to download the files from the URL and send that file data as a response from our proxy, and there are usecases where our file sizes can be greater than 10 MB.

That seems the wrong way around. If your logic requires the Apigee proxy to first call one endpoint to get the second endpoint and then to call that second endpoint and relay the response to the original client, then... the second endpoint should be the target. You should use Servicecallout for the first endpoint, which tells you which target URL to use. You can set the variable target.url in the target request flow to specify a dynamic target, eg, something you retrieve from endpoint1.

I see we have an option to enable streaming to support payloads more than 10 MB,

where? What you describe is not supported. The documentation page that describes limits for Apigee X specifically states that the size limit on the request and response payload is 10MB. At the moment the size limit is not enforced on non-streamed requests, but the plan is to enforce that. It is a poor idea to try to transmit payloads larger than 10mb through the Apigee proxy.

Is there any other option to enable streaming for the payload coming from the Service Callout response ??

This is the wrong question to ask. The right question is, how can I use Apigee in a scenario in which the response payload is larger than 10mb. And the correct answer to that is: use two channels: a control channel and a data channel. Use something like Google Cloud storage for the data channel - you can download a 1gb file if you like from CLoud Storage. And use Apigee to create a signedURL, or some other authenticated one-time-use URL, to allow the download.

Search this community site for more details on this approach. Here you can find an in-depth screencast walkthrough of the idea.

View solution in original post

dchiesa1

We have a target server which returns the URL for downloading the file and we then use a ServiceCallout Policy to download the files from the URL and send that file data as a response from our proxy, and there are usecases where our file sizes can be greater than 10 MB.

That seems the wrong way around. If your logic requires the Apigee proxy to first call one endpoint to get the second endpoint and then to call that second endpoint and relay the response to the original client, then... the second endpoint should be the target. You should use Servicecallout for the first endpoint, which tells you which target URL to use. You can set the variable target.url in the target request flow to specify a dynamic target, eg, something you retrieve from endpoint1.

I see we have an option to enable streaming to support payloads more than 10 MB,

where? What you describe is not supported. The documentation page that describes limits for Apigee X specifically states that the size limit on the request and response payload is 10MB. At the moment the size limit is not enforced on non-streamed requests, but the plan is to enforce that. It is a poor idea to try to transmit payloads larger than 10mb through the Apigee proxy.

Is there any other option to enable streaming for the payload coming from the Service Callout response ??

This is the wrong question to ask. The right question is, how can I use Apigee in a scenario in which the response payload is larger than 10mb. And the correct answer to that is: use two channels: a control channel and a data channel. Use something like Google Cloud storage for the data channel - you can download a 1gb file if you like from CLoud Storage. And use Apigee to create a signedURL, or some other authenticated one-time-use URL, to allow the download.

Search this community site for more details on this approach. Here you can find an in-depth screencast walkthrough of the idea.

Tri-Thong

Hi Dino,

Please let me chime in on this post as I have the same need: Client --> Apigee API proxy (includes a ServiceCallout to get a binary file from another server which may exceed 10MB of size) --> target server.

I understand that as @dchiesa1 said we should leverage Apigee to generate a signed URL for the GCS object, then let the target server use that signed URL to download the object.

However, due to the RnR complexity in the organization, it is hard to allow the target server to download objects from GCS. So that we would like to pass the returned binary file from the ServiceCallout to the target server. Do you have any ideas to achieve this goal?

dchiesa1

Apigee is not a good place to host things that are transmitting large payloads. If I were doing this I would follow my prior advice

@dchiesa1 wrote:

The right question is, how can I use Apigee in a scenario in which the response payload is larger than 10mb. And the correct answer to that is: use two channels: a control channel and a data channel. Use something like Google Cloud storage for the data channel - you can download a 1gb file if you like from CLoud Storage. And use Apigee to create a signedURL, or some other authenticated one-time-use URL, to allow the download.

And if that won't work for you because of other constraints, then ... write a new service, host it in Cloud Run, and let that service be the thing that handles payloads over 10mb. The key thing is, you should not be transmitting those payloads through Apigee.

Good luck.