Streaming of large files with microgateway

galo
Participant I

Microgateway does not seem use the Node “streaming” interface but rather processes each HTTP chunk in Javascript VM memory. Are there any recommendations for using microgateway for REST endpoints that receive or access large binary files ?

Solved Solved
0 6 1,373
1 ACCEPTED SOLUTION

Not applicable

we use the node streaming interface for http. its just abstracted into a middleware like callback system. So if you have a on request_data handler you'll effectively be using http streaming.

https://github.com/apigee/microgateway-core/blob/master/lib/plugins-middleware.js#L287

View solution in original post

6 REPLIES 6

Not applicable

we use the node streaming interface for http. its just abstracted into a middleware like callback system. So if you have a on request_data handler you'll effectively be using http streaming.

https://github.com/apigee/microgateway-core/blob/master/lib/plugins-middleware.js#L287

prabhat
Participant V
@galo

can you pls confirm that it works for you? It's important so that others know that they can use Microgateway for streaming applications too.

galo
Participant I

Thanks. We will repeat the tests with 2.0 and publish here the results and the code/config used..

What did you find here? I've done some load tests recently and it still appears like microgateway 2.0 is NOT streaming the API response.

We found an issue on our testing strategy and ran tests with 2.0. that we fixed. We are a ~20% penalty on the files which are moderate/small (5Mb)

That was with Node 4.4 and 6.3.1 and did not observe much difference.

No EMG

3468-no-emg-aggregate.png

EMG2.0 + Node 4

3470-emg2-aggregate.png

EMG2.0 + Node 6

3471-emg2-node6-aggregate.png

Not applicable

I'm keen to find out whether streaming is at all possible with microgateway. edgemicro is currently a CPU and memory hog on our servers and we need to somehow make it stream the API responses rather than pulling them into memory before responding. It's returning chunked transfer encoding of API responses so there doesn't seem to be any reason that it couldn't be streaming out the API response.

We've tried following the information on http://docs.apigee.com/api-services/content/enabling-streaming but that doesn't seem to have any effect for edgemicro.

Is there some configuration that we haven't found to enable streaming for microgateway as well?