Connection between client and apigee drops during curl request

i have successfully connected my api to apigee and am able to access the api through a curl request pointing to the api proxy. The problem is however that sometimes during the curl request, the client seems to lose connection with apigee because the download speed drops to 0. Because of thebsize of the payload results must be streamed.

I have looked at various things:

Pointing directly to api:

-when i send the curl request directly to the api there are no problems

Pointing to apigee proxy:

-sometimes the request is processed and completed without an error

-The connection from the backend api is intact because I can see it streaming information to apigee

-When all of the connections are intact, when I close the clients request the backend api recognizes its been closed and stops the process

-Once the connection is lost for the client, and download speed goes to 0, if I close the clients request, the backend api doesnt recognize this and continues the process

My guess is that it has to do with the client losing connection to apigee and no longer gets a stream of information. Any information on how to prevent this would be awesome. Thanks

Non related question:

The download speed for my api through apigee is about half the speed if it doesnt go through apigee, is this normal?

0 3 967
3 REPLIES 3

you said "extremely large". how large are the payloads?

How long do the transfers last?

There are timeouts at each end. from client-to-apigee, and from apigee-to-upstream.

Each peer in each connection will have timeouts. Apigee Edge timeouts are probably around 60s. But a regular timeout would be obvious, and it wouldn't cause the behavior you observed which was "transfer slowing to zero, and then connection drop."

If you are using a free trial version of Apigee Edge, and using it to download large streams, the behavior you observe may just be .. what you're gonna get.

Apigee Edge is not designed to be a proxy for large streams of data. It's an API Proxy. APIs are not streams. So your usage may be at odds with the design goals of the service.

The file can range anywhere from 100kb to 1GB. As a result the transfer can last anywhere from 20 seconds to 10 minutes. I am currently using apigee edge development servers.

In terms of the time outs from upstream to client, the pattern is not consistent. For example, a 2 minute request by the client will be completed without an issue but that same request will not be completed and the download speed drops to 0 at what seems like random times during the process. This will continue on the client side indefinitely until the client closes the incomplete request.

On the upstream side, the information is continuously "dumped" or sent to apigee in chunks of data (for example 100 rows at a time).

It is also very peculiar because without any changes to the proxy, client request and upstream api, the request will sometimes work and sometimes not work.

Could this inconsistency be a result of being in dev and be fixed in a UAT or PROD enviornment?

Thank you for your help!

Distributed systems can exhibit frustrating behavior.

But maybe we should not be surprised.

You're sending 1GB over the wires, and ... those wires connect your client to the Apigee-hosted proxy servers (which are shared and multi-tenant, there's no telling how many people are trying the same thing as you), and from there to your upstream, which is ??? I don't know.

Contention anywhere along that path can lead to delays and stops. The Apigee Edge proxy server actually runs on a VM somewhere, and that VM has (probably, I don't know for sure) a single virtual network interface, and that NIC is shared among all the workload carried by that proxy. If you had your own VM and your own process and a fully-open physical 10GB NIC, then I suppose you would see fewer delays. But you don't have that. You have a slice of resources, and sometimes that slice experiences contention.

Anyway the Apigee edge system is not designed to serve 1GB payloads typically. Sure there are some commercial customers who may do that but mostly it's API calls with payloads that are smaller.

So I think my answer is: Don't do that. Or if you do it, then expect glitches.