Proxy sporadically hangs when sending request to backend and returns a 504

[#1]

We have a problem where the Apigee proxy is apparantly hanging on the step where it sends the request to the backend. It happens sporadically, and seems to happen more frequently when the throughout is slow.

See screenshot. When the proxy hangs, it gets to the flow right before it sends to the backend, but the policy/step indicator that it sent to the backend never shows..

NOTE that I am testing this with JMeter, sending the same transactions over and over again.

when I run this wide open, hundreds of transactions as fast as I can, the error rate is considerably lower than if I run them slower.

This one never comes back and eventually a 504 is returned to the customer.

This is what it looks like when the same exact transaction is sent to the exact same backend and it works

we have the backend timeout settings overridden with 5 minute timeouts.

<HTTPProxyConnection> <BasePath>/v2/eligibility</BasePath> <VirtualHost>default</VirtualHost> <Properties> <Property name="io.timeout.millis">300000</Property> <Property name="keepalive.timeout.millis">300000</Property> <Property name="connect.timeout.millis">300000</Property> <Property name="target.copy.pathsuffix">false</Property> </Properties> </HTTPProxyConnection>

I also set up another test in JMeter which sends the Soap Message (which would be created by Apigee using JSON input) directly to the backend server... and I have not received a 504 response... as a matter of fact, errors are rare.

We are at a loss.. please advise.

[dknezic]

Why have you set 5 minute timeouts?

When you test your backend don't just look at the statuses but please review the response times from your backend

Based on the response times from your backend, you may need to use a tool that's capable of maintaining concurrent load on your backend to actually test it properly.

[#3]

We set it to 5 minutes as overkill for another timeout issue we were having. The default values were too low... so yes, we could probably back that down to 30K ms (30 seconds) as a more realistic calue.

The response times for the direct SOAP backend hit are always less than 3000 milliseconds for successful

I have tested various throughput volumes when testing this proxy. I have tested 1 transaction every two minutes, 4 per minute, and as many as 4 threads can put through the system at any given time (wide open, no restraints, 4 threads, 2 transactions per iteration). I have run this for hours at a time... but through the proxy and directly to the backend, and the results are consistent.

When the proxy hangs, it gets to the point of sending the request to the backend, but the indicator on the trace never shows... and the proxy never completes. It just sits there at that point and never returns, as I showed in the screenshots.

It seems like the slower the throughput, the more often the proxy hangs at the point it is sending to the backend.

we have checked TCP dumps and the ones that are hanging at the "send" location are not making it to the backend at all.

what else can we check?