Lost connection within Apigee

[yves_scotto]

I deployed a proxy which manage a CallOut (request 1) and a target (request 2). On 3000 requests, I encountered between 3 and 25 lost requests. So the end-users receives an HTTP error 504 due to the 50 sec timeout on HAProxy in front of my 2 Message processors.

I set the following properties in the Callout policy and in the target

<Property name="keepalive.timeout.millis">60000</Property>

<Property name="connect.timeout.millis">10000</Property>

But it's alway the 50 sec HAproxy timeout which raise the 504 error

These requests seems to be... lost 😞

If I proxify my 2 requests (CallOut and Target) with a NGINX server behind message processor servers , the issue is fixed !!!!!

I checked router logs and I only notice the warning :

WARN CONNECTION-REAPER - ConnectionReaper$ReaperTask.safeRun() : Client-Connection-Reaper reaped inactive client connections

How can I check connection timeout within Apigee?

Thank you

[adevegowda]

Dear @Yves SCOTTO,

1. If I understood your setup correctly it is as follows:

HAProxy (Load Balancer) -> Router -> Message Processor -> Target Server

In this setup, you make a Service Callout and then send the request to the Target server via an API Proxy. During the process, you get 504 Errors because of the 50 sec timeout set in HAProxy.

2. If this is right, then I suspect you are seeing 504 Error either because the Service Callout is taking longer time to respond or the target server is taking longer time to respond to your request. The elapsed time for these two activities is controlled by the IO timeout period set in the components in your setup shown above. In some Load Balancers this timeout period is also referred to as Idle timeout period.

3. In the case of Routers/Message Processors this timeout period is controlled through a property <code>HTTPTransport.io.timeout.millis</code>. You can find this property in the following file:

Router -> /opt/apigee/router/conf/http.properties

Message Processor -> /opt/apigee/messageprocessor/conf/http.properties

Thumb rule is that you need to ensure the timeout values on the servers are always in this order:

• Load Balancer (HAProxy timeoout > Router timeout > MP timeout

You can refer to the timeout value set in your Nginx server (Usual Location: /opt/apigee/nginx/conf.d/0-default.conf) and set that value for the timeout period in the HAProxy. Also, ensure that the timeout value in HAProxy is at least 1-2 seconds higher than that set in the Router. Similarly the timeout value in Router is at least 1-2 seconds higher than in the Message Processor.

Hope that helps.

Regards,

Amar

[yves_scotto]

Hello Amar

First thank you for your answer, unfortunately timeout changes didn't change anything

My flow is:

End user => HA Proxy (timeout 50 sec) => Router (timeout 40 sec) => Messageprocessor (timeout 30 sec) => target

I set the 2 timeout in http.properties

• timeout to connect => HTTPClient.connect.timeout.millis=30000

• timeout to read or write data => HTTPTransport.io.timeout.millis=30000

And it's always the HA proxy time out of 50 sec which is raised !

On a loop of 3000 requests, I have more than 20 HTTP 504 errors of 50 sec on end-user side.

During a lock, I sent curl requests from one MP server to my target and all is fine !

But the more curious is, if I add a Nginx server in the flow

End user => HA Proxy (timeout 50 sec) => Router (timeout 40 sec) => Messageprocessor (timeout 30 sec) => Nginx => target

I have no time out!!

The Nginx is installed on one of my MP servers.

So, I think that the issue is within Apigee but I can’t understand why some requests disappear. My only clue is some WARN in the router log:

WARNCONNECTION-REAPER - ConnectionReaper$ReaperTask.safeRun() : Client-Connection-Reaper reaped inactive client connections

Any idea?

Thank you for your answer

[yves_scotto]

Hi all

For your information, to avoid HTTP error 504, there is a workaround

The issue could be fixed by setting

HTTPClient.keepalive.timeout.millis=0 in http.properties files

I don't understand why but it works...

By default this properties is in comment so, it's tha caller which decide to manage or not keep alive.

in my case the caller always send Connection: Close...

I open a case on Apigee support...