504 gateway timeout between edg-router and edge-message-processor

[yves_scotto]

Hi all

I'm making some stress test on a private Apigee Edge configuration (version 4.17.09).

I have 4 message processor servers 4 VCPU and 16 Go RAM

I allocated 8 Go RAM to run (

-Xms8192m -Xmx8192m -XX:MaxPermSize=512m -XX:MetaspaceSize=512m

)

I generate a load of 2800 TPS and my backend servers reply in less than 200 ms

All seems fine but, I noticed some burst of 504 gateway time out in the edge-router nginx logs.

I have no error on backend side and no 504 gateway time out in edge-message-processor logs

Timeout are set to 30000 ms for edge-message-processor and 40000 ms for edge-router.

I only noticed this error log in garbage collector

[GC (Allocation Failure) [PSYoungGen: 2490389K->37932K(2620416K)] 3179721K->745763K(8212992K), 0.0887289 secs] [Times: user=0.28 sys=0.02, real=0.09 secs]

Other wise the system-monitor.log files indicates

TimeStamp = 2/13/18 6:03 PM ; NextDelay = 112
Memory
HeapMemoryUsage = {used = 1510842056}{max = 8410103808} ; NonHeapMemoryUsage = {init = 2555904}{committed = 265289728} ;
Threading
PeakThreadCount = 301 ; ThreadCount = 293 ;
OperatingSystem
SystemLoadAverage = 6.97 ;

What can generates 504 gateway time-out within Apigee Edge ?

Thank you for your feedback

Yves

[latheefdudakula]

Hi Yves,

How often are you seeing this timeout in logs ?

One of the possibility might the request might have timed-out at the gateway/router component.

Like , the request timeout might be set as 30 seconds on gateway/router , the time took to process the request to back-end or in the message processor might have taken more than 30 seconds . In this scenario the request might have time-out in gateway/router .

So to avoid this , we need to set the timeout bit differently,

loadbalancer_timeout > apigee_router_timeout > apigee_mp_timeout > backend_timeout

Hope this helps!!.

Thanks

[yves_scotto]

Hello APID

On more than 21 M requests during the load test , I had arround 100K 504 timeout errors at nginx edge-router level.

As I said, there is no 504 errors at message-processor level.

I think that if my backend doesn't respond, I will have a 30 sec 504 error at edge-message-processor level ? And it is not the case !!

But maybe I didn't check the right logs on edge-message-processor?

Yves

[yassertaeima]

@Yves SCOTTO

Were you able to identify the 504 from MP?

We are seeing the same, but the timeout looks like a connect timeout between R and MP because it occurs at 5s not 30

2018-10-27T13:10:37+03:00 apigee-mpt x.x.x.x:40794 x.x.x.x:8443 5.001 - - 504 504 324 344 GET /loadTest HTTP/1.1 - xyz xyz apigee-mpt-17277-700362-29 client.ip.x.x - - - - - - - -1 - - dc-1 testgateway green