Apigee Edge & microgateway combo returning intermittant 502

etienned
Participant III

Hi,

So I have Apigee edge connecting to the Microgateway via a Fortinet Loadbalancer and then passing to the backend. The reason for this is the client has a DMZ requirement and also local routing requirements.

I have the microgateway deployed in my own version of a Docker image(I had to create a Windows version of the "Apigee supported Docker deployment"(Linux), since I only had access to machines with Windows Server - Windows Server can't run Linux containers) The docker image works as expected, the Microgateway config has SSL enabled and has requestCert enabled. I have tested it locally on my laptop at a TPS of 300 for a duration of an hour and it does not return a single error (Doing Healthcheck while sending the certs).

I have deployed the docker image, and is executing transactions at about 10 TPS from my Laptop, to Apigee Edge, that will pass it onto the Microgateway via HTTPS. After about 15minutes the first 502 will be returned from Apigee Edge. This will happen for about 0.5% of the transactions.

I know according to best practices the Servers keepAliveTime should be more than Apigee Edge to avoid 502's. Due to server closing the connection and Edge then trying to make an API call.

And from my previous question, it looks like the Microgateway does not have a keepAliveTimeout set, so this should not happen.

Previous Question:

edgemicro: Increasing server.keepAliveTimeout value

The Loadbalancer has a default Idle Timeout of 300seconds. The Apigee Target Server has an default/configured timeout of 60seconds so Apigee will close the connection before the Loadbalancer does.

Do you have any ideas why this is happening and what I can try to solve this?

This was what was received back from a support ticket which we logged.

AbstractResponseListener.onError(HTTPResponse@xxx,
eof unexpected)

SSLClientChannel[xxx] useCount=7 bytesRead=0 bytesWritten=392 age=24244ms 
lastIO=8ms  isOpen=false.onExceptionRead exception: {}

This indicates that the connection was open for 24s, we re-used the connection 7 times, however we receive an EOF Exception. It is recommended, that the target keeps persistent connections open for greater than the keepalive timeout set on Apigee's end.
0 2 319
2 REPLIES 2

Have you checked your Fortinet logs? Depending on how you've got Fortinet set up it may well be doing inspection of the traffic contents, seeing something it doesn't like and closing the connection. Edge can't tell the difference between the Fortinet closing the connection and the Microgateway closing the connection, so the logs won't be able to tell you which one is doing it.

@Etienne Dippenaar Did you ever discover the root cause of this issue?