Getting 5XX error as intermittent proxies are failing in production

snehansu
Participant III

Hi , we are getting 5XX error mostly all the apis intermittently in production last few months. please suggest how to debug and resolve this issue. Log: NIOThread@0 ERROR HTTP.CLIENT - HTTPClient$Context$3.onException() : SSLClientChannel[Connected: Remote:XXXXXX Local:XXXXXXX]@XXXXX useCount=20 bytesRead=0 bytesWritten=843 age=208453ms lastIO=16ms isOpen=false.onExceptionRead exception: {}\njava.io.EOFException: eof unexpected\n\tat com.apigee.nio.channels.PatternInputChannel.doRead(PatternInputChannel.java:45)\n\tat com.apigee.nio.channels.InputChannel.read(InputChannel.java:103)\n\tat com.apigee.protocol.http.io.MessageReader.onRead(MessageReader.java:79)\n\tat com.apigee.nio.channels.DefaultNIOSupport$DefaultIOChannelHandler.onIO(NIOSupport.java:51)\n\tat com.apigee.nio.handlers.NIOThread.run(NIOThread.java:220)"

0 3 438
3 REPLIES 3

According to the error it seems like Apigee Message Processor has received a EOF while waiting a response from the target server. If this is happening intermittently, might need to take a tcpdump and analyze the message flow:

Please refer the following troubleshooting playbook for details:

https://docs.apigee.com/api-platform/troubleshoot/runtime/502-bad-gateway#eofexception-from-the-back...

Not applicable

bytesRead=0 bytesWritten=843 age=208453ms lastIO=16ms isOpen=false.onExceptionRead exception. Here you can see the age is is too high. This means the process is taking longer, byteWritten is nonzero value that means the request is going to backend. byte read is zero means the response is not coming from backend. The reason looks like backend is taking much time to send the response back and by that time the message processor is closing the socket. As its interimittent, it could be because of backend getting slower sometimes else the network is slow sometimes which is eding with EOF error.

Are you whitelisting the Apigee IP addresses? We did and when they changed last week we started getting 503 errors