Hi,
I am seeing "No Active Target server Found" for few of the requests on the message processor logs, any clue on why it is happening only for few requests?
are you seeing any other errors there? Like timeouts? For instance, 504 Gateway Timeout? or maybe a 503?
Also, it might be that you have a misconfigured target server. I've seen this before when someone inadvertantly put 'http' or 'https' into the target server configuration. It should be host name only there. So check those target servers as well.
Hi @Robert Johnson, the target server configurations has host name alone. I see connection timeouts are happening with the backends.
@Robert Johnson i see this is happening with one of the message processor
This error indicates that the message processor is unable to forward the request to any of the target connections.
If you have configured the target connections with accurate configuration then the backend system(target app) is down or not responding.
If the target connection is public you can share the configuration so that I can also look what is different in the configuration.
Thanks,
Latheerf D
@Latheef I added comment under it, with full picture what we have.
@Latheef at general picture looks like this: first of all in message-processors logs we can see:
1) <stage 1>
2019-03-01 08:25:02,177 org:<name> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-18 NIOThread@7 WARN c.a.p.h.d.DNSCachedAddress - DNSCachedAddress.addressNotReachable() : The last address has been removed from Address list <***target_host***>/<***target_ip***> refreshing
2) then: <stage 2>
2019-03-01 08:25:02,179 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-18 NIOThread@7 ERROR HTTP.CLIENT - HTTPClient$Context.onConnectFailure() : connect to <***target_host***>/<***target_ip***> failed with exception {} 2019-03-01 08:25:02,182 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-1-185 NIOThread@0 INFO HTTP.SERVICE - ExceptionHandler.handleException() : Exception java.io.IOException: Broken pipe occurred while writing to channel GZIPOutputChannel(ChunkedOutputChannel(ClientOutputChannel(ClientChannel[A:10.1.1.185:8998 Remote host:10.1.1.185:2254] useCount=1 bytesRead=0 bytesWritten=203749 age=74413ms lastIO=0ms lastIO=0ms isOpen=true))) 2019-03-01 08:25:02,182 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-3-225 NIOThread@4 WARN c.a.p.h.d.DNSCachedAddress - DNSCachedAddress.addressNotReachable() : The last address has been removed from Address list <***target_host***>/<***target_ip***> refreshing 2019-03-01 08:25:02,186 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-18 NIOThread@7 INFO ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Failed to send request to target servers : [***<server_name***>] for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>} 2019-03-01 08:25:02,187 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-3-225 NIOThread@4 ERROR HTTP.CLIENT - HTTPClient$Context.onConnectFailure() : connect to <***target_host***>/<***target_ip***> failed with exception {}
2) and then: <stage 3>
2019-03-01 08:31:53,881 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-200 NIOThread@2 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Unexpected error while sending request 2019-03-01 08:31:53,890 NIOThread@2 INFO ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Failed to send request to target servers : [***<server_name***>] for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>} 2019-03-01 08:31:53,890 NIOThread@2 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : No Active Target server Found for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>} 2019-03-01 08:31:53,893 NIOThread@1 INFO ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Failed to send request to target servers : [***<server_name***>] for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>} 2019-03-01 08:31:53,893 NIOThread@1 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : No Active Target server Found for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>}
between <stage 2> and and <stage 3> and in further logs, "as example" we can see the next:
<stage 4>:
2019-03-01 08:01:48,974 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-35 NIOThread@1 INFO HTTP.SERVICE - ExceptionHandler.handleException() : Exception java.io.IOException: Broken pipe occurred while writing to channel GZIPOutputChannel(ChunkedOutputChannel(ClientOutputChannel(ClientChannel[A:10.1.1.185:8998 Remote host:10.1.5.35:30148] useCount=1 bytesRead=0 bytesWritten=650450 age=770156ms lastIO=0ms lastIO=0ms isOpen=true))) 2019-03-01 08:01:48,977 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-189 NIOThread@5 INFO HTTP.SERVICE - ExceptionHandler.handleException() : Exception trace: 2019-03-01 08:01:48,977 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-189 NIOThread@5 INFO HTTP.SERVICE - ExceptionHandler.handleException() : Closing the channel TrackingInputChannel(FixedLengthInputChannel(ClientInputChannel(SSLClientChannel[C:<***target_ip***>:<***port***> Remote host:10.1.1.185:32350] useCount=84 bytesRead=0 bytesWritten=601 age=470344ms lastIO=9ms lastIO=9ms isOpen=true)))
and after them again repeats <stage 1>, <stage 2> , <stage 3> and between them repeats <stage 4>.
Is any ideas, thoughts, suggestions?
Also I wonder what means age="value". How you can explain what means this value and is it normal in our case?
Thanks in advance.
and anybody who can clarify.
In stage 1, the DNS refresh is in progress, but not sure if it got succeeded before stage 2.
In stage 2, we see connection failure with the target server. This could have failed either due to
The details about the reason for connection failure are missing from the log that you attached, so unable to find out why the connection with the target server failed.
Stage 3 says that Edge could not send the request to the target server. This is because of the connection failure that occurred in Stage 2.
Stage 4 seems to say that the MP tried to write the response to router, but the router had possibly closed the connection, so we get a Broken pipe exception. We need to check the Nginx access logs and whats the actual response that's sent to client and try to understand if it had closed the connection with MP.
The "age" indicates how the long the connection has been kept alive or being reused.
Can you check the following
1. Why did the connection fail ? I mean check if there are any further errors or exceptions in the MP log.
2. Try connecting to the target server using telnet (telnet <target server ip> <port #>) before and at the time of failure and see if it works.
3. Is the target server IP modified during this period by any chance ?
4. You could also take a tcpdump and analyze and get more details about the connection failure.
If you need any further help, please raise a ticket with Apigee Supoort.
User | Count |
---|---|
2 | |
1 | |
1 | |
1 | |
1 |