No Active Target server Found

karlapudiravi · 02-22-2019 01:54 AM

Hi,

I am seeing "No Active Target server Found" for few of the requests on the message processor logs, any clue on why it is happening only for few requests?

robert

are you seeing any other errors there? Like timeouts? For instance, 504 Gateway Timeout? or maybe a 503?

Also, it might be that you have a misconfigured target server. I've seen this before when someone inadvertantly put 'http' or 'https' into the target server configuration. It should be host name only there. So check those target servers as well.

karlapudiravi

Hi @Robert Johnson, the target server configurations has host name alone. I see connection timeouts are happening with the backends.

karlapudiravi

@Robert Johnson i see this is happening with one of the message processor

latheefdudakula

@ ravi kumar karlapudi

This error indicates that the message processor is unable to forward the request to any of the target connections.

If you have configured the target connections with accurate configuration then the backend system(target app) is down or not responding.

If the target connection is public you can share the configuration so that I can also look what is different in the configuration.

Thanks,

Latheerf D

oleksandrmartyn

@Latheef I added comment under it, with full picture what we have.

oleksandrmartyn

@Latheef at general picture looks like this: first of all in message-processors logs we can see:

1) <stage 1>

2019-03-01 08:25:02,177 org:<name> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-18  NIOThread@7 WARN  c.a.p.h.d.DNSCachedAddress - DNSCachedAddress.addressNotReachable() : The last address has been removed from Address list <***target_host***>/<***target_ip***> refreshing

2) then: <stage 2>

2019-03-01 08:25:02,179 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-18  NIOThread@7 ERROR HTTP.CLIENT - HTTPClient$Context.onConnectFailure() :  connect to <***target_host***>/<***target_ip***> failed with exception {}
2019-03-01 08:25:02,182 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-1-185  NIOThread@0 INFO  HTTP.SERVICE - ExceptionHandler.handleException() : Exception java.io.IOException: Broken pipe occurred while writing to channel GZIPOutputChannel(ChunkedOutputChannel(ClientOutputChannel(ClientChannel[A:10.1.1.185:8998 Remote host:10.1.1.185:2254] useCount=1 bytesRead=0 bytesWritten=203749 age=74413ms  lastIO=0ms  lastIO=0ms  isOpen=true)))
2019-03-01 08:25:02,182 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-3-225 NIOThread@4 WARN  c.a.p.h.d.DNSCachedAddress - DNSCachedAddress.addressNotReachable() : The last address has been removed from Address list <***target_host***>/<***target_ip***> refreshing
2019-03-01 08:25:02,186 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-18  NIOThread@7 INFO  ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Failed to send request to target servers : [***<server_name***>] for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>}
2019-03-01 08:25:02,187 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-3-225  NIOThread@4 ERROR HTTP.CLIENT - HTTPClient$Context.onConnectFailure() :  connect to <***target_host***>/<***target_ip***> failed with exception {}

2) and then: <stage 3>

2019-03-01 08:31:53,881 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-200 NIOThread@2 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Unexpected error while sending request 
2019-03-01 08:31:53,890  NIOThread@2 INFO  ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Failed to send request to target servers : [***<server_name***>] for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>}
2019-03-01 08:31:53,890  NIOThread@2 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : No Active Target server Found for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>}
2019-03-01 08:31:53,893  NIOThread@1 INFO  ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : Failed to send request to target servers : [***<server_name***>] for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>}
2019-03-01 08:31:53,893  NIOThread@1 ERROR ADAPTORS.HTTP.FLOW - LBTargetRequestSender.sendRequest() : No Active Target server Found for <***>{Organization=<***name***>Environment=<***env_name***>,<***app_name***>}

between <stage 2> and and <stage 3> and in further logs, "as example" we can see the next:

<stage 4>:

2019-03-01 08:01:48,974 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-35  NIOThread@1 INFO  HTTP.SERVICE - ExceptionHandler.handleException() : Exception java.io.IOException: Broken pipe occurred while writing to channel GZIPOutputChannel(ChunkedOutputChannel(ClientOutputChannel(ClientChannel[A:10.1.1.185:8998 Remote host:10.1.5.35:30148] useCount=1 bytesRead=0 bytesWritten=650450 age=770156ms  lastIO=0ms  lastIO=0ms  isOpen=true)))
2019-03-01 08:01:48,977 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-189  NIOThread@5 INFO  HTTP.SERVICE - ExceptionHandler.handleException() : Exception trace:
2019-03-01 08:01:48,977 org:<***name***> env:<***env_name***> api:<***name_of_api_proxy***> rev:39 messageid:ip-10-1-5-189  NIOThread@5 INFO  HTTP.SERVICE - ExceptionHandler.handleException() : Closing the channel TrackingInputChannel(FixedLengthInputChannel(ClientInputChannel(SSLClientChannel[C:<***target_ip***>:<***port***> Remote host:10.1.1.185:32350] useCount=84 bytesRead=0 bytesWritten=601 age=470344ms  lastIO=9ms  lastIO=9ms  isOpen=true)))

and after them again repeats <stage 1>, <stage 2> , <stage 3> and between them repeats <stage 4>.

Is any ideas, thoughts, suggestions?

Also I wonder what means age="value". How you can explain what means this value and is it normal in our case?

Thanks in advance.

+ @Robert Johnson

and anybody who can clarify.

adevegowda

@Alexander Martynenko

In stage 1, the DNS refresh is in progress, but not sure if it got succeeded before stage 2.

In stage 2, we see connection failure with the target server. This could have failed either due to

timeout - if the target server does not respond to Message Processor within 3 seconds, then it fails
connection refused exception - target server refused the connection, possibly because it is too busy and unable to respond to some of the requests

The details about the reason for connection failure are missing from the log that you attached, so unable to find out why the connection with the target server failed.

Stage 3 says that Edge could not send the request to the target server. This is because of the connection failure that occurred in Stage 2.

Stage 4 seems to say that the MP tried to write the response to router, but the router had possibly closed the connection, so we get a Broken pipe exception. We need to check the Nginx access logs and whats the actual response that's sent to client and try to understand if it had closed the connection with MP.

The "age" indicates how the long the connection has been kept alive or being reused.

Can you check the following

1. Why did the connection fail ? I mean check if there are any further errors or exceptions in the MP log.

2. Try connecting to the target server using telnet (telnet <target server ip> <port #>) before and at the time of failure and see if it works.

3. Is the target server IP modified during this period by any chance ?

4. You could also take a tcpdump and analyze and get more details about the connection failure.

If you need any further help, please raise a ticket with Apigee Supoort.