High response time due to long time taken in "Request sent to target server"

[adevegowda]

We are seeing a high total response time for our API requests. Specifically we see that the time taken in "Request sent to target server" is pretty high while all the other policies are taking very less time (1ms or < 1ms). Here's the information taken for a couple of API requests from the trace in UI:

Request #1 
--------------- 
Time Taken By Phases:
Request sent to target server : 526ms 
Response received from target server : 414ms 
Rest of the Phases : 13ms 
------------------------------------------------------------------- 
Sum of Time Taken by All Phases : 953ms
-------------------------------------------------------------------

Request #2 
--------------- 
Time Taken By Phases:
Request sent to target server : 523ms 
Response received from target server : 573ms 
Rest of the Phases : 8ms 
------------------------------------------------------------------- 
Sum of Time Taken by All Phases : 1104ms 
-------------------------------------------------------------------

Why does the phase "Request sent to target server" take such a long time ?

[adevegowda · Accepted solution]

1. Since the phase "Request sent to target server" was taking longer time, just thought of breaking this phase into the various events :

a) Message Processor to establish the connection with the Backend Server
b) Perform SSL Handshake
c) Message Processor to send the request payload to the Backend Server

2. Initially suspected the request payload may be pretty large. But the request payload was pretty small - just a few KBs.

3. So collected a tcpdump on the Message Processor (MP) and found that for each SYN, ACK, Client Hello messages the MP sends the Backend server took about 100+ ms to acknowledge.

4. Next checked that the Message Processor and Backend Servers were located in different region data centers (MP in US East and Backend Server in South America).

Since the MP and backend server were not located in the same region, it used to take a long time to perform SSL handshake.

The possible options to address this issue were:

1. Use the Backend Server from the same region as the MPs OR
2. Have the MPs in same region as the Backend Server and ensure MPs in a region talk to Backend Servers in the same region, so that we don't experience long response times.