In Apigee, 2 transactions are happening for a single request in case of 503 error from downstream

[Sivaguru]

Hi Team,

We are observing an unexpected behaviour of Apigee, When we are receiving 503 Service unavailable from downstream, there is another transaction with same requestID.

In the above case there are 2 transactions at Apigee for the same request, but we hit the request only once from postman. And we also tried form browser and jmeter, we are getting the same two transactions for a single request.  So, what is the reason behind two transactions happening at Apigee for a single request.

Thanks,

Sivaguru J.

@kurtkanaskie  @dchiesa1  @dknezic  @shrenikkumar-s  @Peeyush_Singhai @Renuka_atnoor 

[dchiesa1]

Do you have a loadbalancer configured in the TargetEndpoint?  and Does it perform Retry?  You could get multiple requests sent from Apigee to the target, if you have a loadbalancer and if Retry is not disabled.  

[Sivaguru]

Hi @dchiesa1 
We dont have  a loadbalacer and retry is also not enabled. Please find an image in which there are two transaction for a single request.

[dknezic]

What are the timestamps of the requests? Review the request headers, are there additional stops before apigee eg indicated by x-forwarded-for header?

[Sivaguru]

Hi @dknezic,

Both requests are having the same timestamp only.

When comparing both the requests only 2 headers are different.

Request 1:

x-request-id 398a08dc-2d2e-4f67-a1c1-e503afc8521b

x-b3-traceid cfa5948e75490f7deed1a4465601ccf0

Request 2 : 

x-request-id b9748cd0-e7d4-468c-99b8-abd9a9741a3c

x-b3-traceid db37fe19fc7016ec4fa124cab6c18d05

All other headers are same.

[Sivaguru]

By extending our analysis, 2 transactions are happening for a single request in case of 503 error from downstream only if the request methods are GET or HEAD

[shrenikkumar-s · Accepted solution]

We've an answer on this after working with Google:

Here is a brief summary of the case:
You inquired about some retries appearing on your debug sessions when only one request was sent which caused an impact on your testing environment.

During the investigation we went through:
- Testing and reproducing your situation.
- Consulting with an Apigee expert to address if the behaviour was expected.
- Finding out this behaviour is an expected behaviour of the External Global Application Load Balancer.

We concluded that the pertinent documentation to address your question would be:
https://cloud.google.com/load-balancing/docs/https#:~:text=Retry%20logic-,Global%20external%20Applic...

As it explains on the documentation for the load balancer this behaviour happens when there is no retry policy enabled, so the solution to it, is overriding the policy with one with no amount of retries.

To disable:
I have found out the following solution to disable the retry behaviour:
1. Get the load balancer ID from the following command, the “id”. [1]
2. With this information we can make a patch to the following endpoint with a body that will override the retry policy in order to disable it. [2] [3]

By performing these actions the debug session should no longer show automatic retries from the load balancer.

[1] https://cloud.google.com/compute/docs/reference/rest/beta/urlMaps/list
[2] https://cloud.google.com/compute/docs/reference/rest/beta/urlMaps/patch
[3]
{
  "defaultRouteAction": {
    "retryPolicy": {
      "numRetries": 1
    }
  }
}