Apigee Edge Cloud: Intermittent 503 "Service Unavailable" instead of 413 "Entity too large" when request payloads > 1MB with Broken pipe

[hkedjour]

We have endpoint that checks the content length of the request body. If it's bigger than 512KB, it returns 413 Payload Too Large with a nice formatted error for the user:

{
    "validation_errors": [
        {
            "name": "Image too large",
            "description": "Content-Length header indicated that the upload stream is too long."
        }
    ],
    "title": "Request Validation Error",
    "status": 413,
    "detail": "See ValidationErrors for details",
    "instance": "xxxxx"
}

It works perfectly. Until someone send a payload bigger than 1MB. With bigger loads, Apigee returns most of the time 503 Service Unavailable. With the following body:

{"fault":{"faultstring":"The Service is temporarily unavailable","detail":{"errorcode":"messaging.adaptors.http.flow.ServiceUnavailable"}}}

And sometimes returns the expected 413.

It seems that Apigee don't try to call the backend.

The error is Broken pipe

Sometimes is a little bit different:

I tried also to use streaming as described in https://docs.apigee.com/api-platform/develop/enabling-streaming . The result stayed the same.

What could be the issue?

Thanks.

[API-Evangelist]

Few ways to address any 503 issues which we generally do to identify issues..

1.Enable debug on mp and see debug logs

2. tcp dump on router/mp servers to identify if it is an apigee issue or backend server.

I can't imagine for 512k it has issues..Check below and modify and see if it makes any difference..

https://docs.apigee.com/private-cloud/v4.19.01/set-message-size-limit-router-or-message-processor

Also make sure you setup streaming in response if it is form backend server..

[imesh]

Hichem, On Apigee Edge, this problem can occur either at the incoming (northbound) or outgoing (southbound) connection. Most often, the error occurs because a server is too busy or is down for some reason, such as for temporarily maintenance:

https://docs.apigee.com/api-platform/troubleshoot/runtime/503-service-unavailable

In this specific scenario as you have mentioned this issue may have occurred due to the large payload size (> 1MB) and high throughput (TPS). By this occurrence, either the target servers or Apigee Message Processors of this Edge Cloud environment may have been overloaded.

You should be able to verify this by doing another test by reducing payload size and throughput for identifying the breaking point. Afterwards, check health of target servers and Apigee Message Processors during each test.

To resolve this problem we may need to do one of the following:

• Use streaming and avoid processing the message payload in API proxies
• Increase capacity of target and Apigee servers

As you have mentioned that you tried streaming, most likely the issue might be with the server capacity. For verifying Edge Cloud Message Processor health you may need to contact Apigee Support.