Intermittent high latency on Apigee X proxy when load testing

[mmagtalas]

Hello.

We are experiencing latency when we are doing load testing on our proxy in Apigee X. As we check on the trace, the highest elapsed time happens on the target request flow.

The target server we are trying to establish a connection with is in a ROSA cluster and we are on a 2-way SSL authentication. 

May I ask for possible causes of this latency? And  just to add as well, the flow of our traffic is from gcp to an on-prem data center and then the on-prem data center to AWS (target server), so we are thinking that this network hops can also contribute to it. 

Thanks.

[dchiesa1]

You may need to connect with Apigee support in order to troubleshoot such latency.  I agree that the network hops may contribute. You may be able to diagnose this further if you have access to logs on either end of the connection. 

[dknezic]

Possible causes could include

- Your target backend. I would suggest load testing your target in isolation first before testing via Apigee

- Looks like you have multiple javascript policies on the response flow. Depending on what these are doing, this could also be adding an overhead

- If you have threat protection policies, presumably these are scanning the payload.. This can be an intensive operation, in particular the RegexProtection policy (you can refer here). This will magnify with larger payloads and under higher tps.