Apigee Gateway load test

[#1]

We are currently using Layer 7 gateway as a primary gateway. The plan is to migrate all from Layer 7 to Apigee. The infrastructure in apigee is now much bigger than the Layer 7. Still, when we did the performance test Apigee is able to handle hardly 50% of the traffic that Layer 7 can handle. We tried to tune apigee with ulimit, pool value etc. Still, we don't see any improvement. This gives a set back to think the prodcut use.

We also tried to bypass F5 and apigee router, directly did hit the message processor to understand if there is any F5 latency. Still, the TPS remained the same.

Can anyone help on this?

[ylesyuk]

> We tried to tune apigee with ulimit, pool value etc.

considering you mention ulimit and other linux-level variables, you're using OPDK Apigee.


I would start with 2 questions.

Are you comfortable with single request processing time?

Yes, I hear you, Layer 7 smaller topology footprint is twice performant. Still, what's the single request latency?

Apigee is horizontally scalable therefore by adding Routers and MPs you should be able to identify required number of MPs to achieve your performance target.

Another factor is proxy complexity and usage of objects like caches, KVMs, etc. To tune the performance, we need now to identify bottlenecks, understand them, and remove them.

I am aware about customers having 40k+ TPS sustained rate at their OPDK topologies. Therefore, the product has no inherent constraints. But its usage patterns definitely differ.

[#3]

The proxy I am testing is a simple pass-through. The latency increases with the vcpu increase. I understand the topology and apigee vs L7. But, with bigger infra also not getting much difference.

[ylesyuk]

Let me clarify, you're not using JMeter, are you?

[#5]

we are using load runner.

[ncardace]

Make sure you use the latest OPDK 4.50.00 release and a very recent or latest JDK version.

Also, which backend are you testing against ? I recommend you have a cluster able to cope directly (i.e. without the API gateway on front) at least with the same traffic throughput you are testing with. I also recommend to use a modern performance testing tool, for example hey:

https://github.com/rakyll/hey

[#7]

We are using second latest version of Apigee as of now. We have other gateways which were load tested against a set of backends, We are using same backends for the tests.

[ncardace]

Could you be more specific on the actual gateway release version ? i.e. It would be important to know if you using 4.19.06 with or without any patch release ? I am asking as there are 7 minor releases after 4.19.06 and before the latest 4.50.00 (4.19.06.01 to 4.19.06.07).

Some of them specifically improve performance and latency behaviours. So it is important to understand if you are including or excluding these.

[#9]

we are using 4.19.06.0.0.1114, apigee

[ncardace]

Which Private Cloud version are you testing with ?

It's very likely you are bumping in a known issue with one of the previous releases, which was addressed by our engineering.

See here the release notes:
https://docs.apigee.com/release/notes/apigee-release-notes

Also, I would make sure that you are compliant with our guidance on sizing for the components:
https://docs.apigee.com/private-cloud/latest/installation-requirements

Let me know if you don't progress, or please open a support ticket so that an engineer can look into the details and potentially collect additional metrics with extra instrumentation during your tests to identify the root cause(s) and resolve.

Best

Nicola

[#11]

We are on the right track of testing. I expect the answer if you have experienced the same and what was the root cause. I am asking on help to increase the TPS.

[#12]

For our infra designing and setup is done by Apigee itself. So, The infra we are having can be trusted. There could be some post-installation configuration or any other fix which we can implement to enhance the TPS.