Apigee strange latency issue: A condition evaluation takings seconds

[shrenikkumar-s]

Dear all,

I'm here again with an issue that seems strange to me & our teams. Our ops team were seeing unexpected latencies for few API calls and the analysis indicated it's probably Apigee taking time to process.

Interesting thing was it was not happening for every single but only a few i.e. same operation would work perfectly with low latency most of the times except a few. We're fortunate enough to get a debug session of the same and when we saw was: A simple string match condition taking 4528ms off total 5089ms of the transaction itself, including the target time.

The condition itself we do not notice anything out-of-the-ordinary and it works fine 90% of the time. Could it be that something else taking the time and the blame is put on this condition? Or what do you think can be happening. I've never seen something like this in the past and hoping for some helpful insights 🙂

@kurtkanaskie @dchiesa1 @Sai Saran Vaidyanathan
@dknezic @ganadurai @Manisha_Chennu @Peeyush_Singhai @markjkelly 
@ishitasaxena 

A screenshot showing the situation (& unaware if I can attach a trace session, do not see an option).

[dknezic]

I'd assume debug is misattributing the latency to the condition evaluation and it's more likely to be caused by one of your policies. It looks like you have a number of javascript policies for example amongst others.. so depending on what you're doing within these policies, traffic, payload sizes etc this is more likely the culprit.

[shrenikkumar-s]

Thanks for your response. If it's misattributing then it's hard suspect or point-out what actually causing the problem. How do we investigate further in such cases, any suggestions? Though there are many javascripts, most of them are simple.

[dknezic]

Maybe load testing directly against the backend to ensure it's providing consistent response times..

Then review your custom javascript/policies and start testing those policies in isolation.

[shrenikkumar-s]

Thanks, are you indicating that target response time could also be misattributed into this condition?

[dknezic]

It's just to rule it out, and unexpected larger target response times could also impact performance of other APIs and policies when combined with high traffic conditions also.