High response time (and timeouts) connecting on proxy for 3 days in a row

Not applicable

Hi,

I'll like to ask whether someone else is suffering from high response times (more than a sec) or even timeouts when using APIGee proxies.

We're using a free-tier instance and this has been a recurring problem since Friday 28th July. Sometimes it got fixed, sometimes the high response times starts occurring again.

Find below a simple benchmark during one of those high-response-cycles done with Apache Benchmak (ab):

ab -n 1000 -c 10 http://podsystem-prod.apigee.net/apps/ping?apikey=XXXXXXXX
This is ApacheBench, Version 2.3 <$Revision: 1706008 {replace0}gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking podsystem-prod.apigee.net (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:        Apigee
Server Hostname:        podsystem-prod.apigee.net
Server Port:            80

Document Path:          /apps/ping?apikey=XXXXXXXXXXXXXXXXXX
Document Length:        4 bytes

Concurrency Level:      10
Time taken for tests:   289.925 seconds
Complete requests:      1000
Failed requests:        2
   (Connect: 0, Receive: 0, Length: 2, Exceptions: 0)
Non-2xx responses:      2
Total transferred:      216139 bytes
HTML transferred:       4226 bytes
Requests per second:    3.45 [#/sec] (mean)
Time per request:       2899.245 [ms] (mean)
Time per request:       289.925 [ms] (mean, across all concurrent requests)
Transfer rate:          0.73 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      132  278 1163.7    168   19582
Processing:   241 2583 3783.4    308   16024
Waiting:      240 2582 3783.7    307   16024
Total:        393 2860 4159.6    484   35346

Percentage of the requests served within a certain time (ms)
  50%    484
  66%    645
  75%   7225
  80%   7968
  90%   8327
  95%   8512
  98%  15706
  99%  16808
 100%  35346 (longest request)

As an additional evidence, is it worth showing what our APM tool (newrelic) has been showing regarding the availability of the APIGee proxy. A simple "Ping" monitoring operation that just queries a method of our API through the APIGee proxy has failed 154 times in 3 days.

5387-synthetics-new-relic-1.png

Solved Solved
2 15 2,098
2 ACCEPTED SOLUTIONS

@Felix Ontanon

Free-tier instances are not designed for performance testing / to run production APIs. They are shared pods sharing with thousands of other similar instances. We do not guarantee any SLA's on free pod. They are only for feature evaluation. Trial pods should not be used for production traffic / run any performance evaluations.

Apigee handles billions of API calls every day with 99.999% SLA. Just FYI, 60% of Top World Global Retailers are Apigee Customers & Their API traffic is powered by Apigee. We handle thousands of TPS, Billions of API calls every day on cloud including events like Black Friday. If you are evaluating Apigee, I suggest not to use free instance to understand the performance / scale. Please get in touch with Apigee sales for a demo or PoC.

View solution in original post

Not applicable

Response from sales team:

Following up here briefly to let you know that we have discovered the issue. It turns out the customer in the community post you referenced also signed up in 2014. The issue is isolated to 5 orgs, three of them commented on the same post you saw. I can confirm that all new trial and paid org are woking flawlessly with no reported issues. Please note the issue is not one we plan on resolving. We thought we had everyone moved out of that pod but it sounds like a few people may not have received our communications.

View solution in original post

15 REPLIES 15

Not applicable

Hi Felix, I have exactly facing the same issue for the past 2-3 days. Sometimes it works. Sometimes it takes more than 30 sec to load and even connection timed out. Please someone look into this on high priority, since the production service is hit worsely.

@Felix Ontanon

Free-tier instances are not designed for performance testing / to run production APIs. They are shared pods sharing with thousands of other similar instances. We do not guarantee any SLA's on free pod. They are only for feature evaluation. Trial pods should not be used for production traffic / run any performance evaluations.

Apigee handles billions of API calls every day with 99.999% SLA. Just FYI, 60% of Top World Global Retailers are Apigee Customers & Their API traffic is powered by Apigee. We handle thousands of TPS, Billions of API calls every day on cloud including events like Black Friday. If you are evaluating Apigee, I suggest not to use free instance to understand the performance / scale. Please get in touch with Apigee sales for a demo or PoC.

Hi @Anil Sagar. Thanks for your answer.

I understand and agree with your comment.What you're saying Is the proper way of proceeding. I'll strongly take your considerations

But given the fact that I'm not the only one affected it would be really nice to know whether this bad performance situation for free instances is something that will continue or if someone in Apigee will dedicate some time to fix.

Thanks again.

@Felix Ontanon ,

I forgot to mention in my earlier post, First of all, Thank you for highlighting this issue. Absolutely, I will let the Engineering team know about this right away. I will keep you posted with updates if any but no ETA that i can assure at this point of time.

Welcome to Apigee Community & Please keep us posted moving forward if any.

@Felix Ontanon , I have notified Eng Team & Internally created a ticket. We will keep you posted if any.

Thanks Anil. As said, I'll start sharing your suggestions with the development team in our office to contemplate the upgrade.

@Anil Sagar Can you please update on this?

Is there anywhere where we can follow the progress of this issue? We are also seeing a significant increase in the number of API calls failing. Are the SLA for the "Team"-pricing package affected by the same issues as the Trial pods?

Totally agree with your answer. However is there any work around over this or can someone really look in to the issue since it only started a couple of days back?

@Arun Babu , Please see my comments above.

We are also having the same issue on our side.

Not applicable

It seems like no one is working on the issue and we are hit badly due to this. Even the Indian contact number given on this page says the "Number doesn't exist".

Mr @Arun Babu ,

Trial orgs don't guarantee any SLA's. If you think it hit you badly please consider upgrading it to enterprise / paid account. As far as i know. I don't think we have any Apigee on call using a "Indian contact number" or any other contact number for trial users evaluation.

For your kind information, We don't have any support on twitter or any other social network medium. More over you are not supposed to run production traffic on Apigee trial orgs. I am afraid trial orgs can be deactivated after 30 days of evaluation.

Not applicable

Response from sales team:

Following up here briefly to let you know that we have discovered the issue. It turns out the customer in the community post you referenced also signed up in 2014. The issue is isolated to 5 orgs, three of them commented on the same post you saw. I can confirm that all new trial and paid org are woking flawlessly with no reported issues. Please note the issue is not one we plan on resolving. We thought we had everyone moved out of that pod but it sounds like a few people may not have received our communications.

Hi, we have not received any emails concerning pod migration, how do we proceed?

Also, if we upgrade to premium, will it resolve the issue?

Thanks