Error : faultstring": Rate limit quota violation. Quota limit exceeded.

Not applicable

Our product is an add-on application for JIRA cloud. The plugin can be used to bring test management capabilities to JIRA projects. At this stage, we're not using any API management solution. Currently, I'm trying to implement APIGEE Edge. I am quite new to the product and want some help.

1) We've a test case importer utility(java executable jar file), which allows the user to easily import test cases from Excel spreadsheets or XML files. To run the import utility we must give the appropriate parameters to the importer(URL for our installation, Username and password, API Key, Secret Access Key). If connection attempt is successful the Project list box will appear and now user can start importing the test cases.

Requirement: We want to restrict access to our API on per-user basis. We want implement quota based limitations(API limitations) based on every user API key that is unique for each user account(JIRA), there are specific header fields that we send in our API request and based on those we need to say if this tenant has reached their limit or not say 1000 per day per tenant.

I created two test user accounts and generated API keys for each user. Post that I setup the Proxy in Apigee Edge and attached Quota based policy. Did slight modification in the default Quota policy.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Quota async="false" continueOnError="false" enabled="true" name="Quota-1" type="calendar">
    <DisplayName>Quota-1</DisplayName>
    <Properties/>
    <Allow count="2" countRef="request.header.Zapiaccesskey"/>
    <Interval ref="request.header.quota_count">1</Interval>
    <Distributed>false</Distributed>
    <Synchronous>false</Synchronous>
    <TimeUnit ref="request.header.quota_timeout">hour</TimeUnit>
    <StartTime>2016-12-14 12:00:00</StartTime>
    <AsynchronousConfiguration>
        <SyncIntervalInSeconds>20</SyncIntervalInSeconds>
        <SyncMessageCount>5</SyncMessageCount>
    </AsynchronousConfiguration>
</Quota>

Here is how the header looks like :

request.headers
Out[1]: EnvironHeaders([('Authorization', u'JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsInFzaCI6ImZjZDU1YjIzNGE0NGFlMTBjYWJiYTQ4OTUyNTJlMTYwkd93kdjkfdjkJhbWx5WVRwaE9XVTRZell3WlMwME16RXhMVFJqTkRBdE9ERmxNUzFoTURFeU5HUmhNR0l5TVRZZ1lXUnRhVzQiLCJleHAiOjE0NTY0ODI5NzUsImlhdCI6MTQ1NjQ4Mjc5NX0.chMhYrYob99FYc8Dv6VuHC8CLnx5-aRNTHLC2znNMrI'), ('Content-Length', u'69'), ('User-Agent', u'ZFJImporter'), ('Connection', u'Keep-Alive'), ('Host', u'192.168.2.100:500'), ('Zapiaccesskey', u'amlyYTphOWU49dkkk39292EyNGRhMGIyMTYgYWRtaW4'), ('Content-Type', u'application/json; charset=UTF-8')])

The problem what I'm facing is when I try to upload a test case using any of the user account it throw below error :

Error:Unexpected response code: 500{"fault":{"faultstring":"Rate limit quota violation. Quota limit  exceeded. Identifier : _default","detail":{"errorcode":"policies.ratelimit.QuotaViolation"}}}<br>

Any help would be greatly appreciated.

Solved Solved
1 4 4,709
1 ACCEPTED SOLUTION

Yep, ok...

First, consider doing an RFC7523 swap of JWT for OAuth token. It makes for more efficient processing of requests.

ok, now, onto your specific questions.

The Quota policy in Apigee Edge includes an Identifier parameter. (This, and everything else I will describe below, is explained in the documentation for the Quota policy, which is available here). This is the thing which tracks the accrual of calls. So if you have a unique username, you might use username as the identifier. If you have a user guid, then you could use THAT. It would look like this:

<Quota name='Quota-1'>
    <Identifier ref='unique_user_id' />
    <Allow count='1000'/>
    <Interval>1</Interval>
    <TimeUnit>month</TimeUnit>
    <Distributed>true</Distributed>
    <Synchronous>false</Synchronous>
    <PreciseAtSecondsLevel>false</PreciseAtSecondsLevel>
    <!--
    variables set by this policy include:


    ratelimit.{policy_name}.allowed.count
    ratelimit.{policy_name}.used.count
    ratelimit.{policy_name}.available.count
    ratelimit.{policy_name}.exceed.count
    ratelimit.{policy_name}.total.exceed.count
    ratelimit.{policy_name}.expiry.time
    ratelimit.{policy_name}.identifier
    ratelimit.{policy_name}.class
    ratelimit.{policy_name}.class.allowed.count
    ratelimit.{policy_name}.class.used.count
    ratelimit.{policy_name}.class.available.count
    ratelimit.{policy_name}.class.exceed.count
    ratelimit.{policy_name}.class.total.exceed.count
    -->
</Quota>

The API Key is generally NOT a user identifier. In the Apigee Edge model, the API key is an identifier for the App. The app might have one instance installed, or in the case of a consumer-oriented app that runs on mobile phones, there could be thousands, or hundreds of thousands, or millions of instances of that one app, all using the same API key. Whether 1 instance or millions of instances of the app are in use, the user identification is entirely independent of the app identification.

The API key identifies the app. The developer of the app obtains a key and embeds it into the app he or she delivers. If the developer of the app revises the app, then the developer should get a new API key and embed the new one into the newly revised App. The API key does not identify the user of the app.

What identifies the user? I don't know - that depends. If the app is built by the same people as the people that build the API (for example, let's say a Walgreens consumer app built by Walgreens that uses Walgreens APIs), then it's ok for the app to ask the user for user credentials, and for the app to pass those credentials to the user authentication endpoint, whatever that may be.

On the other hand if the app is built by a party other than the owner of the user authentication service, then you need some sort of intermediary. you don't want to encourage users to key in their Walgreens credentials into a 3rd party app not controlled by Walgreens. The app might store the creds, misuse them, and so on. This is where OpenID Connect comes into play.

When the user is authenticated, the API Proxy can generate a token and return it to the app. This token, just an opaque string of N characters (usually 28) - uniquely identifies the user and the app he is using.

Then, the app passes that token in with every API request. Apigee can validate the token, retrieve the user id associated to the token, and enforce the quota using that User Identifier as the Identifier in the Quota policy. Ya get it?

Ok, but let us suppose that you don't want to incur the hassle of implementing OpenID Connect. In fact you said you have a userid + password.

So I suppose you have some sort of user authentication service. From that service, if you get a "good" response, can you generate or retrieve an ID that is unique to the user, secret (not known publicly, so an email address is no good) and also stable over time? You could even compute an HMAC of the username + some other value (as a salt for the user), and return THAT. But it must be stable. It must be the same ID every time the user logs in.

In that case, you can pass that ID back to the app (the Java code), and have the Java code simply pass in that identifier with every subsequent API request made on behalf of the authenticated user, probably in an HTTP Header. Then the Quota policy would use THAT as the Identifier. IT works the same way as a token, for the purposes of the Quota policy.

The reason you need the ID to be stable - if the user logs out and then re-logs in again, if you generate a new ID, then ... the Quota will use that new identifier, and the counting will start from zero. I think you don't want that.

I suppose you *could* use the API Key to identify users, but... it was never intended for that purpose.

Now, let's look at YOUR configuration of the API proxy.

<Quota name="Quota-1" type="calendar">
    <Allow count="2" countRef="request.header.Zapiaccesskey"/>
    <Interval ref="request.header.quota_count">1</Interval>
    <Distributed>false</Distributed>
    <Synchronous>false</Synchronous>
    <TimeUnit ref="request.header.quota_timeout">hour</TimeUnit>
    <StartTime>2016-12-14 12:00:00</StartTime>
    <AsynchronousConfiguration>
        <SyncIntervalInSeconds>20</SyncIntervalInSeconds>
        <SyncMessageCount>5</SyncMessageCount>
    </AsynchronousConfiguration>
</Quota>

This policy config isn't going to work. Let me explain why.

You have the "Allow" element referencing the variable request.header.Zapiaccesskey. The countRef is intended to be the name of a variable that holds a count. Like 1000, 10000, 5000.... This is the allocated count for a quota. This is not the same as the Zapiaccesskey. Furthermore you would not want to use a request header to hold the quota count, because then any user could pass in a large number in the header, and would get THAT a their quota allottment. (And looking at the doc, I see we have an example that shows EXACTLY this. Doing it that way is handy when you are exploring the Quota policy, but it's not useful in a practical sense)

In the simple case, the count attribute (2 in your case) is hard-coded into the Quota. In the more complicated case, the count value can be retrieved from metadata attached to an API product. This is the purpose of the countRef attribute. Also, as the doc states, the countRef will supercede the count value, when both attributes are present. I suggest while you work the basics out, you omit the countRef attribute, and just use the count attribute, and hardcode it to something reasonable like 100 or 1000. Later when you get the basics sorted, you can use an API product and put the quota value there.

Probably the reason you are getting "quota exceeded" is that the Zapiaccesskey is being interpreted as a number at runtime, which resolves to zero. Therefore a quota of zero is allowed, in which case the Quota policy will always trigger an overage.

ok, Also,... There is no Identifier element in your configuration. You need that, and as I explained above, it should be unique to the user. If you look at the error message being generated, Apigee Edge is telling you that the identifier it used is "_default". This is because you didn't explicitly specify an Identifier element. This means that every API request will fall under the same quota, and a high volume user could consume all of the quota, starving any other users. Not what you want.

Fix those two things and the Quota policy should start to work.

In addition though, I have some further suggestions:

1. consider whether you need to continue to use the AsynchConfiguration element (probably not) and whether you need the ref attribute on TimeUnit. Just make it "hour" or "day" or "month" and be done with it.

2. Make it use a Distributed Quota. Set Distributed to true. Trust me.

let me know how it goes.

View solution in original post

4 REPLIES 4

Yep, ok...

First, consider doing an RFC7523 swap of JWT for OAuth token. It makes for more efficient processing of requests.

ok, now, onto your specific questions.

The Quota policy in Apigee Edge includes an Identifier parameter. (This, and everything else I will describe below, is explained in the documentation for the Quota policy, which is available here). This is the thing which tracks the accrual of calls. So if you have a unique username, you might use username as the identifier. If you have a user guid, then you could use THAT. It would look like this:

<Quota name='Quota-1'>
    <Identifier ref='unique_user_id' />
    <Allow count='1000'/>
    <Interval>1</Interval>
    <TimeUnit>month</TimeUnit>
    <Distributed>true</Distributed>
    <Synchronous>false</Synchronous>
    <PreciseAtSecondsLevel>false</PreciseAtSecondsLevel>
    <!--
    variables set by this policy include:


    ratelimit.{policy_name}.allowed.count
    ratelimit.{policy_name}.used.count
    ratelimit.{policy_name}.available.count
    ratelimit.{policy_name}.exceed.count
    ratelimit.{policy_name}.total.exceed.count
    ratelimit.{policy_name}.expiry.time
    ratelimit.{policy_name}.identifier
    ratelimit.{policy_name}.class
    ratelimit.{policy_name}.class.allowed.count
    ratelimit.{policy_name}.class.used.count
    ratelimit.{policy_name}.class.available.count
    ratelimit.{policy_name}.class.exceed.count
    ratelimit.{policy_name}.class.total.exceed.count
    -->
</Quota>

The API Key is generally NOT a user identifier. In the Apigee Edge model, the API key is an identifier for the App. The app might have one instance installed, or in the case of a consumer-oriented app that runs on mobile phones, there could be thousands, or hundreds of thousands, or millions of instances of that one app, all using the same API key. Whether 1 instance or millions of instances of the app are in use, the user identification is entirely independent of the app identification.

The API key identifies the app. The developer of the app obtains a key and embeds it into the app he or she delivers. If the developer of the app revises the app, then the developer should get a new API key and embed the new one into the newly revised App. The API key does not identify the user of the app.

What identifies the user? I don't know - that depends. If the app is built by the same people as the people that build the API (for example, let's say a Walgreens consumer app built by Walgreens that uses Walgreens APIs), then it's ok for the app to ask the user for user credentials, and for the app to pass those credentials to the user authentication endpoint, whatever that may be.

On the other hand if the app is built by a party other than the owner of the user authentication service, then you need some sort of intermediary. you don't want to encourage users to key in their Walgreens credentials into a 3rd party app not controlled by Walgreens. The app might store the creds, misuse them, and so on. This is where OpenID Connect comes into play.

When the user is authenticated, the API Proxy can generate a token and return it to the app. This token, just an opaque string of N characters (usually 28) - uniquely identifies the user and the app he is using.

Then, the app passes that token in with every API request. Apigee can validate the token, retrieve the user id associated to the token, and enforce the quota using that User Identifier as the Identifier in the Quota policy. Ya get it?

Ok, but let us suppose that you don't want to incur the hassle of implementing OpenID Connect. In fact you said you have a userid + password.

So I suppose you have some sort of user authentication service. From that service, if you get a "good" response, can you generate or retrieve an ID that is unique to the user, secret (not known publicly, so an email address is no good) and also stable over time? You could even compute an HMAC of the username + some other value (as a salt for the user), and return THAT. But it must be stable. It must be the same ID every time the user logs in.

In that case, you can pass that ID back to the app (the Java code), and have the Java code simply pass in that identifier with every subsequent API request made on behalf of the authenticated user, probably in an HTTP Header. Then the Quota policy would use THAT as the Identifier. IT works the same way as a token, for the purposes of the Quota policy.

The reason you need the ID to be stable - if the user logs out and then re-logs in again, if you generate a new ID, then ... the Quota will use that new identifier, and the counting will start from zero. I think you don't want that.

I suppose you *could* use the API Key to identify users, but... it was never intended for that purpose.

Now, let's look at YOUR configuration of the API proxy.

<Quota name="Quota-1" type="calendar">
    <Allow count="2" countRef="request.header.Zapiaccesskey"/>
    <Interval ref="request.header.quota_count">1</Interval>
    <Distributed>false</Distributed>
    <Synchronous>false</Synchronous>
    <TimeUnit ref="request.header.quota_timeout">hour</TimeUnit>
    <StartTime>2016-12-14 12:00:00</StartTime>
    <AsynchronousConfiguration>
        <SyncIntervalInSeconds>20</SyncIntervalInSeconds>
        <SyncMessageCount>5</SyncMessageCount>
    </AsynchronousConfiguration>
</Quota>

This policy config isn't going to work. Let me explain why.

You have the "Allow" element referencing the variable request.header.Zapiaccesskey. The countRef is intended to be the name of a variable that holds a count. Like 1000, 10000, 5000.... This is the allocated count for a quota. This is not the same as the Zapiaccesskey. Furthermore you would not want to use a request header to hold the quota count, because then any user could pass in a large number in the header, and would get THAT a their quota allottment. (And looking at the doc, I see we have an example that shows EXACTLY this. Doing it that way is handy when you are exploring the Quota policy, but it's not useful in a practical sense)

In the simple case, the count attribute (2 in your case) is hard-coded into the Quota. In the more complicated case, the count value can be retrieved from metadata attached to an API product. This is the purpose of the countRef attribute. Also, as the doc states, the countRef will supercede the count value, when both attributes are present. I suggest while you work the basics out, you omit the countRef attribute, and just use the count attribute, and hardcode it to something reasonable like 100 or 1000. Later when you get the basics sorted, you can use an API product and put the quota value there.

Probably the reason you are getting "quota exceeded" is that the Zapiaccesskey is being interpreted as a number at runtime, which resolves to zero. Therefore a quota of zero is allowed, in which case the Quota policy will always trigger an overage.

ok, Also,... There is no Identifier element in your configuration. You need that, and as I explained above, it should be unique to the user. If you look at the error message being generated, Apigee Edge is telling you that the identifier it used is "_default". This is because you didn't explicitly specify an Identifier element. This means that every API request will fall under the same quota, and a high volume user could consume all of the quota, starving any other users. Not what you want.

Fix those two things and the Quota policy should start to work.

In addition though, I have some further suggestions:

1. consider whether you need to continue to use the AsynchConfiguration element (probably not) and whether you need the ref attribute on TimeUnit. Just make it "hour" or "day" or "month" and be done with it.

2. Make it use a Distributed Quota. Set Distributed to true. Trust me.

let me know how it goes.

Hey @Floyd Jones - you may want to have a look at the Quota example in the docs. At least one example shows the use of request headers as the references for the count and interval. That's handy for exploration but it's not a good model to follow for real-world implementation. If we could add a disclaimer, or somehow provide a different example... that'd BE GREAT.

4074-screenshot-20161214-144330.png

4075-thatdbegreat-1.jpg

That's an excellent call, Mr. Lumbergh! Docs are updated. I used the variables from the API product config instead.

@Varun Malhotra, you can have the Quota policy get the Allowed count, Interval, and TimeUnit from the API product that the API key (or OAuth token) is associated with. Following is a screenshot of the UI showing an API product defined to allow 7 calls (within 5-hour intervals--yeah, it's a goofy setting).

4077-product-quota.png

Following is the variable you'd use in the Quota policy's Allow element to read the API product's quota setting (7, in this case).

<Allow countRef="verifyapikey.VerifyAPIKey.apiproduct.developer.quota.limit"/>

Here's the flow:

Call comes in with an API key -> "VerifyAPIKey" policy reads the key, knows which API product it's associated with -> policy reads the variable values from the product and enforces quota accordingly.

The docs also show the variables for Interval and TimeUnit.

Thanks a lot for the detailed explanation. Actually, Zapiaccesskey in the request header holds the API Keys which is always unique for each user. 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Quota async="false" continueOnError="false" enabled="true" name="Quota-1">
    <DisplayName>Quota-1</DisplayName>
    <Properties/>
    <Allow count="10"/>
    <Identifier ref="request.header.<strong>Zapiaccesskey</strong>"/>
    <Interval ref="request.header.quota_count">1</Interval>
<strong>    <Distributed>true</Distributed>
</strong>    <Synchronous>true</Synchronous>
    <TimeUnit ref="request.header.quota_timeout">month</TimeUnit>
</Quota>
<br>

Here is how my new policy looks like.

See the header request snippet :

request.headers
Out[1]: EnvironHeaders([('<strong>Authorization</strong>', u'JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsInFzaCI6ImZjZDU1YjIzNGE5NzUsImlhdCI6MTQ1NjQ4Mjc5NX0.chMhYrYob99FYc8Dv6VuHC8CLnx5-aRNTHLC2znNMrI'), ('Content-Length', u'69'), ('User-Agent', u'ZFJImporter'), ('Connection', u'Keep-Alive'), ('Host', u'192.168.200.91:5000'), ('<strong>Zapiaccesskey</strong>', u'amlyYTphOWU4YzYwZS00MzExYgYWRtaW4'), ('Content-Type', u'application/json; charset=UTF-8')])<br>

Like you suggested, I used the <Identifier> element to configure the quota policy to create unique counters based on unique variables like "Authorization" or "Zapiaccesskey". But again, I'm seeing the same error:

6. Error:Unexpected response code: 500
{"fault":{"faultstring":"Rate limit quota violation. Quota limit  exceeded. Identifier : ZjU3OTRlNWUtZjYwMi0zYTJlLWI2M2MtOTQyOGFmZWI4NTdhIGFkbWluIFVTRVJfREVGQVVMVF9OQU1F","detail":{"errorcode":"policies.ratelimit.QuotaViolation"}}}
, Thu Dec 15 04:46:46 PST 2016 : bvt_import_file.xls imported successfully..!


<br>

At this stage, we don't want to use Apigee for authentication. Just simple proxy and throttling. @Dino @Jose Badino