Quota counter not getting reset after restarting 4 MP's in 9 node cluster

[gauravjain_59]

Hi

I have a API Proxy having Quota defined as 10 requests per day.

Now I want to test same API Proxy in same day with quota reset to 0.

I am using 9 node cluster having 4 MP's and tried restarting all 4 MP's to get the quota count reset, but it's not working.

Thanks

[ssvaidyanathan]

Hi @GAURAV

Not sure if you are restarting just to reset the counter. But why not use the Reset Quota policy ?

Have a logic in your code to reset only when you pass some header or else create another conditional flow only some have access to call so that they can invoke this policy and reset the counter.

[gauravjain_59]

No we just need for testing as we don't want to wait for another day to test quota limits. This is not for production , we don't want to use Reset Quota Policy , there is no requirement to reset quota. This is just for resetting the quota counter for testing purpose.

[ssvaidyanathan]

Ok.. Why not externalize the quota values so that you can have different quota configs. You can define the quota in the Product configuration so when you run VerifyAPIKey or VerifyAccessToken, these variables can be loaded and that can be used dynamically

<Quota name="CheckQuota"> 
  <Interval ref="verifyapikey.verify-api-key.apiproduct.developer.quota.interval">1</Interval>
  <TimeUnit ref="verifyapikey.verify-api-key.apiproduct.developer.quota.timeunit">day</TimeUnit>
  <Allow count="10" countRef="verifyapikey.verify-api-key.apiproduct.developer.quota.limit"/>
</Quota>

Then you can default the values to 10 request per day as per the policy if the product config is not set.

For your testing, you can configure a product with very high quota and then use that for your testing. With this there is no change to your code.

[gauravjain_59]

This is how we are also using the Quota Policy.

But in our case we have Org Level KVM where we store the quota limit and that value is directly referred in Quota Policy like

<Allow countRef="quota_limit_day"/>

and Quota Policy looks like below

Now when we change the value in KVM , then quota variable should be automatically updated with new value. Suppose quota_limit_day was 10 requests per day and now KVM is updated to 15 then quota_limit_day will be automatically updated to 15 and hence the quota should reflect 15 requests instead of 10 requests per day.

And this is not happening currently, and we want to achieve this. Please suggest

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Quota async="false" continueOnError="false" enabled="true" name="Quota-Day">
    <DisplayName>Quota-Day</DisplayName>
    <FaultRules/>
    <Properties/>
    <!--
    <Interval>1</Interval>
    <TimeUnit>minute</TimeUnit>
    -->
    <Interval ref="verifyapikey.Verify-API-Key.apiproduct.quota_interval_day"/>
    <TimeUnit ref="verifyapikey.Verify-API-Key.apiproduct.quota_timeunit_day"/>
    <Allow countRef="quota_limit_day"/>
    <Distributed>true</Distributed>
    <Synchronous>true</Synchronous>
    <PreciseAtSecondsLevel>false</PreciseAtSecondsLevel>
    <Identifier ref="request.queryparam.tenantId"/>
</Quota>

[williamssean]

Hello @GAURAV.

This seems like it could be a KVM expiry time issue. The KVM policy retrieves the value from its cache. If you update the KVM via the UI or via the management API, then the cached value is not updated immediately. So if you have 10 set in your KVM, and then you update the KVM value via the UI to 15, your KVM policy will still return 10 to your proxy until the expiry time occurs (default expiry is 300 seconds). Once the cache entry expires, then it will refresh its cached value with the entry that you added via the UI or management API.

If you want to update the KVM cached value immediately, then you then you should use the KVM policy with a PUT operation instead; otherwise you have to wait until the cached entry expires before you see the updated value.

[gauravjain_59]

But we are clearing the KVM cache when we are testing the updated values in KVM, so it's not that the entry is in cache, cache is empty but still Quota limit is not refreshed and we get the Quota Limit exceeded error.

[williamssean]

Ok. So you update the KVM cache by using a KVM policy that has the PUT operation included. This updates the KVM to the new quota value (from 10 to 15) that you want to test. But when you start Apigee trace and send requests to the proxy, you still see the old KVM value (10) when the quota policy executes.

Additionally, when you send a request to Edge, you receive a quota violation error. If this is the case, then there are a couple of approaches that you could take to reset the quota and I have verified that they both work.

1) Reset the quota with the reset quota policy as Sai mentioned

If a developer has exceeded their quota count, then you can reset the allowed count with Reset quota policy. The policy below will provide an additional 10 requests to the developer's allowed quota counter.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ResetQuota async="false" continueOnError="false" enabled="true" name="Reset-Quota-1">
    <DisplayName>Reset Quota-1</DisplayName>
    <Properties/>
    <Quota name="Quota">
        <Identifier name="_default">
            <Allow>10</Allow>
        </Identifier>
    </Quota>
</ResetQuota>

2) Increase the allowed quota counter in the Quota policy. It seems that you are already doing this by modifying the KVM value, but you still receive a quota violation error. I tested this in my Apigee instance, with a KVM and without it and it works correctly.

<Quota async="false" continueOnError="false" enabled="true" name="Quota-Day">
    ...
    <Allow countRef="quota_limit_day"/>
</Quota>

Can you share your proxy?

[gauravjain_59]

But it's not working in our case. We don't want to reset the quota using Reset Quota Policy as it's not needed.

We will just update the KVM and updated Quota Limit from KVM should reflect.

Attaching the proxy for your reference

Thanks

legacy-api-v1-document-rev1-2017-07-20.zip

[gauravjain_59]

Also it works on AIO environment but problem is with 9 node cluster with 4 RMP

[williamssean]

I don't see how this proxy could work in your aio env, but not in the 9 node cluster. Here is what I noticed in your proxy.

I see that you have two quota policies.

Quota-Minute

 <Allow countRef="quota_limit_minute"/>

Quota-Day

 <Allow countRef="quota_limit_day"/>

Your KVM policy fetches a KVM value into the tenantConfig variable.

  <Get assignTo="tenantConfig">
        <Key>
            <Parameter ref="request.queryparam.tenantId"/>
        </Key>
    </Get>

Where do you set the quota_limit_day and the quota_limit_minute flow variables?

You have a JS-SetTenantId JavaScript policy before the quota policies, but the JavaScript code is not included in the proxy bundle, so I can't see this code. Are those flow variables set in this JS policy? Screen shots of what you are seeing would help too.

[gauravjain_59]

quota_limit_day and the quota_limit_minute values are present in KVM and these are fetched in Javascript which is present at org level not at bundle level.

Attaching the JS file

settenantid.zip