How to Meter Api involving background job

[#1]

We have three endpoints available involving Bulk api job processing

• Api to submit job
• Api to get job status
• Api to download data

Here is the current flow.

• Developer makes an api call to submit a job which in return he will get the jobId in response.
• Job will run in the background. Run will take atleast 10 mins after which we will get the product count in order to meter.
• On successful job run, we make an api call to the api gateway to report the usage.
• we are currently using 3scale as api management platform

Need to understand how we can achieve above via apigee. From docs, I see this can be achieved via post client flow but not sure. Please suggest.

Thanks,

Jeevan

[anilsr]

@jeevan , What exactly do you mean "Meter API" , Is it monitoring ? Are you talking about analytics ? What is the end goal here ? What results you would like to see ?

[#3]

@Anil Sagar By "Meter api" I mean metering an api call.

Pls refer to the second step in the current flow mentioned above once. The end goal is to avoid we making an api call to api gateway to report usage and how to achieve the same in apigee itself.

[anilsr]

@jeevan , I am still clueless with the sentence "metering an api call". I assume, background process you mentioned is running on your backend servers. Once the job is completed, obviously your backend has to initiate mechanism of notifying others. Would you like to run background job on Apigee when you mean "The end goal is to avoid we making an api call to api gateway to report usage and how to achieve the same in apigee itself" ? Is the word "meter/metering" your internal code word ?

[adas]

@jeevan If you simply want to get usage of an API, you can use Apigee Analytics that is built in to Apigee Edge API Management platform. You can build sophisticated custom reports based on your business needs too.

If you want to track usage, generate revenue or build a revenue model around your api usage you should look at Monetization. This is an add-on module that comes with Apigee's API Management, but has separate licensing, pricing etc. I have provided the relevant links for you to understand these offering and you can also do an evaluation by signing up for a trial account, which you might have already done.

[mschreuder · Accepted solution]

Hi @jeevan if I have understood your question, the quantity you want to meter is only known once the job in the background has finished processing, i.e. it's not the number of API requests.

If I got that correct, then I can't think of a practical and reliable way to avoid that API call you currently need to make to the API Gateway to report usage but maybe others on this forum can? Here are some thoughts along these lines:

In theory, you could maintain an open connection between your API Gateway and the backend for the duration of processing the job and then send a response back on that connection once the processing ended. That response would contain the product count usage to meter which you could then process in response processing but that's not practical because you will quickly tie up all connections when they each have to be open for 10+ minutes.

Another idea, has some side effects too: You could put a header into the response when the developer uses the API to download the data. That header could contain the product count for metering. You could use response processing to pull that header and then trigger whatever metering related processing you wanted to do. There are two potential issues with this I can think of:

• I imagine that there will be times when the developer doesn't end up using the API to download. So you have done the processing because they submitted the job but they don't come back to get the result. In that case above idea would result in that work not being metered.
• Conversely, and this will also depend on how your API that is used to download the data is working (e.g. can you only download once or unlimited times or maybe some limited amount) you could have situations where you end up metering the same job multiple times. Of course, you could build specific logic to handle that situation.

[#7]

@mschreuder You got my question rightly and thanks for your explanation. Is there any solution in apigee to mitigate this particular scenario?

I would be glad to hear other suggestions as well.

Thanks in advance.

[mschreuder]

@jeevan I've been thinking more but I still can't think of any other options than above - the only absolutely guaranteed way to meter is for the backend to send some sort of event, such as making an API request, when it finishes the processing to report usage, which is what you have already.

There are things you can do in Apigee to track, manage and charge for usage once you have that event. For example you could use Apigee Monetization to track the overall consumption per developer / app / etc in an automated fashion. You could develop rate plans based on what you wanted to structure your pricing around and have Monetization automatically sum up the usage that is posted and then start rejecting requests if the developer consumes their allowance and enable them to buy more for example. See the section called Custom Attributes in this part of Monetization docs for how you could do that for your specific scenario.

Hopefully someone else can add some other thoughts too.