Designing API Proxy to act as messages collector

[#1]

Is it possible to design an API Proxy in such a way that it collects incoming messages for certain period of time and combine them to send to a Target endpoint.

[anilsr]

@RadhamaniRamadoss ,

I think it's an interesting use case. I believe you can able to implement same using cache policies, key-value-map-operations-policy, javascript policies. You can also consider our BaaS which is real handy in this use case i believe.

Step 1 : Maintain a variable which counts number of records

Step 2 : If records let's say < 10

Step 3 : Store the request in cache/ kvm / baas

Step 4 : Increment count number

Step 5 : Don't hit target , return back to client with custom message

Step 6 : If records let's say > 10

Step 7 : Pull records, Append to request payload

Step 8 : Reset Counter

Step 9 : Let the request hit Target Server

Cheers,

Anil Sagar

[#3]

Can you please clarify why Step5 is required?Are you considering this as Synchronous requests?

[anilsr]

@RadhamaniRamadoss , Step 5 is optional, It will help if you would like to give some feedback back to the client.

[#5]

Great ! thankyou!

[anilsr]

Any time 🙂 Keep us posted if you have any queries while implementing same. Looking forward to see it in action. You can also post an Article here in Apigee Community if you would like to share your solution with others in community..

[#7]

Sure...Will do that 🙂

[#8]

@Anil Sagar,

Could you please provide some pointers in creating Counter variable?

I tried with Key-Value map Policy.But its not working as expected after some 4 requests.But I dont find any issue with the logic.Hence,raised the issue in this thread: https://community.apigee.com/questions/10080/keyva...

I checked Cache options but initialisation seems to be a problem.As everytime the first time value ('1') will be assigned.With Custom attributes also,I find the same issue.

[#9]

I am trying the implementation using KeyValue Map Policy.I could store the values but while fetching the values stored from Map,I am facing an issue.Please help to solve the issue.

https://community.apigee.com/questions/9978/fetching-all-the-values-stored-in-cache-or-map-wit.html

I have opened a separate thread for discussing the issue "

[#10]

Issue in the thread https://community.apigee.com/questions/9978/fetchi... has been resolved.

[sarthak]

I agree to @Anil Sagar that it is an interesting use case. And the solution above is definitely one of the possible approaches.

I however personally would do it using Node. When you have this much of business logic needed node is just a much more flexible.

Also I think you might hit some inconsistencies with the above approach (theoretically I think , I have to test it though). Because KVM AFAIK is backed by cache and has L1 (which is per node) and then L2 (which is distributed cache). And behind the scenes the request is getting processed across multiple nodes (we call them MPs) inside Apigee. So before the entry hits distributed cache it is stored in the local per node cache and is not synced across nodes.

So your counter needs to be something outside of the message processors for consistency. In BaaS there is an inbuilt counter API which you can use. You can see here : http://apigee.com/docs/api-baas/content/creatingincrementing-counters.

If you develop in node another advantage will be that it is not distributed, so each node is will have it's own node logic to handle.

I am also curious to understand the nature of the API. Can you please elaborate a bit more what exactly is the use case ? That way we can suggest better which approach should be better for you.

Thanks

Sarthak

[#12]

@sarthak ,Thanks for your response.

I am not sure using Counters would be better approach in this scenario as the doc says "It may take up to 30 seconds after an event has been posted for the counter to be incremented.".So,this delay may not work fine for the counter in this scenario.

I will check with node possibilities.

As Anil Sagar mentioned in his comments,I understand that Message Collection scenario is good to be handled either at client or at server side.But just wanted to know whether Message collection is possible at proxy level.Trying out some options to achieve the scenario.

Use case is,API Proxy gets some number of client requests(say '10').Once this number is reached,API Proxy combines the client requests and sends it to the Target.

The client requests can be combined even after a particular time period.So,Collection can be done based upon either when maximum number of requests reached or upon certain timeperiod.

Using KVM policy ,its working well for collecting messages but implementing Counter is the problem there due to its aformentioned consistency problems.