Implementing an Event Driven Framework with Apigee

[SidDas]

Hi,

I'm trying to create a framework in which, if a proxy is triggered by the backend, the proxy uses the callback URLs in all the Apps which have subscribed to a specific Product, to push the data received - a typical webhooks design. This way all Developers can create Apps and associate it with the WebHooks Product to get data for the subscribed event without the backend or the proxy being made aware of the same.

I can achieve the same using Management APS as below:

1. Get a list of all App Ids using: /v1/organizations/{org name}/apps
2. Iterate through each of the Apps and get its respective details using : /v1/organizations/{org name}/apps/{app id}
3. Get the callback URL from step 2 if the Product Name in the App matches the Webhook Product name and a custom attribute called event matches the event which invoked the proxy
4. Make an API call to invoke the subscriber using the callback URL in the App

However, it is an antipattern if I use Apigee Management calls within the proxy so I thought I should use the AccessEntity Policy to get the above information. However I could not find a way to get the list of all Apps using the AccessEntity policy. I can only get the information of a specific Product or oa Specific App but that does no help since I will not be aware of all the developers who will be subscribing to the WebHook product at design time.

I have also thought of other designs where the mapping of the event to the callback URLs is held in some sort of DB or KVM, but then it brings the overhead of maintaining that list. Since this association can be found in the App and Product mapping, I thought its best to leverage the same.

Thanks in advance in how I can achieve the design using a proxy.

[SidDas]

Just to elaborate on the details I'm looking for to filter the correct Apps, I need to match the three highlighted details on the output of the AccessEntity for a specific AppId as in the screenshot. However the problem is to get a list of all App Ids (using the AccessEntity) so that I can iterate and get the details of all Apps to match the ones i need

[SidDas]

I have adopted the following solution, but will be happy to get any comments or suggestions to improve the same.

1. Apigee Main Proxy: The main proxy is triggered by the backend application and calls a sub proxy endpoint using Local Target to get the list of all App Ids

2. Apigee Sub-Proxy 1: The subproxy endpoint which fetches the list of all App Ids by making an Apigee Management call is cached to reduce the exposure to the anti-pattern of calling the management API server from within a proxy code. Once the list of all App Ids are fetched a local target call is made to a second subproxy endpoint.

3. Apigee Sub-Proxy 2: The second subproxy endpoint takes the first App Id from the list, calls AccessEntity policy to get all details for the App and if the App qualifies the filters for the event name and product id, then the request data is sent to the callback URL of the app. The processed App Id is removed from the list and the updated list is sent to the same endpoint using local target connection. This recursive calls to the second subproxy is continued when there are no more App IDs remaining in the list and there is a condition to break the recursion 

The above recursion design addresses the two key challenges:

1- No ability to iterate within an Apigee proxy (without using custom Javascript/Java)

2- No ability to use Local Connections in custom Javascript/Java)

3- No ability to call an Apigee Policy from within custom Javascript/Java

4- Avoid calling management APIs from within proxy code (non scalable anti pattern)

The diagram to explain the design is attached

@DChiesa @adas Apologies for tagging you here, but based on some of your solutions on how to handle iteration in proxy and use of proxy chaining, I thought it will be good to get your views on the above design.

[ganadurai]

I would use a KVM managed at the environment scope.

You will have one proxy the consumers will invoke (post app registration/creation) to register their webhook and the event name associated. This proxy will add the webhook to a list which is stored as a value in the KVM and the event name will be the key.

When the event occurs, the second proxy is called which will take in event name as the input (in body or param) and access the same KVM (in env scope) to get the webhook list stored against the key (event name). Use the same iteration you have suggested in your approach to post the data to each of the webhook.

Hope this helps.
Thanks.

[SidDas]

Thanks for your thoughts on this. KVM and PropertySets were my initial thoughts on storing the map between events and call back URLs. However, that would introduce an additional component for management and maintenance overheads.

To minimize the required components I thought I should use the Apps and Products data which get mapped during the registration process. And hence the design decision. 

[dknezic]

Given you have potentially multiple subscribers, this type of scenario may better fit application integration or a another solution outside of Apigee.

https://cloud.google.com/application-integration/docs/overview

[SidDas]

Thanks for your thoughts, there are indeed specialized platforms to implement webhooks like Hook0 and HostedHooks. However, I saw the possibility to build a similar platform leveraging Apigee's capabilities which can provide better features in terms of API Management than some of these readily available products and also avoid adding complexity to the customer's IT landscape as well as simplifying operations by avoiding the introduction of another platform/services. Given that motivation, I was testing the waters to see how I could design something on the API Gateway layer in a simplistic manner to provide basic webhooks framework.