Apigee Kafka Integration using Avro schemas

[sunithabprasad]

1) Is it a good idea for critical customer facing applications to have each of the proxies for their endpoints validate an avro schema ? 2) Is there any performance impact in doing so? 3) Also, do we strore these schemas in KVMs? 4) assuming these are the same schemas used by backend services, there is a tight coupling between the proxies and the backend endpoints?

[madhans]

Answering the points in a different order.

Let us start with the design view.

4) The API layer allows you to create business APIs. A payload for a Kafka topic may or may not have a direct mapping to an API endpoint. The key here is to think outside-in rather than inside-out. What does an API consumer (app or client) need and is this met by one or more backend services including a Kafka endpoint.

1) Maybe. For the benefit of early and quick failure detection validating the schema at the API layer seems to be a good idea but there are caveats as you have rightly raised in the other questions.

2) There will be, we are talking about a library that performs the schema validation and its overhead is not going to be 0. A way to decide is to look at the size of the payload and the overall traffic you are pushing through, if the payload is not tiny and the schema validator loads objects in memory your throughput is going to go down.

3) KVM, probably. The key decision here is to know how frequently the schema will change. You are looking at making releases/config changes to Apigee for any changes to the schema. Any update to KVM does not reflect immediately, is this acceptable?

In summary, Apigee remaining a thin layer delegating the schema validation to a backend given the backend will be performing it anyway is probably more suitable given the caveats.

[sunithabprasad]

Thank you

[sunithabprasad]

The idea for this approach was for a different application outside of the proxy flow to receive the request/response data. Looks like sending events in this case is not a good design.

do you have any suggestion on how an application outside of the proxy flow can get this request/response?

[madhans]

Can you clarify more. What is the API client and what is the backend ? Where is the queue in the client or backend ? Where is the application that needs to receive events ?

[sunithabprasad]

Client/Apigee/Backend - is the actual flow path. An application outside of this flow path is interested in knowing the request/response, within the enterprise

[madhans]

You can use a callout policy to pump request and responses to a Kafka queue. This will serve as an event log.

[sunithabprasad]

what other options are supported ? with kafka i have to use avro.