Advice on changing from ESB to API Approach: Calling a REST API Asynchronously makes it not a REST API?

[pranavchaudhary]

Hi,

Is there a problem with the following/Can I have your thoughts and inputs on the following:-

Preface/Motivation: I feel like the following design/approach is wrong and want to discourage it but I don't know the underlying technical field (APIs, REST, ESBs, past state of technology trends, current modern thought) well enough to explain and defend myself so I am approaching the council of API experts on Apigee.

Situation:-

1. As an Enterprise CTO I have made an executive decision to adopt REST (build/expose/consume REST APIs) and interact with SOAP services only by exception. Essentially, this indicates my preference to modernize my tech landscape, adopt the API approach, go, lightweight, improve their complicated processes, etc.

2. At the same time, I am insisting on and have nearly decided upon using an Asynchronous integration pattern for a particular client-->server interaction say for example POST /{resource}/{resourceID} because that's how I've always done it before Apigee came in the picture.

Context: Earlier, before Apigee came into the picture, an ESB alone would handle all my integration needs. I would either batch a massive file every 2 weeks, FTP it to some File Server, have BizTalk fetch the file and send it to the target or just have BizTalk (my ESB) HTTP Post to the target endpoint. This used to be done Asynchronously in a Fire and Forget manner so if something went wrong BizTalk would queue and retry.

Q: Is there anything wrong with wanting to do it this way?

Q: Is there a better way to do this?

Q: Is there any value add or reason why one would want to change this to a synchronous system?

Now my Backend Service Provider's Server exposes an API on Apigee which my client applications consume and APIs by nature are meant for synchronous real-time use cases.

3. Other Technical Questions:

a) Isn't Asynchronous communication not REST ? (i.e. violating REST constraints)

b) Would you call it a REST API or a RESTful Web Service if consumers called it Asynchronously?

c) Is it possible to have REST Architecture and Asynchronous communication within that architecture?

Because if it is Asynchronous you are holding state somewhere that's not the client..either a queue in the middleware/ESB or in the server. That to me is violating the stateless constraint and hence not REST.

[pranavchaudhary]

@Dino-at-Google @Dino @Chris Hood @Martin Nally

[ChrisHood · Accepted solution]

First, you can absolutely have either synchronous or asynchronous returns within a RESTful API.

A synchronous call will wait for the API call to be returned before continuing.
An asynchronous API call will not wait for a return and data is queued for processing.

At the most simply view, synchronous vs asynchronous only determines the amount of time it takes for the API to complete it's process.

REST defines how the API is both called (HTTP) and formatted (HTML, JSON, etc).

So to answer your questions.

a) RESTful APIs can be both synchronous and asynchronous, and there are tons of use cases out there that explain both.

b) To me, it's still an API. I feel by default organizations need to be developing RESTful APIs and if they aren't, then I call them SOAP services. Of course I realize not ALL APIs are REST APIs, but I do tend to personally distinguish them between "API" or "SOAP" - either way, the difference between an API and a Web Service really depends on what it's being used for, not how it is returning the data.

c) Yes, as noted above, there are several use cases for requiring REST to perform Asynchronously. I think what you are getting at though is that REST is typically called through HTTP, and HTTP by default is synchronous. If you are calling web services to build web pages, this may become a challenge, but as noted above, there are multiple ways that we get around the synchronous mode of HTTP to provide asynchronous features (like through HTML5).

Examples of Asynchronous applications that use APIs would be Facebook Messenger, YouTube streaming, and mobile games. Interesting enough, mobile games also use synchronous API calls, such as updating your score when you are offline.

There are several topics in the community on this. Maybe there are some examples that are close to what you are trying to address.

[pranavchaudhary]

@Chris Hood I rephrased the first part and added few more questions right above #3. Could you please address those questions? And I'm afraid if your answer to those is "No" then doesn't that weaken the case for Apigee and APIs?

[ChrisHood]

As always, it depends. As a default rule I would argue that fire and forget is asychronous, but doesn't always have a firm return response time frame. Meaning you can send the massive file, or data to your ESB, but you don't necessarily need an immediate response that it was successful, and in some cases, if it doesn't work, you can queue and wait for it at a later time.

This scenario is completely different than something like a chat tool, where when I send you a message, I want to get an immediate response if that was sent, read, replied to.

Q: Is there anything wrong with wanting to do it this way?

No, there is nothing wrong with doing it this way, but Apigee can in part assist with the message patterns especially between multiple systems such as an asynchronous call between the ESB, CRM and a 3rd party Database.

Q: Is there a better way to do this?

Not necessarily a better way, but again, it depends. In some scenarios developing an API proxy that can manage the calls and responses can help in your design. In other cases, you can keep things simple. I would still recommend an API for the simply scenarios, to also leverage other platform tools like security, analytics, against those calls.

Q: Is there any value add or reason why one would want to change this to a synchronous system?

Once more, it depends. This really boils down to a user experience, or timing that you are looking to achieve. For example if you could potentially shave a customer 15 seconds on loading something, because you did it synchronously, then there's potential value there.

[pranavchaudhary]

@Chris Hood What do you think about the following two statements:-

1. "APIs are only meant for use cases which are both synchronous AND real-time in nature."

2. "If your use case involves heavy-duty batch job processing that's going to hold up the backend then you shouldn't use APIs to accomplish that job and hence Apigee does not belong in such an environment"

I think you would agree with 2nd and not the 1st, correct?

[ChrisHood]

1. Disagree. APIs are meant for just about anything today, and I would argue, half of the things you do on a daily basis wouldn't be possible without asynchronous APIs. Also, the very definition of "real-time in nature" is asynchronous.

2. Partially agree. In general I would never want to hold up a response to a USER. If that means there is some extremely large processing that has to happen behind the scenes, then I might do that with an API, but return a response to the user that your batch is being processed and I will notify you once that process is complete. In this case I'd still use the API, but connect it with a webhook for the response.

Keep in mind, we use APIs for more than just an integration of services. Even in a scenario where something large has to be processed in the background, that API could be packaged, reused, protected, analyzed, and shared with potential partners to develop new tools, or new integration points, or even new products. If you don't build an API for it, then all you have is a backend system processing backend jobs. Where's the innovation in that?