Are APIs just an evolution of SOA?

[#1]

Summary: If you need two disparate systems to talk to each other, think SOA. If you need to build apps, especially apps on any sort of modern platform, think APIs. And if you want to build APIs that are secure, scale, and will be able to handle all the challenges that are rapidly coming your way, think Apigee. But, please - please - don't think of APIs as just an incremental evolution of SOA.

If we were to go back in time to 2004, we'd find a drastically different technology landscape compared to today. The internet had only recently delivered the long-awaited promises of connecting the world to a superhighway of information. Companies who had for years operated their technology on a foundation of client-server programs, flat file exchanges and batch processes were struggling to cope with consumer expectations for real time information available through self-service websites. To some companies, the answer to this problem was clear - batch jobs and file exchanges, the existing foundation of their enterprise - they merely needed to produce the same files and run the same jobs more frequently! Other companies were recognizing the fundamental shift in how they needed to approach technology. These companies were excited about a new approach called Service Oriented Architecture and were starting to see how this approach could facilitate these real-time system interactions.

This transition required a lot of smart people to solve some very real, non-trivial problems. Business critical systems weren't built to talk to each other. Getting data from one system into another required a lot of discussion between teams. They'd settle on a file format, build test files, write programs to pull the data out of the source system, generate a file, build programs to parse that file and import it into the destination. That's setting aside all the exception scenarios that could arise, like how we would handle duplicate files or if things ever got processed out of order. In order to allow data exchange to take place in real time between any two systems, a series of problems needed to be solved, and SOA solved them.

We need a common way to represent data - XML provided a way of expressing data using plain text, so if your system could process plain text, or if you could create a connector, you could take advantage of data in XML format.

We need a common way to validate data - DTDs and later XML schemas provided a way to define, describe and validate the structure of an XML document.

We need a standard protocol to exchange data - SOAP provided a standard mechanism to structure messages between different systems

We need a standard way to define our service interface - WSDL allowed us to define the set of operations so that we could define exactly how to interact with each system.

When you view SOA in the right historical context, it's easy to appreciate the drastic improvements that SOA facilitated in the field of system integration. Like many other technologies, SOA had the misfortune of stalling out in what Gartner refers to as 'the peak of overinflated expectations'. Not only would SOA revolutionize integration, but it was going to revolutionize business processes. Service re-use would be so extraordinarily pervasive that software developers may become required personnel only at pure technology companies. Business users would be able to simply learn graphical tools to modify service behavior and projects that formerly took months would now be completed in just a few days.

And the technology to enable all these promises quickly followed. New standards emerged in the form of WS-* specifications with mechanisms to handle everything from addressing to distributed transactions. Fat bloated middleware products popped up like mushrooms in a cow pasture after a spring rain: ESBs, registries, repositories, governance tools, lifecycle managers, process designers, process executors, and on and on and on...

Today, if you talk to most organizations about their SOA landscape (perhaps after a tiny drop of truth serum), you'll usually find a mix of services. You'll find services that facilitate system-to-system integration between disparate systems that work very well. You'll also find a bunch of services that were built with the intention of modeling business processes. These are usually the services that nobody wants to touch because their guts are buried inside of a piece of middleware which is 'going away sometime in the future'. You'll also find giant service behemoths which have been steadily growing bigger, year after year, to maximize reuse across different applications, usually degrading the user experience of every application they touch.

Today, these same organizations must solve new and evolving technology challenges. Just like in 2004, these are problems that our existing technology solutions, such as SOA, were never intended to solve. Organizations already struggle to keep pace with the trickle (yes, this is just a trickle) of new devices and form factors that are steadily coming out. iOS and Android smart phones and tablets, fitness trackers and smartwatches may seem like a headache to support today, but compared to the torrent of devices that the Internet of Things will unleash in the next decade, managing today's device ecosystem is a cakewalk. Staffing development expertise in-house for such a wide variety of technologies is going to be cost prohibitive. Partners and specialist firms will play an increasingly important role in application development. Many of those devices will have inconceivably small processing capabilities and network bandwidth compared to the smartphones we have today.

Some organizations are trying to solve these problems with SOA the same way some organizations in 2004 thought they could keep up with the internet by running their old batch jobs more frequently. Other organizations are recognizing that these problems require a much bigger shift in the way they operate and the way they think about their technology. Their focus isn't on building the a service-oriented snapshot of their business model as it exists today. It's about building an organization that is agile and adaptable enough to capitalize on opportunities as technology rapidly advances. These organizations universally embrace APIs as the cornerstone of their architectural strategy - not because REST is philosophically better or worse than SOAP, but because APIs are the defacto standard for building applications quickly on modern platforms. SOAP was amazingly good at bridging the gap between disparate systems. APIs are amazingly good at allowing application developers to build apps and deliver value very, very quickly. When you consider the direction technology is headed, it's easy to see why so many organizations are starting view their strategy through an API-centric lens.

But wait a minute, isn't an API really just a RESTful Service? If so, aren't APIs really just the same thing as SOA, but with RESTful services instead of SOAP? That's what a multitude of SOA middleware vendors would love for you to believe. The problem with this viewpoint is that the SOA most middleware vendors are talking about bears little resemblance to the original tenants of Service Oriented Architecture. The principles that originally surrounded Service Oriented Architecture - things like the thoughtful, deliberate crafting of clean interfaces, putting testability first, and clearly defining behavior, are timelessly valid. Over the last decade, the term SOA has been twisted to mean bloated services, heavyweight governance, complex tangles of tools and middleware that fail to deliver on any of the promises of the past and are terribly suited to help organizations into the future.

So does that mean that organizations should just throw away all of their SOA investment? No, of course not. The same way organizations still find value in some of their flat files and batch jobs, organizations are still going to find value in some of their web services. It is, however, time for organizations to stop throwing good money after bad. With SOA, this means acknowledging that we're never going to represent our business as a single set of complete services. We're never going to have a business user with a graphical tool quickly modify the way systems interact and eliminate the need for a 3-4 month IT effort to accomplish the same thing. And if we're never going to solve those problems, why are we going to submit our organization to the types of heavyweight governance and rigid practices that were supposed to accomplish those objectives?

TL;DR: If you need two disparate systems to talk to each other, think SOA. If you need to build apps, especially apps on any sort of modern platform, think APIs. And if you want to build APIs that are secure, scale, and will be able to handle all the challenges that are rapidly coming your way, think Apigee. But, please - please - don't think of APIs as just an incremental evolution of SOA.

[#2]

@clatimer1

@Birute Awasthi

- Great summary. I like to think of APIs breathing new life into SOA or giving it a second chance. A somewhat abstract comparison can be gained from this article - The Internet inside the enterprise: We don't have it, and we need it where the Internet represents APIs and Enterprise represents SOA for the purposes of this comparison.