APIGEE replacing systems like Datapower

Not applicable

As I understand APIGEE is only meant for API management with minimal functions for data transformation but I see XSLT policy, Java callout for other data transformation, Javascript policy for json transformation, SOAP to REST conversions in APIGEE. Can we replace systems like Datapower which is completely meant for transformation by APIGEE if performance can be compromised?

When there is lot of transformation in data format, its elements and attributes required, whether we can still recommend to replace Datapower with APIGEE?

3 27 3,412
27 REPLIES 27

there isn't a simple answer - i think there will be lot more considerations than just the transformation functionality itself, for eg, two-speed IT - there was a recent webcast by @Dino and @Robert C. Broeckelmann Jr. regarding this - check out this webcast replay - http://apigee.com/about/resources/webcasts/modernize-soa-apis

I don't agree with your statement that "Apigee Edge has minimal functions for data transformation."

I think Apigee Edge can transform data; you even cited a number of different possibilities. Now, the next question is, "Will the data transform capabilities in Apigee Edge be sufficient?" And to answer that, you need to get specific. What data transformations would you like to perform? XML to JSON? XSLT? JSON transforms? Something else?

**by the way, here's a custom policy I recently built for one limited case of XML transforms.

LOL... I think Apigee Edge can transform data too.

@Dino so now i have to ask - since this is a java callout - does that mean it doesnt run as a shared resource (using full memory, etc every time its called) ?

Hi @Benjamin Goldman - I don't quite understand the question you're asking. This thing is a Java class that runs as a custom policy in the Edge message processor. Edge loads it via a separate classloader, but the class runs in the Java VM, alongside all the other classes for other policies, including the builtin policies. I don't know what you mean by "using full memory, etc". Sorry to be dense! But if you elaborate I'll try to answer.

ref: Apigee Edge docs.

Thanks for your comments.

Huge data transformation from SOAP XML to JSON.Many elements names and their attributes name also have to be changed.

I thought this might be the case. Many of the use cases we use Apigee for involve converting large complex SOAP web services to Hypermedia inspired REST facades.

In terms of HUGE.... can you give me an approximate size of the XML payload in bytes and number of elements? Also what sort of TPS would you be expecting during peak traffic?

I don't have all the details so I feel bad asking this question, but why? If you're only converting from massive SOAP/XML to massive JSON, I'm not sure I get the point. Should be easy enough to do with Edge or DP, and if performance is acceptable you could probably eliminate DP from doing that, for reasons others have suggested here around ease of use, maintenance, etc.

On the other hand, if you're taking a huge chunk of SOAP/XML and thinking about turning it into multiple, more useful JSON web apis, then full steam ahead and Edge is great at doing that. 😉

Carlos, I didn't ask the question and I don't have all the details, but some people want to convert from a Massive SOAP/XML with all the soap envelope stuff, to a JSON, and at the same time, insert default values or "values from partner profile" into the JSON.

What I mean is, the app sends in a slim JSON, the Edge policies convert to XML, then augment that payload with defaulted values from the partner profile (looked up from API key or product ID, etc) and then wrap that all in the correct SOAP envelope.

Even this makes the job of the consuming developer much simpler.

But I agree with your idea that there are huge opportunities for getting more agile, if people can just imagine going beyond the basic "SOAP-to-REST" conversion concept.

I think you could use XSLT for that purpose - transforming and changing names. If you don't like that approach, and prefer to code in JS rather than XSLT, then you could use the out-of-the-box XMLToJSON policy and then write a JS policy that uses Array.map and Array.filter. I find it much easier to do this way. It depends on the particulars of your case though - the size of the payload, the request rate, the familiarity and comfort you have with XSLT, and so on. Maybe you even already HAVE the XSLT written? (If you're a Datapower shop, then this is not far fetched). In which case, just use it within Edge.

But anyway, what you're describing sounds pretty mainstream.

I think it is safe to try. You can run both easily enough. Should be possible to compare. I would look at more than just performance; consider ease of change, etc.

kkleva
New Member

Please define 'a lot' of data transformations. Maybe if you provided a few more specific examples I'd could let you know if it's feasible. I have some exposure to both solutions but less so on Datapower.

We've almost complete with our migration from DP to Apigee Edge and have been pretty happy so far mostly from a sheer visibility perspective. We didn't do any significant level of transformations in DP and it was mostly a facade to Tibco. Like the previous commentors have mentioned, Apigee definitely supports transformations but it's not clear what specifically you're looking to do. I'll say one of the best things I like about being on Apigee from DP is the analytics and visibility into what traffic is passing through the platform. With DP we could see there was traffic but figuring source/destination or even which operations were being used was a complete nightmare. We hired consultants just to help us figure out that part alone...imagine hiring an Apigee consultant to figure out the traffic patterns of your proxies.(ridiculous)

So to answer your question directly, yes, you can replace DP with Apigee and you're on the right track if you're considering that path.

Great feedback @Alan Williams , Glad to know analytics is helpful.

Not applicable

Hello RK4. Sorry I couldn't respond yesterday. As noted above, Dino and I touched on the topic of datapower vs. Apigee Edge in our webcast last month.

Performance does tend to be important, but as several other comments point out there are potentially numerous other factors. Most shops that use datapower likely have a sizeable investment in it. Does it make sense to the business to do a wholesale replacement from an ROI perspective? The answer will vary depending on the organization's circumstances.

Performance for data transformations on DataPower vs. Apigee Edge depends on a variety of factors. So, it is hard to make any general observations. For pure XSLT transformations on a datapower hardware appliance, there is likely to be a performance and scalability benefit--that is what the thing was invented for. The moment you move that idea onto DataPower virtual edition, the performance aspects will likely see a more level playing field. But, all of this depends on the size of the XML document being manipulated, complexity of the XML structure, how much of it is being manipulated, how much other security processing is also occurring, etc. For JSON transformations or JSON<->XML transformations, results will definitely vary depending upon the complexity of the data structures and other factors mentioned. So, bottom line, one has to try it with their specific use cases.

Likewise, on the Apigee Edge side, on-premise vs. cloud will likely have different performance characteristics depending upon the on-premise hardware platform that is running the VMs. On Apigee Edge public cloud, the public/free/trial version is probably going to behave differently in most situations than the dedicated/paid/enterprise version. Again, needs to be tested with realistic data sets.

It also depends on what patterns you are using datapower for. Is it your internal ESB? Is it acting as a gateway (security and interface wise) to the internal ESB? Or, gateway in the DMZ? Is it dealing with data transformations whose inputs/outputs are something other than JSON or XML/SOAP? If so, that may not necessarily be the use case that Apigee Edge is targeted towards--Apigee Edge can run custom Java code, so just about anything is possible, but again, not necessarily a targeted use case. Apigee Edge could act as a replacement or alternative gateway to an existing ESB on the internal network. It is also very common to see it as the gateway in a DMZ. It depends who the actors are that are invoking these APIs.

Summarizing, at a very high-level, I've been in datapower shops where I honestly believe Apigee was the better strategic fit and replacing it makes sense. I have also been in shops where keeping datapower to let it do the things it is good at or run a lot of legacy stuff that it made no sense to migrate in the short-to-medium term was the wise choice--but, augment it with the things that Apigee is good at. The webcast describes such a situation--obviously, there are costs associated with this that must be weighed. I could also envision scenarios where datapower is probably the better choice--but, this is an apigee forum.

There is also the API Management angle(governance, life-cycle management, development community, etc), are you using datapower as part of an API Management solution already? Or, it's more traditional use cases? Is Apigee Edge in your organization being used for the full API Management story or is it more about being a datapower replacement? Many ways this can go...ties back to your use cases/requirements and what problem(s) you are trying to solve.

If you have more specific details about your scenario, I'll be happy to respond back.

RCBJ

Thanks @Robert C. Broeckelmann Jr. for your perspective! This forum is intended to provide broad views around APIs and digital, including API management. It's totally fine to have discussion beyond Apigee 🙂

Hi @Robert C. Broeckelmann Jr.

thanks for providing such a detailed reply.

In our scenario,Datapower is used mainly for transformations.And consumers send SOAP messages to consume the services through datapower.

Now,the consumers want to consume services as REST calls and APIGEE comes into picture there as they also want API management layer.

APIGEE not only has to do mere xml(SOAP) to json transformation but also has to pick and restructure only necessary fields from xml before converting to json.

So,I think XSLT will be better choice to convert SOAP message to another xml.

But some XMLs are big and complex.So,XSLT is not good performance wise.Is there any other way I can do this transformation?Or XSLT is better choice?

> But some XMLs are big and complex.So,XSLT is not good performance wise.

Don't be so sure about that! Before making that conclusion, i suggest you test it.

Sure @Dino .In Message broker tool,XSLT is not preferrable as it slows down the performance.I thought the same here.

Hello again @RK4. Just now getting around to responding to these.

I see you mentioned Message Broker for the first time in this discussion thread. So, it sounds like you have implemented the standard IBM Integration/SOA pattern with Broker as the Integration Bus and DataPower as the Gateway (front door) sitting in front of it.

I would expect the performance of XSLT to be better on DataPower (hardware appliance)than on message broker--as I noted in my last post, that is what it does.

As @Dino and others have pointed out, you will have to test the performance of XSLT against your message types on Apigee to see how it performs. I wouldn't assume anything.

In general, I think XSLT is the right tool for the job when dealing with XML-to-XML transformations. But, as you noted, performance can become a concern. Again, test your use case with XML messages that test what your system would see for real.

As for other potential approaches, Apigee supports (in addition to XSLT) Python, Java, and Javascript. Any of those would be capable of manipulating an XML message. Those all give you plenty of room to get yourself in trouble. And, call me a broken record, you will have to try it with your specific use cases:). If XSLT doesn't work out, I'd try Java next--noting, of course, you'll have to try it with your specific use case.

I'm going to reply to your other post below about ESB patterns and Apigee to keep things readable.

@Robert C. Broeckelmann Jr.

I checked out your presentation http://apigee.com/about/resources/webcasts/modernize-soa-apis.

Towards the end of webcast,you are talking about API wrappers for ESBs to communicate with APIGEE.Please correct me if I am wrong.

In my scenario,I am trying to have SOAP to REST conversion in APIGEE by importing WSDL in APIGEE.And by doing so,I am trying to connect with external world from APIGEE through REST.And to internal applications is from APIGEE to Datapower.And this later part is through SOAP.Here I am not using any API wrappers or libraries for Datapower.By importing WSDL into APIGEE,APIGEE directly could communicate with Datapower.And Datapower to backend systems.Could you advise if this is right approach?

Also,if we say the applications transformation into APIs and APIGEE as the gateway ,I am not sure if we can completely have solution for all kinds of ESB scenarios.Especially Message collection (Check here).Kindly share your thoughts on this.

Hi @RK4 ,

Do you mind starting this comment as a new question? Could add a link to this thread just for context. This will make it easier to parse through information for others looking for the same info.

Btw, LOVE your questions, keep them coming 🙂 what a great way to create shared knowledge by everyone.

Sure @Birute Awasthi ....I will open new thread if any more questions around this topic.

Also @Birute Awasthi ..I love this forum because even if we are ignorant while asking questions,we are enlightened by the answers shared by experts.Experts are very much approachable 🙂 .My learning curve in APIGEE is good because of this forum...

"I am not sure if we can completely have solution for all kinds of ESB scenarios"

I am certain you cannot. If you try, you're going to run into trouble. Some patterns you may be able to get near, or find workarounds for.. others, like with the collection example you link to, you'll find trouble implementing. The risk, IMO, is that while you may find a way to do it with some combination of persistence either in KVM or BaaS and perhaps node.js or Java programmability you'll run into problems when we go off the happy path and need to try to recover from an error of some sort. Once you add multiple, stateless code execution nodes and eventually consistent persistence the challenge of replicating some of those ESB patterns safely becomes a real problem. I would advocate for re-designing the API rather than trying to lift and shift the same logic from SOA/ESB to the API world.

I wouldn't describe it as "API wrappers for ESBs to communicate with Apigee". What I was describing in the WebCast was using Apigee (as an API Gateway) to provide a wrapper around existing SOAP Web Services. Those SOAP Web Services were advertised/secured/other-things using the IBM Integration Stack as an ESB pattern. In that particular shop, there were also APIs advertised on the ESB (Broker + DataPower). There was a desire to continue to leverage the IBM infrastructure that had been deployed and extend the integration stack capabilities into the cloud--that's where Apigee Edge came into play.

The term ESB(Enterprise Service Bus) can mean numerous different patterns. If you are using your ESB for asynchronous messaging or complex orchestration(between several services), these are probably not things you want to to do with Apigee (@Dino or others, please comment). You could probably do some things, but others would likely get difficult (as noted by @Carlos Eberhardt above).

Another approach would be to replace datapower as the "ESB Gateway". It sounds like some of the earlier posters are doing this. I suspect it is the most common approach. At this point, Apigee is the gateway to the ESB(message broker), but, not the part responsible for persistence or asynchronous messaging. Another approach would be to continue using DataPower for SOAP/XML and using Apigee for APIs--having both of those on premise would likely create far more political drama than any problems it might solve. That approach wouldn't be my first choice.

Bottom line, Apigee could fill the role of datapower in your ESB pattern. It isn't a good match for replacing message broker. I'm working with a shop right now that is having a similar conversation. They are looking at using Mule (deployed to the cloud) as a migration path away from Message Broker (IIB). Whether you call this Apigee + Mule construct an ESB is open for debate, but that would then be able to address most of the same use cases DataPower + WMB/IIB are for you today. The are other products on the market that could fill this role. Mule works well especially if you are already working in a Java shop. It should also be noted that Mule has an API Management product that does similar things to Apigee--just to make this more confusing--I don't really have any experience with it.

As always, test things before committing to a direction.

Thanks a lot @Robert C. Broeckelmann Jr. for your clear explanation...I am starting with my usecase keeping all of these in my mind.Thumbrule as you and @Dino adviced : "test things before commiting 🙂 "