XSL transforming UTF8 responses from Service callout fails special characters for only about 50% of the time

[#1]

Hi

I have en API that has been running for well over a year. Now, all of the sudden, I start to get errors with special characters like ÆØÅ.

My service is making calls to one of four independent severs depending the query parameters set and I have verified that the error is not caused by any of these servers. The XML response from these callouts are then XSL transformed and lastly converted to JSON.

I have made a trace session with two call made immediately after each other. The first call was succesfull (well formatted ÆØÅ) while the second was not. It seems that the XSL transformation is what is failing. The problem is I cannot for the love of God figure out why it is not consistently failing or passing. Can anybody use this to help me where to look?

Thank you so much.

Trace Session: https://www.dropbox.com/s/ptzzq37nck5t45c/trace-1483436472434.xml?dl=0

p.s. Look for the text "Elite Købmand" which is formatted correctly for the first call and wrongly for the second call. It seem that the transformation from line 2131 to line 2300 is the reason for the fail.

XSL transformation:

<?xml version="1.0" encoding="UTF-8"?>
<!-- xsl stylesheet declaration with xsl namespace:
Namespace tells the xlst processor about which
element is to be processed and which is used for output purpose only
-->
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:gls="http://gls.dk/webservices/">
<!-- Output declaration -->
<xsl:output method="xml" indent="yes"/>  
<!-- xsl template declaration: 
template tells the xlst processor about the section of xml 
document which is to be formatted. It takes an XPath expression.
In our case, it is matching document root element and will 
tell processor to process the entire document with this template.
-->
<xsl:template match="/gls:ParcelShopSearchResult">
<!-- HTML tags
Used for formatting purpose. Processor will skip them 
and browser will simply render them.
-->
  <PickupPoints>
	<!-- for-each processing instruction
          Looks for each element matching the XPAth expression
           -->
    <xsl:for-each select="gls:parcelshops/gls:PakkeshopData">
    <PickupPoint>
      <pickupid><xsl:value-of select="gls:Number"/></pickupid>
      <carrier>GLS</carrier>
      <distance></distance>
      <latitude><xsl:value-of select="gls:Latitude"/></latitude>
	  <longitude><xsl:value-of select="gls:Longitude"/></longitude>
      <name1><xsl:value-of select="gls:CompanyName"/></name1>
      <name2></name2>
      <address1><xsl:value-of select="gls:Streetname"/></address1>
      <address2><!--<xsl:value-of select="gls:Streetname2"/>--></address2>
      <zip><xsl:value-of select="gls:ZipCode"/></zip>
      <city><xsl:value-of select="gls:CityName"/></city>
      <country><xsl:value-of select="gls:CountryCodeISO3166A2"/></country>
      <type></type>
    </PickupPoint>
    </xsl:for-each>
  </PickupPoints>
</xsl:template>


</xsl:stylesheet>

Update

It seems that the error has something to do with the callout service. I have not altered my API, so that instead of making callouts, I change the target depending on query parameters and now the error is gone.

I am still using the same XSLT scripts, but I had to create a new API to work from. Might be that this is working from other servers?

[DChiesa]

How annoying! Sorry to hear you're having trouble. But this is a good question!

I don't know why your XSL would be intermittently failing. It seems that it should not be the XSL that is at fault, if it fails only some of the time. Seems like it might be a bug on the part of Apigee.

Some things for you to try

1. How often does it fail? If it is 50% (or pretty darn close), then it could be a misconfigured message processor. Typical environments have 2 MPs, both configured identically. If you get 50/50 success/failulre, it could be that one MP has a correctly configured XSL engine, and the other does not. I have seen inconsistent configs occur between supposedly identically provisioned MPs in the past, but have not seen it in a looooong time. To investigate it, you can deploy this Java callout and examine the results from multiple sequential calls. If the JAR list is different between different requests, it means your MPs are configured differently, which indicates emphatically a provisioning bug by Apigee! Are you using a free trial version of Apigee Edge? If so, would it be possible for you to use a different email address, sign up for a new trial (30-day limited) organization, and try the same thing there? If it works consistently from a different Edge organization, then it may be another indication of a bug in the organization provisioning for your original org. Trying it in a different environment in the same org is usually not enough.

2. I don't think it's your XSL. But maybe there is a cached XSLT processor that is being used incorrectly. Maybe you could avoid that by explicitly specifying the output encoding? like this?

   <xsl:output method="xml" indent="yes" encoding="utf-8" version="1.0" />

3. It seems clear from the trace (thanks for that) that the content is being correctly received by Apigee as UTF-8. Upon first receipt, it is displayed in the trace buffer appropriately. One thing I cannot see... the Response Headers. Let me ask you... why is the call to the backend system being made in a ServiceCallout ? If you had configured the backend as a target, we'd be able to see the response headers in the trace output. But because it's a ServiceCallout, we cannot. (though it would be a nice feature to add). Have you tried configuring it as a regular target?

[#3]

Hi Dino

Thank you for answering my question.

1. I just tested 30 times with the result: Passing 18 times while failing 12 times. The order was totally random. This could indicate a wrongly configured XSL engine. It should be noted that I had the same problem back in april 2015 (Please see https://community.apigee.com/questions/2814/utf8-encoding-of-xsltransform-failes-every-second.html#c... ) where the problem all of the sudden just disappeared.
2. This did not help unfortunatly.
3. I have now made a new Proxy where I change the TargetEndpoint instead of making ServiceCallout (propably a better way), and the problem does not seem to be present there. But this might just as well be since I am using other MP XSL engines correct?

Looking forward to your reply.

[DChiesa]

Hi Anders,

Hmmm, well my theory was that it could be a matter of misconfigured MPs. BUT, if you are seeing the problem only when using ServiceCallout, and not when using a TargetEndpoint, then my theory is disproved. But it would be more definitive if you were able to run the Java callout policy I suggested. Is that possible?

In any case, your new results points to a different kind of bug that is present only in the ServiceCallout code chain.

I will work up some tests to see if I can duplicate the behavior you're reporting.

In the meantime - is the approach of using a TargetEndpoint rather than a ServiceCallout, an acceptable workaround?

-Dino

[#5]

That might be the case, but do note that I also created a new proxy for testing. I did NOT change ServiceCallouts to TargetEndpoints on the same proxy, but it was on the same organisation.