How to iterate through a JSON object to find/replace

Not applicable

Hello, this question is regarding parsing JSON to modify an undesired null syntax when instances are found.

I have a server which provides XML/SOAP responses. These responses then get converted to JSON using a XMLtoJSON policy. There are two ways the server sends back an indication of null data:

  1. Sometimes, the null XML field is returned as <ExampleNull></ExampleNull> or simply <ExampleNull/>. When passed through the XMLtoJSON policy, the output is the expected and desired form of "ExampleNull": null.
  2. Other times, the null XML field is returned as <ExampleNull i:nil="true"/>. In this case, after passing through the XMLtoJSON policy, the output is the undesired form:

"ExampleNull":

{

"@nil": true,

"TEXT": null

}

It is not an option for me to change the fields being returned in the SOAP response to exclude the attribute causing these issues.

Example response: (With undesired formatting)

{
  "Response": {
    "AccountId": "12345",
    "CompanyCode": 1,
    "CustomerName": "Joseph X. Schmoe",
    "EmailAddressList": {
	"Response.EmailAddressDTO": {
	  "AlertOptionList": null,
	  "ContactMethodSeqNum": 2,
	  "EmailAddress": null
	}
    },
    "MailingAddress": {
	"NonStandard": {
	  "@nil": true,
	  "TEXT": null
	},
	"Standard": {
	  "Address": "Example",
	  "DisplayAddressText": {
	    "@nil": true,
	    "TEXT": null
	  }
	}
    },
    "LastBill": null,
    "LastPayment": null
  }
}

My question is how would I go about recursively iterating through my JSON response to catch the instances of the case 2) null response type, and simply change the whole outer field to null? The reason for recursion is that the hierarchy of the response is different between different endpoints and I need to account for the variation.

The goal would be to turn the above simplified example into something like this:

{
  "Response": {
    "AccountId": "12345",
    "CompanyCode": 1,
    "CustomerName": "Joseph X. Schmoe",
    "EmailAddressList": {
	"Response.EmailAddressDTO": {
	  "AlertOptionList": null,
	  "ContactMethodSeqNum": 2,
	  "EmailAddress": null
	}
    },
    "MailingAddress": {
	"NonStandard": null,
	"Standard": {
	  "Address": "Example",
	  "DisplayAddressText": null
	}
    },
    "LastBill": null,
    "LastPayment": null
  }
}

I am open to various possibilities on how something like this can be fixed. If a JavaScript policy isn't my best bet, I'm all ears, but for now, I am trying to implement it in a JavaScript policy, and the content I have so far is:

 var resp_Object = JSON.parse(context.getVariable("response.content"));
 

 function checkAndFixNull(obj)
 {
    var k;
    if (obj instanceof Object)
    {
        if (obj.hasOwnProperty("@nil"))
        {
            obj = null;
            return obj;
        }
        for (k in obj)
        {   
            if (obj.k instanceof Object)
            {
                obj.k = checkAndFixNull(k);
            }
        }
    } else {}
    
    return obj;
 }

 
 var Edited_Response_Object = checkAndFixNull(resp_Object);
 context.setVariable("response.content", JSON.stringify(Edited_Response_Object));

This is not working, and I know a potential lead is that the typeof k returns a string, not an object, so it never descends into the deeper hierarchies. I have been stuck on this for quite some time and am hoping for some suggestions or help. Thank you.

Solved Solved
1 5 28.8K
1 ACCEPTED SOLUTION

I can think of two ways to solve the problem.

  1. Use an XSL policy to Transform the returned XML to translate <ExampleNull i:nil="true"></ExampleNull> into <ExampleNull/>
  2. Transform the JSON output of the XMLToJSON policy using a JavaScript policy that correctly walks the graph of the response object.

Which you prefer depends on your situation. The XSL policy does the recursion you want implicitly - that's how XSL works. So you may find it easier to do the replacement there. Or, You may prefer to write things in JavaScript, if you don't know XSL. JS may be easier to maintain in that case.

As an example of case 1, XSL

Suppose the XML returned is like this:

<Response xmlns:i='i'>
    <AccountId>12345</AccountId>
    <CompanyCode>1</CompanyCode>
    <CustomerName>Joseph X. Schmoe</CustomerName>
    <EmailAddressList>
      <Response.mailAddressDTO>
        <AlertOptionList/>
        <ContactMethodSeqNum>2</ContactMethodSeqNum>
        <EmailAddress/>
      </Response.mailAddressDTO>
    </EmailAddressList>
    <MailingAddress>
        <NonStandard i:nil='true'/>
        <Standard>
          <Address>Example</Address>
          <DisplayAddressText i:nil='true'/>
        </Standard>
    </MailingAddress>
    <LastBill/>
    <LastPayment/>
</Response>

Running that through this XSL:

<xsl:stylesheet version="1.0"
                xmlns:i="i"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output indent="yes" omit-xml-declaration="yes"/>
  <!-- pass through for all elements and attributes -->
  <xsl:template match="node()|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>
  <!-- for elements with a nil='true' attribute, strip the attribute -->
  <xsl:template match="*[@i:nil='true']">
    <xsl:element name="{local-name()}"/>
  </xsl:template>
</xsl:stylesheet>

...results in this:

<Response xmlns:i="i">
    <AccountId>12345</AccountId>
    <CompanyCode>1</CompanyCode>
    <CustomerName>Joseph X. Schmoe</CustomerName>
    <EmailAddressList>
      <Response.mailAddressDTO>
        <AlertOptionList/>
        <ContactMethodSeqNum>2</ContactMethodSeqNum>
        <EmailAddress/>
      </Response.mailAddressDTO>
    </EmailAddressList>
    <MailingAddress>
        <NonStandard/>
        <Standard>
          <Address>Example</Address>
          <DisplayAddressText/>
        </Standard>
    </MailingAddress>
    <LastBill/>
    <LastPayment/>
</Response>

If you then pass THAT ^^ through the XMLToJSON policy I think you will get what you desire.

For case 2, Post-Process in JavaScript

you need to walk the graph of the JSON hash. For that you can use logic like this:

// walkObj.js
// ------------------------------------------------------------------
var what = Object.prototype.toString;

function walkObj(obj, fn) {
  var wo = what.call(obj);
  if (wo == "[object Object]") {
    Object.keys(obj).forEach(function(key){
      fn(obj, key);
      var item = obj[key], w = what.call(item);
      if (w == "[object Object]" || w == "[object Array]") {
        walkObj(item, fn);
      }
    });
  }
  else if (wo == "[object Array]") {
    obj.forEach(function(item, ix) {
      fn(obj, ix);
    });
    obj.forEach(function(item, ix) {
      var w = what.call(item);
      if (w == "[object Object]" || w == "[object Array]") {
        walkObj(item, fn);
      }
    });
  }
}

function checkAndFixNull(parent, key) {
  var value = parent[key], w = what.call(value);
  if ((w == "[object Object]") && (value.TEXT === null) && (value['@nil'] === true)) {
    parent[key] = null;
  }
}

var source = JSON.parse(context.getVariable('request.content'));
walkObj(source, checkAndFixNull);

// source now contains the transformed JSON hash
context.setVariable('request.content', JSON.stringify(source, null, 2));

Supposing input JSON like this:

{
  "Response": {
    "AccountId": "12345",
    "CompanyCode": 1,
    "CustomerName": "Joseph X. Schmoe",
    "EmailAddressList": {
      "Response.EmailAddressDTO": {
        "AlertOptionList": null,
        "ContactMethodSeqNum": 2,
        "EmailAddress": null
      }
    },
    "MailingAddress": {
      "NonStandard": {
        "@nil": true,
        "TEXT": null
      },
      "Standard": {
        "Address": "Example",
        "DisplayAddressText": {
          "@nil": true,
          "TEXT": null
        }
      }
    },
    "LastBill": null,
    "LastPayment": null
  }
}

...the output is like this:

{
  "Response": {
    "AccountId": "12345",
    "CompanyCode": 1,
    "CustomerName": "Joseph X. Schmoe",
    "EmailAddressList": {
      "Response.EmailAddressDTO": {
        "AlertOptionList": null,
        "ContactMethodSeqNum": 2,
        "EmailAddress": null
      }
    },
    "MailingAddress": {
      "NonStandard": null,
      "Standard": {
        "Address": "Example",
        "DisplayAddressText": null
      }
    },
    "LastBill": null,
    "LastPayment": null
  }
}

View solution in original post

5 REPLIES 5

I can think of two ways to solve the problem.

  1. Use an XSL policy to Transform the returned XML to translate <ExampleNull i:nil="true"></ExampleNull> into <ExampleNull/>
  2. Transform the JSON output of the XMLToJSON policy using a JavaScript policy that correctly walks the graph of the response object.

Which you prefer depends on your situation. The XSL policy does the recursion you want implicitly - that's how XSL works. So you may find it easier to do the replacement there. Or, You may prefer to write things in JavaScript, if you don't know XSL. JS may be easier to maintain in that case.

As an example of case 1, XSL

Suppose the XML returned is like this:

<Response xmlns:i='i'>
    <AccountId>12345</AccountId>
    <CompanyCode>1</CompanyCode>
    <CustomerName>Joseph X. Schmoe</CustomerName>
    <EmailAddressList>
      <Response.mailAddressDTO>
        <AlertOptionList/>
        <ContactMethodSeqNum>2</ContactMethodSeqNum>
        <EmailAddress/>
      </Response.mailAddressDTO>
    </EmailAddressList>
    <MailingAddress>
        <NonStandard i:nil='true'/>
        <Standard>
          <Address>Example</Address>
          <DisplayAddressText i:nil='true'/>
        </Standard>
    </MailingAddress>
    <LastBill/>
    <LastPayment/>
</Response>

Running that through this XSL:

<xsl:stylesheet version="1.0"
                xmlns:i="i"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output indent="yes" omit-xml-declaration="yes"/>
  <!-- pass through for all elements and attributes -->
  <xsl:template match="node()|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>
  <!-- for elements with a nil='true' attribute, strip the attribute -->
  <xsl:template match="*[@i:nil='true']">
    <xsl:element name="{local-name()}"/>
  </xsl:template>
</xsl:stylesheet>

...results in this:

<Response xmlns:i="i">
    <AccountId>12345</AccountId>
    <CompanyCode>1</CompanyCode>
    <CustomerName>Joseph X. Schmoe</CustomerName>
    <EmailAddressList>
      <Response.mailAddressDTO>
        <AlertOptionList/>
        <ContactMethodSeqNum>2</ContactMethodSeqNum>
        <EmailAddress/>
      </Response.mailAddressDTO>
    </EmailAddressList>
    <MailingAddress>
        <NonStandard/>
        <Standard>
          <Address>Example</Address>
          <DisplayAddressText/>
        </Standard>
    </MailingAddress>
    <LastBill/>
    <LastPayment/>
</Response>

If you then pass THAT ^^ through the XMLToJSON policy I think you will get what you desire.

For case 2, Post-Process in JavaScript

you need to walk the graph of the JSON hash. For that you can use logic like this:

// walkObj.js
// ------------------------------------------------------------------
var what = Object.prototype.toString;

function walkObj(obj, fn) {
  var wo = what.call(obj);
  if (wo == "[object Object]") {
    Object.keys(obj).forEach(function(key){
      fn(obj, key);
      var item = obj[key], w = what.call(item);
      if (w == "[object Object]" || w == "[object Array]") {
        walkObj(item, fn);
      }
    });
  }
  else if (wo == "[object Array]") {
    obj.forEach(function(item, ix) {
      fn(obj, ix);
    });
    obj.forEach(function(item, ix) {
      var w = what.call(item);
      if (w == "[object Object]" || w == "[object Array]") {
        walkObj(item, fn);
      }
    });
  }
}

function checkAndFixNull(parent, key) {
  var value = parent[key], w = what.call(value);
  if ((w == "[object Object]") && (value.TEXT === null) && (value['@nil'] === true)) {
    parent[key] = null;
  }
}

var source = JSON.parse(context.getVariable('request.content'));
walkObj(source, checkAndFixNull);

// source now contains the transformed JSON hash
context.setVariable('request.content', JSON.stringify(source, null, 2));

Supposing input JSON like this:

{
  "Response": {
    "AccountId": "12345",
    "CompanyCode": 1,
    "CustomerName": "Joseph X. Schmoe",
    "EmailAddressList": {
      "Response.EmailAddressDTO": {
        "AlertOptionList": null,
        "ContactMethodSeqNum": 2,
        "EmailAddress": null
      }
    },
    "MailingAddress": {
      "NonStandard": {
        "@nil": true,
        "TEXT": null
      },
      "Standard": {
        "Address": "Example",
        "DisplayAddressText": {
          "@nil": true,
          "TEXT": null
        }
      }
    },
    "LastBill": null,
    "LastPayment": null
  }
}

...the output is like this:

{
  "Response": {
    "AccountId": "12345",
    "CompanyCode": 1,
    "CustomerName": "Joseph X. Schmoe",
    "EmailAddressList": {
      "Response.EmailAddressDTO": {
        "AlertOptionList": null,
        "ContactMethodSeqNum": 2,
        "EmailAddress": null
      }
    },
    "MailingAddress": {
      "NonStandard": null,
      "Standard": {
        "Address": "Example",
        "DisplayAddressText": null
      }
    },
    "LastBill": null,
    "LastPayment": null
  }
}

I am just wondering if this can be achieved by <Options> on the XML to JSON Policy..and if not..should there be support for it..

https://docs.apigee.com/api-platform/reference/policies/xml-json-policy#recognizenull

Because xsi:nilable is kind of an odd ball XML spec item and should be considered in the standard use case.

Good feedback. I think it is not possible with the Options today.

Thank you for your time and help, I very much appreciate it. I decided to go with the JavaScript version and it worked perfectly.

I will note that I tried the XSL version as well and that did not work for me. I admit I do not know XSL very much at all and so it could have come down to a few minor configuration tweaks that I was unable to notice.

Glad to hear you have solved it. Just FYI: The problem you had with the XSL approach probably comes down to namespaces. XSL is tricky.