Tutorial: Service Virtualization - Dynamic Field Filtering of JSON responses

[DChiesa]

Let's talk about Service Virtualization

We sometimes speak of "service virtualization" - this means that the Apigee Edge proxy that you configure, modifies the response from the backend in some way, so that the app sending requests through Edge is seeing a different, virtualized service. This could mean different verb + resource pairs, different formats (transforming XML to JSON), or different payloads. It could also mean "server side mashups" where the API exposed from Edge calls multiple backend systems.

OK, What is Field Filtering?

Within the realm of Service Virtualization, sometimes you want to change the response payload. Specifically the backend may return a very large payload, and you'd like to winnow it down to the minimum required for the App, or for the API Product you're exposing.

Maybe it's a healthcare scenario and you want to eliminate some information for privacy purposes. Maybe it's a retail scenario and you want to eliminate internal part numbers or inventory-on-hand information. Maybe it's just because the client is a mobile app and you want to economize on the payload size.

We can call the general approach "Field Filtering", and you can do it within Apigee Edge. Very easily! This code repo provides an example that you can use, and extend or apply to your own scenario.

The logic for filtering fields (include or exclude) from a JSON hash is provided in a JavaScript callout - a bit of custom JavaScript that runs within an Edge policy.

There is one interesting method:

 function applyFieldFilter(action, obj, fields) {...}
   @action : 'include' or 'exclude'
   @obj : a JS hash
   @fields: an array of strings, referring to fields within the hash

Example 1: The Basics

Assume a JS hash like this:

{
  "prop1" : 7,
  "prop2" : [ 1, 2, 3, 4],
  "prop3" : {
    "key1" : "A",
    "key2" : null,
    "key3" : true
  }
}

With action = 'include' and the fields array like this: ['prop1', 'prop3.key1'] ...the output from the call to applyFieldFilter() will be a hash like so:

{
  "prop1" : 7,
  "prop3" : {
    "key1" : "A"
  }
}

Example 2: Arrays

Assume a JS hash like this:

{
  "prop1" : 7,
  "prop2" : [ 1, 2, 3, 4],
  "data" : [{
    "key1" : "A",
    "key2" : null,
    "key3" : true
  },{
    "key1" : "B",
    "key2" : "alpha",
    "key3" : false
  },{
    "key1" : "C",
    "key2" : "yertle",
    "key3" : false
  }]
}

With action = 'include' and the fields array like this: ['prop2', 'data.key1'] ...the output will be:

{
  "prop2" : [ 1, 2, 3, 4],
  "data" : [{
    "key1" : "A"
  },{
    "key1" : "B",
  },{
    "key1" : "C",
  }]
}

Example 3: Arrays and excluding

Assume the same JS hash as above. With action = 'exclude' and the fields array like this: ['prop2', 'data.key1'] ...the output will be:

{
  "prop1" : 7,
  "data" : [{
    "key2" : null,
    "key3" : true
  },{
    "key2" : "alpha",
    "key3" : false
  },{
    "key2" : "yertle",
    "key3" : false
  }]
}

The JavaScript that makes this possible is pretty straightforward. It just is a recursive forEach() on the Object.keys() of the root hash. Just walk the tree, and include or exclude properties based on the paths provided. You can see the JS here.

Integrating that into Edge

The interesting question is, how to integrate that behavior into Edge? Ideally you'd like to have the fields-to-filter specified on a per-product or per-Developer App basis. And that's exactly what we can do, using the Custom Attribute concept that is available for each entity under management in Apigee Edge.

For ONE API Product, you might attach a list of fields like prop1.field1, prop1,field2. Whereas for another API product, wrapping the same API Proxy, you would have a different set of fields. At runtime, the field filtering logic retrieves the values configured on the product and applies them dynamically.

You could also attach those attributes to the Developer App. Or, if you're using 3-legged authentication, to the user.

Below is a 7-minute screencast demonstrating this technique. Click it to view in this web page, or right click HERE to view it full-screen via a new browser tab.

I think this provides a remarkable capability within Apigee Edge proxies. I hope you find it useful! If you'd like to do this yourself, on your API Proxies, here's the code.

[anton]

Nice work!

[dchiesa1]

Glad you enjoyed it, Anton!

[anton]

Just rolled out something based on your article, works like magic 🙂

[liorkarol]

Hi @Dino, thank you very much for the tutorial. Is there another way to implement "Field Filtering" without using the javascript callout policy? Are there other built in policies that apigee suggests that can work?

[dchiesa1]

Hmm, well if you have an XML payload then you could use an XSL transform to modify the payload. The key thing you need to do is modify the payload, and the JS policy works nicely for that.

I suppose you could do it in Python or Java as well. JS seems more suited to this purpose, for JSON payloads though.

Any reason JavaScript is not suitable for your purposes?

[liorkarol]

@Dino Just wanted to update that the JavaScript works great. Another question, is it possible to make a conditional "Field Filtering"? Let's take your example from above - Let's assume I want to exclude Prop3 only if Prop3.key1 = "A", any suggestion on how it can be done easily?

[dchiesa1]

Hi Lior

Maybe, it's possible? It depends on what you want.

It is possible to do 'static' filtering; the case in which the fields to filter (Exclude or include) are not known at design time, but can be determined at runtime. In that case just set the array that is passed to the input function, dynamically. fieldThe same filter applies to all nodes within the original hash, and the value of the node doesn't affect it.

If you want the filtering on a node to depend on the value of the node at any point in the graph... well then I think you need to do something a little more elaborate than what I've produced here.

Probably you will need something analagous to XSL for JSON. As you can imagine, this is a question that has been asked before, from contexts not including Apigee Edge. https://stackoverflow.com/questions/1618038/xslt-equivalent-for-json . I like the sounds of the JOLT package, but I haven't tried it in a Java callout in Apigee Edge.

It should be pretty easy to try out, though.

[mdunker]

Hi @Lior Karol -- sorry for the late response, I just found this article. For conditional filtering, and other transformations, I've written a library called JSMapr that might help you with that. An example of it in use in a proxy is at https://github.com/apigeecs/soap-js-mapr. It will let you set up a series of rules to apply transformations to a JavaScript object.