How can we store, log or modify response when streaming is enabled ?

[adevegowda]

When streaming is enabled, I understand that response.content and message.content are not available for any of the policies as documented here:

http://apigee.com/docs/api-services/content/best-practices-api-proxy-design-and-development

The response payload size > 10 MB, so I have to use streaming. But I want to store the response, log it in Splunk/Loggly, modify the response or use only a part of the response in my API proxy on the response path. Since response.content and message.content are not populated (will be empty) .. Is there any other way to do this ?

[DChiesa · Accepted solution]

Yes, You could do it by writing your own streaming code in nodejs. In slightly more detail:

1. include a nodejs target in your API proxy
2. write some logic in nodejs that calls your "real" target, and streams the response back out to Edge. It can also do whatever else it wants to do, with the response

It's essential that the nodejs target not accumulate the full response in memory, otherwise you completely lose the memory efficiency benefit of streaming. So any transforms of the message payload must be done "chunkwise", not on the full message content.

The Script Target should look something like this:

<TargetEndpoint name="default">
  <ScriptTarget>
    <Properties>
      <Property name="response.streaming.enabled">true</Property>
      <Property name="request.streaming.enabled">true</Property>
    </Properties>
    <ResourceURL>node://app.js</ResourceURL>
  </ScriptTarget>
</TargetEndpoint> 

And then in your app, you can do this:

var express = require('express'),
    request = require('request'),
    app = express(),
    s = require('stream');

app.get("/some-path", function(req, response, next) {
  var handler = s.PassThrough();
  handler.on('data', function(data) {
    // handle the inbound data here, ...
    // possibly stream to a logfile or remote log system
    console.log('data :', data.toString());
  });
  handler.pipe(response);

  // initiate the request and pipe the response through the handler
  request('http://api.lo5.at/info?fribble=8,9,2,3')
    .pipe(handler);
});

// catch and return 404
app.use(function(req, res, next) {
  var payload = { message: "Not found" };
  res.status(404);
  res.json(payload);
});

var listener = app.listen(process.env.PORT | 8124, function() {
      var host = listener.address().address;
      var port = listener.address().port;
      console.log('listening at http://%s:%s', host, port);
    });

This is just nodejs streaming. You should be able to test it outside of Edge, and then just bundle it into your API proxy...

[adevegowda]

@Dino

Thanks a lot for your suggestion. Do you have any sample nodejs code that does this ? It will be very helpful if could share code that I could use it as a reference.

[DChiesa]

@AMAR DEVEGOWDA - I updated my answer with some sample code.

[adevegowda]

@Dino,

Many Thanks for taking time to put together the code. We will try it out and let you know.