Node > Express > Compress with gzip returns garbage

[#1]

When I run locally I get gzipping just fine and the response is readable. But plugging it into apigee I get a bunch of junk returned. Even though I get a good response header...

http://dev.api.meredith.com/rest/application/v1/advertisement/gpt?callback=adSetup&adSlot=div-gpt-sq...

[venkataraghavan]

@jheister

I did run the call and what looked like 'valid values'. Could you please elaborate on what you mean by garbage ? What is the expected result ?

Also, Could you try running out of apigee using trireme :

https://github.com/apigee/trireme

and see if there is any difference in response from node ? (Apigee uses trireme underneath)

[john_young]

@jheister - Check to see if you're getting things gzipped twice, that's a problem I'm currently looking at (sorry, no solution yet!)

[sudheendras]

Hey @jheister Did you try the compression module?

Looks like the JSON being returned by this end-point is invalid.

I created a simple test node.js proxy using Compression and Express modules (attached). I used your JSON and returned the same by compressing it.

var express = require('express');
var compression = require('compression');
var app = express();
app.use(compression());


var response={
	"gptAds": {.....


//Entire JSON

};

app.get('/', function (req, res) {
  res.set('Content-Encoding', 'gzip');
  res.set('Content-Type', 'application/json');
  res.send(response);
});

app.listen(3000, function () {
  console.log('Example app listening on port 3000!');
});

I see it generating valid response and headers..

Hope this help!

gziptest-rev1-2016-03-18.zip

[#5]

The problem seems to be double gZip encoding when using a ResponseCache policy on the endpoint. When ResponseCache is disabled, the compression middleware works fine.

[#6]

i sware this was a bug we reported in 14.07 and got fixed in 15.03...

[#7]

I'm definitely still seeing the issue when I enable the ResponseCache on my endpoint. I have a support ticket open, and Apigee engineers have confirmed the problem. Without the ResponseCache, it's fine.

[adas]

@Christa Dickson thanks for reporting the problem. We are aware of it and actively working on fixing the problem. There seems to something wrong with the cache policy that its doing a gzip of a gzipped response stored in cache based on the accept-encoding header.

For the benefit of everyone else following this thread, there were two issues with nodejs+gzip:

1. If streaming is enabled

2. If streaming is not enabled

The earlier issue that we had addressed was for the streaming enabled usecase where response cache doesn't come into the picture. The second one was recently fixed as @Christa Dickson mentioned and only rolled out to few customers who reported the problem m. This fix is working when response cache is not used, with response cache there's a problem that has been identified and we are working on a fix. This may get addressed in the next cloud release. I thought I would give an update so that everyone is aware of the situation and knows the workaround.

[#9]

Thanks for the update, Arghya.