An issue with Extended ascii characters when using node and apigee.

[#1]

Hi All,

I have an issue using an apigee JavaScript policy along with with hosted node app. I have created this little node app that to returns the payload posted demonstrate.

var express = require('express');
var bodyParser = require('body-parser');
var app = express();
app.use(bodyParser.json());
app.use(function (req, res) {
    res.setHeader('Content-Type', 'text/plain');
    res.write('you posted:\n');
    res.end(JSON.stringify(req.body, null, 2));
});
app.listen(3000);

Payload with extended ascii character

{
    "name": "€"
}

Works fine as long as no policies touch the payload in apigee.

you posted:
{
  "name": "€"
}

Adding the following simple JavaScript policy ( to basically parse, stringify and replace the payload) changes the behavior.

(Another policy updates Content-Length)

var inboundPayload = context.getVariable("request.content");
print("\n inbound: " + inboundPayload);

var parsedPayload = JSON.parse(inboundPayload);
print("\n parsed: " + inboundPayload);

var stringifiedPayload = JSON.stringify(parsedPayload);
print("\n string: " + inboundPayload);

context.setVariable("request.content", stringifiedPayload);
context.setVariable("hermes.flow.content.length", stringifiedPayload.length);

The app still works fine with standard ascii characters but the following error message is received from node when using the payload above.

[object Object] at onEnd (/organization/environment/api/node_modules/body-parser/node_modules/raw-body/index.js:298)

Thanks in advance for any ideas

Carl

[ozanseymen · Accepted solution]

Hi @carl.robinson

Line 298 in raw-body/index.js is:

done(createError(400, 'request size did not match content length', 'request.size.invalid', { 

And the actual problem is that string.length property does not count multi-byte sequences in UTF-8 so you end up with an incorrect Content-Length header value.

Put this function in your JS file (to the top of the file):

function getLengthInBytes(str) {
    var s = str.length;
    for (var i=str.length-1; i>=0; i--) {
        var code = str.charCodeAt(i);
        if (code > 0x7f && code <= 0x7ff) s++;
        else if (code > 0x7ff && code <= 0xffff) s+=2;
        if (code >= 0xDC00 && code <= 0xDFFF) i--; //trail surrogate
    }
    return s;
}

And change the line where you set the content.length variable to:

context.setVariable("hermes.flow.content.length", getLengthInBytes(stringifiedPayload));

I've tried above changes locally and it is working as expected.

For more length calculation approaches: http://stackoverflow.com/questions/5515869/string-length-in-bytes-in-javascript

[ozanseymen]

Also you don't have to set a length variable and use another policy to set the content-length. You can directly do:

context.setVariable("request.header.Content-Length", getLengthInBytes(stringifiedPayload));