How to extract multipart/form-data from POST request and convert it to query params

Not applicable

I have a request made to server like

curl -H 'Host: test.apigee.net' -H 'Content-Type: multipart/form-data; boundary=beachbodyboundarystring' -H 'Accept: application/json' -H 'Accept-Language: en-us'
-H 'User-Agent: BOD/9 CFNetwork/808.0.2 Darwin/16.0.0' --data-binary '--beachbodyboundarystring

Content-Disposition: form-data; name="sourceSystem" appletv --beachbodyboundarystring Content-Disposition: form-data; name="email" test@test.com ' --compressed 'https://test.apigee.net/user/callAPI'


I need to use Apigee to convert this POST into GET call and passing the email from FORM data as query string.
I am unable to extract the FORM data, I don't want to use Javascript as that is slowing the call by >1000 ms.

Here is what my policy looks like.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage async="false" continueOnError="false" enabled="true" name="ChangeRequestMethod">
  <DisplayName>ChangeRequestMethod</DisplayName>
  <Properties/>
  <Copy source="request">
    <Headers/>
    <QueryParams/>
    <FormParams>
      <FormParam name="sourceSystem"/>
    </FormParams>
    <Payload/>
    <Verb/>
    <StatusCode/>
    <ReasonPhrase/>
    <Path/>
  </Copy>
  <Remove>
    <FormParams>
      <FormParam name="email"/>
      <FormParam name="guid"/>
    </FormParams>
    <Payload/>
  </Remove>
  <Add>
    <QueryParams>
      <QueryParam name="email">{request.content}</QueryParam>
      <QueryParam name="guid">{request.formparam.sourceSystem}</QueryParam>
    </QueryParams>
  </Add>
</AssignMessage>
Solved Solved
1 9 42.2K
1 ACCEPTED SOLUTION

A couple things for you

  1. you cannot retrieve data out of request bodies in requests with content-type = multipart/form-data using {request.formparam.xxxx}
    This variable syntax allows you to retrieve data from content-type = application/x-www-form-urlencoded .
  2. There is something else wrong with your system if your JS callouts are taking 1000ms. Something else is amiss there. Maybe the code is doing something not quite right. Parsing multipart/form-data will not take that long, unless your postbody is gigabytes in length. is it? It looks to me like you have 2 parameters - sourceSystem and email. Why make this multi-part form? Why not use x-www-form-urlencoded? (simpler, much easier)

Let me know if I can help further. I will be glad to look at your code, if you care to share it.


Here's some additional help, in case you decide to stick with the multipart/form-data payload.

This JS can be used to naively process a multi-part form of the kind you showed. (It will not work for the general case!)

// processMultipartForm.js
// ------------------------------------------------------------------
//
// process a simple multipart form. This works only with single-line data items.
//

var ctype = context.getVariable('request.header.content-type');
var re1 = new RegExp('^multipart/form-data; *boundary=(.+)$');
var match = re1.exec(ctype);
var boundary = match[1];
var blength = boundary.length;
var body = context.getVariable('request.content');
var re2 = new RegExp('^--' + boundary + '$', 'gm');
var re3 = new RegExp('^Content-Disposition: form-data; name="(.+)"$[\r\n]+(.+)$', 'm');
var section;
var payload = {};

while ((section = re2.exec(body)) !== null) {
  var start = section.index + blength + 1;
  var blob = body.substring(start, section.lastIndex);
  match = re3.exec(blob);
  if (match) {
    context.setVariable('extracted_' + match[1], match[2]);
    payload[match[1]] = match[2];
  }
}

context.setVariable('resp_payload', JSON.stringify(payload, null, 2));

I just tried this and get these results. It takes about 5 ms for this entire proxy to run.

Request:

$ curl -i -X POST \
-H 'Content-Type: multipart/form-data; boundary=beachbodyboundarystring' \
-H 'Accept: application/json' \
-H 'Accept-Language: en-us'  \
-H 'User-Agent: BOD/9 CFNetwork/808.0.2 Darwin/16.0.0' \
--data-binary '
--beachbodyboundarystring
Content-Disposition: form-data; name="sourceSystem"

appletv
--beachbodyboundarystring
Content-Disposition: form-data; name="email"

test@test.com
--beachbodyboundarystring--' \
--compressed 'https://MYORG-test.apigee.net/multipart-form-1/t1'

Response:

HTTP/1.1 200 OK
Date: Sat, 14 Jan 2017 01:49:32 GMT
Content-Type: application/json
Content-Length: 61
Connection: keep-alive
Accept-Encoding: deflate, gzip
Accept-Language: en-us
Server: Apigee Router

{
  "sourceSystem": "appletv",
  "email": "test@test.com"
}

See the attached proxy bundle for working code.

apiproxy-multipart-form-1-20170113-175029.zip

View solution in original post

9 REPLIES 9

A couple things for you

  1. you cannot retrieve data out of request bodies in requests with content-type = multipart/form-data using {request.formparam.xxxx}
    This variable syntax allows you to retrieve data from content-type = application/x-www-form-urlencoded .
  2. There is something else wrong with your system if your JS callouts are taking 1000ms. Something else is amiss there. Maybe the code is doing something not quite right. Parsing multipart/form-data will not take that long, unless your postbody is gigabytes in length. is it? It looks to me like you have 2 parameters - sourceSystem and email. Why make this multi-part form? Why not use x-www-form-urlencoded? (simpler, much easier)

Let me know if I can help further. I will be glad to look at your code, if you care to share it.


Here's some additional help, in case you decide to stick with the multipart/form-data payload.

This JS can be used to naively process a multi-part form of the kind you showed. (It will not work for the general case!)

// processMultipartForm.js
// ------------------------------------------------------------------
//
// process a simple multipart form. This works only with single-line data items.
//

var ctype = context.getVariable('request.header.content-type');
var re1 = new RegExp('^multipart/form-data; *boundary=(.+)$');
var match = re1.exec(ctype);
var boundary = match[1];
var blength = boundary.length;
var body = context.getVariable('request.content');
var re2 = new RegExp('^--' + boundary + '$', 'gm');
var re3 = new RegExp('^Content-Disposition: form-data; name="(.+)"$[\r\n]+(.+)$', 'm');
var section;
var payload = {};

while ((section = re2.exec(body)) !== null) {
  var start = section.index + blength + 1;
  var blob = body.substring(start, section.lastIndex);
  match = re3.exec(blob);
  if (match) {
    context.setVariable('extracted_' + match[1], match[2]);
    payload[match[1]] = match[2];
  }
}

context.setVariable('resp_payload', JSON.stringify(payload, null, 2));

I just tried this and get these results. It takes about 5 ms for this entire proxy to run.

Request:

$ curl -i -X POST \
-H 'Content-Type: multipart/form-data; boundary=beachbodyboundarystring' \
-H 'Accept: application/json' \
-H 'Accept-Language: en-us'  \
-H 'User-Agent: BOD/9 CFNetwork/808.0.2 Darwin/16.0.0' \
--data-binary '
--beachbodyboundarystring
Content-Disposition: form-data; name="sourceSystem"

appletv
--beachbodyboundarystring
Content-Disposition: form-data; name="email"

test@test.com
--beachbodyboundarystring--' \
--compressed 'https://MYORG-test.apigee.net/multipart-form-1/t1'

Response:

HTTP/1.1 200 OK
Date: Sat, 14 Jan 2017 01:49:32 GMT
Content-Type: application/json
Content-Length: 61
Connection: keep-alive
Accept-Encoding: deflate, gzip
Accept-Language: en-us
Server: Apigee Router

{
  "sourceSystem": "appletv",
  "email": "test@test.com"
}

See the attached proxy bundle for working code.

apiproxy-multipart-form-1-20170113-175029.zip

I think this is a good option, let me plug this in and check the api RTT.
Thank you!

@Dino can you please share me any sample proxy , which will convert the JSON request message with multipartform/data(multiple boundaries as more than one documents will be uploaded) to SOAP with MTOM message.

Not applicable

@Amrita: use extract policy to retrieve value from the form parameters in the request.

<ExtractVariables name="Extract-Variables-1"> 
  <DisplayName>Extract Variables-1</DisplayName>
  <Properties/>
  <FormParam name="email">
    <Pattern ignoreCase="true">{email}</Pattern>
  </FormParam>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables> 
  <VariablePrefix>apigee</VariablePrefix>
</ExtractVariables>

then use assign message policy to set in the query parameters.

<AssignMessage name="ChangeRequestMethod">
  <DisplayName>ChangeRequestMethod</DisplayName>
  <Properties/>
  <Add>
    <QueryParams>
      <QueryParam name="email">{apigee.email}</QueryParam>
    </QueryParams>
  </Add>
</AssignMessage>

The problem is I don;t get any data in FormParam but only in Payload.

This ExtractVariables will work if the content-type is application/x-www-form-urlencoded. Her payload is multipart/form-data , which is different.

Hi @Dino, i am trying to retrieve data from multipart/form-data, but i am getting null values.

All the below values are resulting in null values in javascript resource,

context.getVariable ("request.formparam") - null

context.getVariable ("request.formparam.payload") - null

where payload is one of the form param name.

Hi - please don't ask new questions in comments attached to 18-month old answers.

If you have a new question, ask a new question.

Not applicable

Hi Amrita,

You can you do it in two steps because removal and assignation can not be done in same policy or you can use JS to do in single step.

<AssignMessage name="AddQueryParamEmail"> <DisplayName>AddQueryParamEmail</DisplayName> <Properties/> <Add> <QueryParams> <QueryParam name="email">{request.formparam.email}</QueryParam> </QueryParams> </Add> </AssignMessage>

<AssignMessage name="RemoveFormParamEmail"> <DisplayName>RemoveFormParamEmail</DisplayName> <Properties/> <Remove> <FormParams> <FormParam name="email"/> </FormParams> </Remove> </AssignMessage>