Copy a file from proxy request to target rest service

Not applicable

Hi All,

I have a proxy that accepts an uploaded file as part of multipart/form-data request. I would like to pass the uploaded file on to a target rest server.

What is the best way to approach this? Currently, I'm using an assign-message policy that copies the incoming request and modifies it into the request that target expects i.e.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage async="false" continueOnError="false" enabled="true" name="PassthroughAssign">
    <DisplayName>PassthroughAssign</DisplayName>
    <Properties/>
    <Copy source="request">
        <Headers/>
        <FormParams/>
        <Payload/>
        <Verb/>
    </Copy>
    <Remove>
        <Headers>
            <Header name="jwt"/>
        </Headers>
    </Remove>
    <Add>
        <Headers>
            <Header name="x-facade-token">XXXX-XXXXXXXXXXX-XXXXXXXXXXXX</Header>
        </Headers>
    </Add>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="true" transport="http" type="request">target.passthru</AssignTo>
</AssignMessage>

...and then I use a callout policy to call the target rest service i.e.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ServiceCallout async="false" continueOnError="false" enabled="true" name="Call-File-Upload-Service">
    <DisplayName>Call File Upload Service</DisplayName>
    <Properties/>
    <Request clearPayload="false" variable="target.passthru">
        <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
    </Request>
    <Response>openTextResponse</Response>
    <HTTPTargetConnection>
        <Properties>
            <Property name="success.codes">1xx,2xx,3xx,4xx,5xx</Property>
        </Properties>
        <URL>http://{ot.host}/{ot.base.path}/workspace/{target.clientId}/{target.workspacePath}</URL>
    </HTTPTargetConnection>
</ServiceCallout>

However, the target server responds with an error 500 - "unexpected end of input".

Is there something I'm doing obviously wrong? Should I handle this differently?

Solved Solved
1 4 1,528
1 ACCEPTED SOLUTION

Not applicable

After much wrangling, I managed to solve the issue - actually a couple of issues

The quick one to resolve (after getting a view of the request sent out) was the fact that my content body was missing. This is because my initial <copy> from my question needed to look more like this:

<Copy source="request">
        <Headers/>
        <Payload>true</Payload>
        <Verb>true</Verb>
    </Copy>

I eventually shifted away from the copy and built a new request (due to a change of requirements)

The more difficult issue to resolve (and having a more annoying resolution) was that content length was being calculated incorrectly. The answer lay in my AssignMessage policy which looked as follows:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage name="PassthroughAssign">
    <DisplayName>PassthroughAssign</DisplayName>
    <Set>
        <Headers>
            <Header name="x-facade-token">XXXXXXXXXXXXXXXXXXX</Header>
            <Header name="Content-Type">multipart/form-data; boundary={boundary}</Header>
            <Header name="content-length">{multipartLength}</Header>
        </Headers>
        <Verb>POST</Verb>
        <Payload variablePrefix="@" variableSuffix="#">
            @multipart#
        </Payload>
    </Set>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="true" transport="http" type="request">target.passthru</AssignTo>
</AssignMessage>

I found out that the formatting inside the <Payload> tag was actually being added to the payload. i.e. apigee was sending "\n {contents of multipart}". So payload ended up larger than my contentLength header. Also, because there were spaces before the payload boundary, the target server would reject the message.

I changed the <Payload> XML to look as follows:

<Payload variablePrefix="@" variableSuffix="#">@multipart#</Payload>

That resolved my issues. Thanks for your assistance @Dino

View solution in original post

4 REPLIES 4

I don't see anything obviously wrong at first glance.

Is there a reason you want to use a ServiceCallout instead of just using the target entity within the Apigee Edge model? It's not "wrong" to do so, I'm just curious.

When debugging this sort of thing, it may be helpful to send the ServiceCallout to.... another API Proxy in Edge. Routing through that second proxy would give you the chance to see what is being sent to the backend. It will allow you to verify that the verb, headers, payload, and etc are all as expected. The proxy might not be a completely transparent proxy. You might need to pass the desired location in a new header... so the "passthrough" proxy would know where to proxy to.

"Unexpected end of input" suggests that the backend is looking for more. One possible scenario is that the boundary and payload are not being passed through correctly. What you want is something like

Content-Type: multipart/form-data; boundary=UNIQUE_BOUNDARY_STRING-31322

..and then within the payload, something like

--UNIQUE_BOUNDARY_STRING-31322
Content-Disposition: form-data; name="myuploadedfile"; filename="whatever.gif"
Content-Type: image/gif

GIF87a.............,...........D..;
--UNIQUE_BOUNDARY_STRING-31322--

If somehow the AssignMessage is not preserving the content-type header or is somehow modifying the payload, then it would cause the backend to choke. The inter-posed passthrough proxy I suggested above would allow you to see the outbound message clearly.

EDIT

In fact, I just tested this, and i see an empty payload body at the endpoint invoked by ServiceCallout. The payload is not being "passed through" with the AssignMessage + ServiceCallout.

I don't know why that would be the case. But I was able to workaround that problem by including a 2nd policy like this, following the first assignmessage:

<AssignMessage name="AM-Payload">
    <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
    <AssignVariable>
        <Name>passthru-message.content</Name>
        <Ref>request.content</Ref>
        <Value>BADDBEEF</Value>
    </AssignVariable>
</AssignMessage>

I tried including that AssignVariable into the first AssignMessage policy but it didn't satisfy. When included as a separate policy immediately following the first AssignMessage, it worked.

(Also you will have noticed that I don't use target. as the prefix for variables. There are various target.xxx variables set and used by Apigee Edge, and I don't like the idea of using the same prefix. So I used "passthru-message" as the variable name for the message that was being sent out by ServiceCallout. )

Hi, Thankyou for your detailed response.

The reason we're not using the target entity in many of our proxies is because many of our proxies compose a response from multiple targets (not in my particular case), but this is the model we have adopted.

I like your suggestion of having an intermediate proxy - and have actually changed my proxy to do that even though In the time since posting this question, I managed to get the developer of the target service to provide me with logs so I could see where my request was incorrect.

The current situation:

I build my request a lot more simply now using:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage name="PassthroughAssign">
    <DisplayName>PassthroughAssign</DisplayName>
    <Set>
        <Headers>
            <Header name="x-facade-token">XXXXXXXXXXXXXXXXXXX</Header>
            <Header name="Content-Type">multipart/form-data; boundary={boundary}</Header>
            <Header name="content-length">{multipartLength}</Header>
        </Headers>
        <Verb>POST</Verb>
        <Payload variablePrefix="@" variableSuffix="#">
            @multipart#
        </Payload>
    </Set>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="true" transport="http" type="request">target.passthru</AssignTo>
</AssignMessage>

I build up the payload in a javascript policy following the example given here

var content = context.getVariable("request.content")
var ctype = context.getVariable("request.header.content-type")

var boundaryPattern = new RegExp('^multipart/form-data; *boundary=(.+)$')
var boundary = boundaryPattern.exec(ctype)[1]

var multipart = ""

multipart += content
           + "Content-Disposition: form-data; name=\"additionalData1\""
           + "\r\n\r\n" + additionalData1 + "\r\n"
           + "--" + boundary+"\r\n"
           + "Content-Disposition: form-data; name=\"additionalData2\""
           + "\r\n\r\n" + additionalData2 + "\r\n"
           + "--" + boundary+"\r\n"
           + "Content-Disposition: form-data; name=\"additionalData3\""
           + "\r\n\r\n" + additionalData3 + "\r\n"
           + "--" + boundary+"--"

context.setVariable("multipart", multipart)
context.setVariable("boundary", boundary)
context.setVariable("multipartLength", byteLength(multipart))

When I debug the request, I can see that my content-length header gets altered i.e. if my calculated content-length comes to 741 (which I can verify in a tool like Postman by sending an identical request direct to the target), the debug in the intermediate proxy shows 761 - which seems obviously wrong.

In summary, there has been some movement, but I still have issues. Let me know if you've got further suggestions.

Not applicable

After much wrangling, I managed to solve the issue - actually a couple of issues

The quick one to resolve (after getting a view of the request sent out) was the fact that my content body was missing. This is because my initial <copy> from my question needed to look more like this:

<Copy source="request">
        <Headers/>
        <Payload>true</Payload>
        <Verb>true</Verb>
    </Copy>

I eventually shifted away from the copy and built a new request (due to a change of requirements)

The more difficult issue to resolve (and having a more annoying resolution) was that content length was being calculated incorrectly. The answer lay in my AssignMessage policy which looked as follows:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage name="PassthroughAssign">
    <DisplayName>PassthroughAssign</DisplayName>
    <Set>
        <Headers>
            <Header name="x-facade-token">XXXXXXXXXXXXXXXXXXX</Header>
            <Header name="Content-Type">multipart/form-data; boundary={boundary}</Header>
            <Header name="content-length">{multipartLength}</Header>
        </Headers>
        <Verb>POST</Verb>
        <Payload variablePrefix="@" variableSuffix="#">
            @multipart#
        </Payload>
    </Set>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="true" transport="http" type="request">target.passthru</AssignTo>
</AssignMessage>

I found out that the formatting inside the <Payload> tag was actually being added to the payload. i.e. apigee was sending "\n {contents of multipart}". So payload ended up larger than my contentLength header. Also, because there were spaces before the payload boundary, the target server would reject the message.

I changed the <Payload> XML to look as follows:

<Payload variablePrefix="@" variableSuffix="#">@multipart#</Payload>

That resolved my issues. Thanks for your assistance @Dino

great! glad to hear it.

BTW, you do not need the variablePrefix and variableSuffix. The default prefix and suffix are open and close curly. So you can do this:

<Payload>{multipart}</Payload>