Apigee URL encoding issue

[ketanpandya]

I've got an issue with encoding. The issue is that when an email with + is added as queryparam in AssignMessage, Apigee encodes + as 20 rather than 2B. I suppose the issue will be any queryparam with + in it. Is this correct behavior? What is the fix/workaround?

john.doe+test@gmail.com

get encoded as john.doe%20test%40gmail.com when calling backend rather than john.doe%2Btest%40gmail.com

<Set>
  <QueryParams>
    <QueryParam name="id">john.doe+test@gmail.com</QueryParam>
  </QueryParams>
</Set>

We are using Apigee edge cloud

[dchiesa1]

Hmmmmmm

I don't see the behavior you're describing.

Here's the result of a proxy I'm using:

$ curl -i https://$ORG-$ENV.apigee.net/assignmsg-2/f1
HTTP/1.1 200 OK
Date: Mon, 18 Nov 2019 16:30:53 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 932
Connection: keep-alive
Vary: Accept-Encoding
ETag: W/"3a4-yqrLqX9QMRgKl8Y4wfQ1NkL/T/4"
X-Cloud-Trace-Context: 19291c091702b1c32e93af4afdf9d086
Server: Google Frontend
{
  "url": "/?id=john.doe%2Btest%40gmail.com",
  "method": "GET",
  "headers": {
    "host": "buoyant-climate-208500.appspot.com",
    "x-forwarded-for": "104.132.51.85,34.82.153.38, 169.254.1.1",
    "x-forwarded-proto": "https",
     ....
  },
  "body": {}
}

Which looks to me like... the backend got john.doe%2Btest%40gmail.com

..which is what you wanted. Right?

Here's the policy I used in the request flow.

<AssignMessage name='AM-QueryParam-Unencoded'>
  <Set>
    <QueryParams>
      <QueryParam name="id">john.doe+test@gmail.com</QueryParam>
    </QueryParams>
  </Set>
</AssignMessage>

[ketanpandya]

Thanks for your response. It is strange. Attached is the proxy I used. Do you see anything unusual?

In trace it shows following after AssignMessage step

GET /echo2?id=john.doe%20test%40gmail.com

I invoked the end-point using postman and had header Content-Type = application/json; chartset=utf-8

If invoked with header header (without utf-8) Content-Type = application/json then encoding is okecho2-rev1-2019-11-18.zip

GET /echo2?id=john.doe%2Btest%40gmail.com

There was similar issue in past https://community.apigee.com/questions/38149/apigee-url-encodes-plus-sign-incorrectly.html

[dchiesa1]

ack

I can't download your attachment. I get an "access forbidden" message.

Can you try attaching again? In the meantime, here's mine.

apiproxy-assignmessage-encoding.zip

[ketanpandya]

echo2-rev1-2019-11-18.zip

if you cann't download, then will copy paste policy etc. Its pretty small

[ketanpandya]

My proxy is very basic. Checked yours and saw use of "encodeHTML" . Tried that and it does encoding of encoded.

FYI - we are using cloud based Apigee

[dchiesa1]

Try both flows. The f1 flow does not use encodeHTML. If I have understood what you want, that example works for you. The backend receives

/?id=john.doe%2Btest%40gmail.com

...which is what you wanted. Right?

[ketanpandya]

With

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage async="false" continueOnError="false" enabled="true" name="Assign-Message-1">
    <DisplayName>Assign Message-1</DisplayName>
    <Properties/>
    <Set>
        <QueryParams>
            <QueryParam name="id">{encodeHTML('john.doe+test@gmail.com')}</QueryParam>
        </QueryParams>
        <FormParams/>
    </Set>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

I get (seems like encoding of encoded)

Assign Message-1
GET /echo2?id=john.doe%26%23x2b%3Btest%26%23x40%3Bgmail.com

[ketanpandya]

with

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage async="false" continueOnError="false" enabled="true" name="Assign-Message-1">
    <DisplayName>Assign Message-1</DisplayName>
    <Properties/>
    <Set>
        <QueryParams>
            <QueryParam name="id">john.doe+test@gmail.com</QueryParam>
        </QueryParams>
        <FormParams/>
    </Set>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

I get

Assign Message-1
GET /echo2?id=john.doe%20test%40gmail.com

[dchiesa1]

"I get" means... what?

Where do you "get", or see, this information?

Are you observing this in the Trace UI? Or are you seeing this in the upstream/backend ?

If you are observing this in the trace UI, don't rely on it. Check what the upstream actually receives.

2nd. Have you tried my proxy?

I've told you it works and I uploaded it. Did you try it? What results did you see? Do you see the results that I see?

[ketanpandya]

results.txtThanks for response.

1. I did upload the proxy you provided and it is not working as expected. Please try it out with URLs in the file. Seems that Content-Type changes how encoding of url is done. Is that correct?

2. My observations that I put it under "I get" were from Trace UI

3 URLs and results in attached file. Please try it out and see what results you observe

results.txt

[dchiesa1]

What problem are we solving? I've lost track.

I thought your goal was to accomplish a specific thing, specifically: find the answer to the question, "how can I send as a query parameter, an email that has an embedded plus sign?" I think I answered that question. Have we solved the problem?

If that is not what you're trying to do, then what?

• is your goal to run through a set of test cases and check whether the results fit your expectations?
• Something else?

I see you have conducted a bunch of different tests. What is your goal here?

[ketanpandya]

The objective still the same - being able to have query param (email) with plus sign in value. The problem is/was that when request is made with header Content-Type: application/json; charset=utf-8, the encoding for plus sign is incorrect.

Problem: query param with value john.doe+test@gmail.com

john.doe%20test%40gmail.com (this is what we get -incorrect value)

john.doe%2Btest%40gmail.com (it should be this value - correct value)

The problem is not solved. I have tried your proxy (please see attached file with result.txt).

results.txt

The purpose of tests was to illustrate the issue.

[dchiesa1]

I think you're saying,

if you send a GET request with a Content-type header, then ... Apigee does not behave the way it ought to behave.

That sounds like a weird bug in Apigee. I've created a bug on your behalf. b/145164863

A suggested workaround to that is:

Don't send a content-type header with a request that has no content.

Content-Type is intended to communicate the type of the payload, and as you know there is no payload in a GET, therefore the content-type header is not needed, and some might say it is incorrect for a client to include it in a GET request.

But it's possible the behavior you are describing might also apply to POST requests. And in any case it is also true that the encoding of query params sent from Apigee to a target should not be dependent upon the Content-type header sent inbound.

[ashwithds123]

Hi @dino-at-Google,

Is this issue fixed in Apigee?

This can be fixed in MessageProcessor level, can we fix it in proxy level?

Concern - Query param contains the "%" char in the value but Apigee is encoding it to "%25".

[dknezic]

%25 is the correct hex for a % character.

[ashwithds123]

@dane knezic - My ask is Apigee should pass "%" as is to the target endpoint without converting it to "%25". Hope you understand my concern?

[dknezic]

That would be the wrong thing to do, please refer to https://tools.ietf.org/html/rfc3986#section-2.1

[dchiesa1]

I agree with Dane. Also, if you have a new question you should ask a new question. If you like you can reference this existing one, if it's relevant and similar.

Otherwise your question and the back-and-forth gets buried in a deep thread of comments. That's not helpful for anyone.

If you have a new question, please ask a new question.

[kurtkanaskie]

I know this is old, but, someone else asked about Apigee X always encoding the query params sent to a backend service.

When you create a query param with a special character, Apigee URL encodes it before sending it to the backend.

After trying a few things, it appears that Apigee cannot be coerced into behaving like an "improper" client.

Using an Assign Message correctly encodes the query param: "id=jane+doe@any.com" to: "id=john.doe%2Btest%40gmail.com"

I even tried setting the target.url directly, which correctly sets the value in DEBUG to:
"https://mocktarget.apigee.net/echo?id=jane+doe@any.com"

But the URL actually sent becomes encoded and low and behold, incorrectly to:
"https://mocktarget.apigee.net/echo?id=jane%20doe%40any.com"
Notice it is using "%20" instead of "%2B" 

Bottom line, use Assign Message to set query params and expect that Apigee will properly URL encode them, as any "proper" client should.

[dchiesa1]

Kudos to you for curating the old stuff!  

Salute to the maintainers!