How to use the RegularExpressionProtection policy to scan Form Parameters for special characters?

[muzamilbcs]

I have created a proxy that accepts only post requests and passing any four parameters in the body like FirstName, LastName, Emp_Id, Email which is successfully created.

Now the question is that I want to apply the regex to avoid special characters  on parameters that are passing  in the body  

And then apply checks Mandatory to check the number of parameters i.e. if it is set  to four then it should make sure that it is four otherwise just display an error message

[dchiesa1 · Accepted solution]

Yes, this is pretty straightforward to do with the builtin RegularExpressionProtection policy.  I looked in the documentation and didn't see an example specifically for this case.  so here it is.

<RegularExpressionProtection name="REP-Formparam">
  <Source>request</Source>
  <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
  <FormParam name="firstname">
    <Pattern><![CDATA[[-+=$%&]]]></Pattern>
  </FormParam>
  <FormParam name="lastname">
    <Pattern><![CDATA[[-+=$%&]]]></Pattern>
  </FormParam>
</RegularExpressionProtection>

OK, what's going on there? 

The Source element tells the REP policy to check the request message. That's the usual case, checking the inbound request. 

There are two FormParam elements, one for each form parameter we want to check. Both use the same Pattern.  BTW it is possible for you to include multiple distinct patterns under each FormParam. The pattern itself is [-+=$%$] . If you speak regex, that means "match any character in the set between the two square brackets.  But because & need to be escaped when embedded within XML, I wrap the entire pattern in a CDATA section, which just tells the XML processor, the thing inside this cdata section is just plain text, don't look for encoded entities here.

Results when applied to an inbound message: 

$ curl -i $endpoint/regex-formparams/t1 -d firstname=Chris -d lastname=Shatner
HTTP/2 200 
content-length: 23
content-type: application/json
apiproxy: regex-formparams r1
x-request-id: 072393ac-f5a8-47de-a835-bc35bd307d8c
date: Thu, 07 Oct 2021 15:12:03 GMT
server: apigee
via: 1.1 google, 1.1 google
alt-svc: clear

{
    "status" : "ok"
}

$ curl -i $endpoint/regex-formparams/t1 -d firstname=Chris -d lastname=-hatner
HTTP/2 500 
content-type: application/json
apiproxy: regex-formparams r1
x-request-id: 1a53e73e-ec50-4d10-9553-58c93aa627b0
content-length: 173
date: Thu, 07 Oct 2021 15:12:05 GMT
server: apigee
via: 1.1 google
alt-svc: clear

{"fault":{"faultstring":"Regular Expression Threat Detected in REP-Formparam: regex: [-+=$%&] input: -hatner","detail":{"errorcode":"steps.regexprotection.ThreatDetected"}}}

The example API proxy is attached. Try it in your own org and environment.

I'll get the documentation updated to include this as an example.

[dchiesa1 · Accepted solution]

I understand what you're telling me.

I can't help you unless you provide some further information about your exact policy configuration, as well as where you have the policy attached. This is XML configuration files. Show them here. It's nice if you can format them as code - use the toolbar in the editor.

What would help even more is if you attach a trace session for your proxy that isn't working. It's an XML file.  To get it:

• deploy your proxy
• start your trace session
• send in a request
• go back to the trace session and download it
• attach it here.

[dchiesa1 · Accepted solution]

Yes, this is pretty straightforward to do with the builtin RegularExpressionProtection policy.  I looked in the documentation and didn't see an example specifically for this case.  so here it is.

<RegularExpressionProtection name="REP-Formparam">
  <Source>request</Source>
  <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
  <FormParam name="firstname">
    <Pattern><![CDATA[[-+=$%&]]]></Pattern>
  </FormParam>
  <FormParam name="lastname">
    <Pattern><![CDATA[[-+=$%&]]]></Pattern>
  </FormParam>
</RegularExpressionProtection>

OK, what's going on there? 

The Source element tells the REP policy to check the request message. That's the usual case, checking the inbound request. 

There are two FormParam elements, one for each form parameter we want to check. Both use the same Pattern.  BTW it is possible for you to include multiple distinct patterns under each FormParam. The pattern itself is [-+=$%$] . If you speak regex, that means "match any character in the set between the two square brackets.  But because & need to be escaped when embedded within XML, I wrap the entire pattern in a CDATA section, which just tells the XML processor, the thing inside this cdata section is just plain text, don't look for encoded entities here.

Results when applied to an inbound message: 

$ curl -i $endpoint/regex-formparams/t1 -d firstname=Chris -d lastname=Shatner
HTTP/2 200 
content-length: 23
content-type: application/json
apiproxy: regex-formparams r1
x-request-id: 072393ac-f5a8-47de-a835-bc35bd307d8c
date: Thu, 07 Oct 2021 15:12:03 GMT
server: apigee
via: 1.1 google, 1.1 google
alt-svc: clear

{
    "status" : "ok"
}

$ curl -i $endpoint/regex-formparams/t1 -d firstname=Chris -d lastname=-hatner
HTTP/2 500 
content-type: application/json
apiproxy: regex-formparams r1
x-request-id: 1a53e73e-ec50-4d10-9553-58c93aa627b0
content-length: 173
date: Thu, 07 Oct 2021 15:12:05 GMT
server: apigee
via: 1.1 google
alt-svc: clear

{"fault":{"faultstring":"Regular Expression Threat Detected in REP-Formparam: regex: [-+=$%&] input: -hatner","detail":{"errorcode":"steps.regexprotection.ThreatDetected"}}}

The example API proxy is attached. Try it in your own org and environment.

I'll get the documentation updated to include this as an example.

[muzamilbcs]

hi,
It's still not working properly, I have just applied a solution in RegularExpressionProtection Policy but facing the same issue when I send a post request with body parameters having special characters I.e.  +muzamil, -hussain, $address, 1666@ then I get 200 OK response
My question is still there I want to apply restriction no special characters passed in parameters if it passes then just get an error

[dchiesa1 · Accepted solution]

I understand what you're telling me.

I can't help you unless you provide some further information about your exact policy configuration, as well as where you have the policy attached. This is XML configuration files. Show them here. It's nice if you can format them as code - use the toolbar in the editor.

What would help even more is if you attach a trace session for your proxy that isn't working. It's an XML file.  To get it:

• deploy your proxy
• start your trace session
• send in a request
• go back to the trace session and download it
• attach it here.

[muzamilbcs]

Hi,

It's working now,
but it does not fulfill the third requirement.
i.e. the third requirement is that it applies mandatory checks on parameters like I want to apply three parameters then it must b ensure  that parameters are three, no more than three, and no less than three,
I think it is resolved by 'raise fault policy' by define conditions but don't know how to write conditions for this.

[dchiesa1 · Accepted solution]

>  I want to apply three parameters then it must b ensure  that parameters are three, no more than three, and no less than three,

Maybe you want a Condition to check for the presence of specific parameters?

<Step>
  <Name>RF-Missing-Parameters</Name>
  <Condition>request.queryparam.a = null OR request.queryparam.b = null OR request.queryparam.c = null</Condition>
</Step> 

The definition of RF-Missing-Parameters is left to you. That should be a RaiseFault policy. You can issue whatever error code and message you like. Probably a 400 or 4xx status. 

This is completely independent of the check you configure in the RegularExpressionProtection policy. 

[muzamilbcs]

The Above condition is valid for query params can you please rewrite it to apply to body parameters.

 
body (raw) json formate 

[dchiesa1]

The various fields of a JSON payload are not available "automatically" as context variables, as are query params.  With a query param, I can refer to a variable like request.queryparam.a and refer to the query param "a".  But there is no analogous mechanism for JSON payloads. 

The full text of the request body is available as request.content.  But that variable doesn't let you peek into the JSON fields. To do that you will need to use something like an ExtractVariables or a JavaScript step to explicitly parse the JSON payload and set your own variables. Then you can test the value of those variables in the flow. 

So the solution is 2 pieces: 

1. ExtractVariables or JavaScript to extract fields from the JSON into custom variables that you specify
2. A Condition of the form I showed above, but it tests your custom variables.

Search the community or read the documentation for how to do part #1.  The second part should be easy.