How to handle unwanted bots and scripts in apigee

[sjm2000]

Dear All,

There are too many unwanted web crawlers(scraper), bots and scripts hitting 3 public api end points in apigee. Currently, they originate from just 2 ips. I am able to block them in apigee and send http 403 as response.

However what is the best practice for sending response code for unwanted web crawlers(scraper)/bots/scripts . Basically i want to discourage the web crawlers(scraper)/bots/scripts .

Is it

a. http 404

b. http 403

c. http 401

any suggestions ?

In addition, do we have any mechanism to send a delayed response for the bots?.

Thanks

Sujith

[#2]

the best practice is 404. This doesn't give any hint of possibilities. In my experience, I have seen use of 404 in most cases.

Sending delayed response makes more sense if the response from backend is delayed. At Apigee end delaying response may be difficult. You may use java callout and add sleep in that. In javascript it will not work.

[sjm2000]

Thanks for the response.

Regarding the delaying response via java , i presume that would create apigee "threads" to be blocked . I am wondering if there is any other way in apigee.

[#4]

Yes, the thread block will make the response slower. I have noticed larger payload also mkaes the response delayed. If you can make a request to process larger payload then it can increase response time.