Preventing crawlers such as Google indexing your dev portal

Oliver_Ogg
Participant II

Hi,

I've had a look in the docs, and the menu structure on our Dev Portal, but I can't see a way of preventing Google from indexing our Dev Portal. We might well like this in the future, but our current dev portal is not yet ready for the public.

How can I control this? Am I missing a setting, or should I be editing content templates?

Solved Solved
2 6 549
1 ACCEPTED SOLUTION

Dear @Oliver.Ogg ,

You can achieve same using multiple ways.

  • Restrict access to site using htaccess locking mechanism. If you are on cloud , out of the box it's available in pantheon.
  • Using Robots.txt in your developer portal root, please find more about it here and here.
  • Edit robots.txt with below content...
User-agent: *

Disallow: /
  • Apigee Developer Portal is built on top of famous open source CMS Drupal. See similar question answered here.

Cheers,

Anil Sagar

View solution in original post

6 REPLIES 6

Dear @Oliver.Ogg ,

You can achieve same using multiple ways.

  • Restrict access to site using htaccess locking mechanism. If you are on cloud , out of the box it's available in pantheon.
  • Using Robots.txt in your developer portal root, please find more about it here and here.
  • Edit robots.txt with below content...
User-agent: *

Disallow: /
  • Apigee Developer Portal is built on top of famous open source CMS Drupal. See similar question answered here.

Cheers,

Anil Sagar

Thanks, sorry I wasn't more clear. Robots.txt is definitely the thing I want to edit, but I don't seem to be able to do this from the Drupal User Interface. Is it a code level change?

Dear @Oliver.Ogg , Yes, It's a code level change.

Dear @Oliver.Ogg , Is your developer portal in Cloud or OnPrem ?

Thanks for the pointer to the code - thought it must be something within the Drupal UI. We're in the cloud, so I've made the change and pushed through. Thanks @Anil

Awesome, Glad issue is resolved. If you are in the cloud you can also lock your site using .htaccess. It's a configurable thing. You can do it using pantheon. Check here.

Also, If you feel above answer resolved your question, You can click on Accept link below the answer which marks it as best answer and will be helpful for others looking for same. Thank you 🙂