Stop website from being crawled by Google and other bots

Started by Shanaya, April 22, 2014, 12:10:17 PM

Previous topic - Next topic
Print
Go Down Pages1
User actions

Shanaya

April 22, 2014, 12:10:17 PM Last Edit: April 28, 2014, 11:58:30 AM by Shanaya
I would like to know what is the solution if I don't want a website to get indexed by any Google bots or other bots. Which is the best way of doing this?

I' have added the following code in the .htaccess file:

Header set X-Robots-Tag "noindex, nofollow, noarchive, nosnippet"

Is there any other way of doing this?

Brain Coffey

#1
April 28, 2014, 12:00:05 PM
The way that you mentioned is the best for stopping your website by getting crawled by Google and other bots. You can also try removing your website from Google Webmaster Tools.

Kavita Menon

#2
May 05, 2014, 01:17:12 PM
You can also make use of the meta tags for blocking access to your website. In order to prevent all robots from indexing a page on your website, place the following meta tag into the <head> section of your web page:

<meta name="robots" content="noindex">

For allowing other robots to index the page on your site, preventing only Google's robots from indexing the web page:

<meta name="googlebot" content="noindex">

Sarah

#3
May 12, 2014, 02:46:07 PM
Here's another technique:

In order to completely prevent a web page's contents from being listed in the Google web index even if other websites have linked to it. Use a 'noindex' meta tag. As long as Googlebot fetches the web page, it will come across the'noindex' meta tag and it will prevent that page from getting displayed in the web index.

Shanaya

#4
May 15, 2014, 05:49:56 PM
Thanks everyone  :) That's like quite a lot of information and a great help for me.
Print
Go Up Pages1
User actions