MilesWeb
MilesWeb Hosting Forum

Please login or register.

Login with username, password and session length

 

News:

SMF - Just Installed!


Author Topic: Stop website from being crawled by Google and other bots  (Read 6011 times)

0 Members and 1 Guest are viewing this topic.

Offline Shanaya

  • Administrator
  • Jr. Member
  • *****
  • Posts: 96
  • Karma: +1/-0
    • View Profile
Stop website from being crawled by Google and other bots
« on: April 22, 2014, 12:10:17 PM »
I would like to know what is the solution if I don't want a website to get indexed by any Google bots or other bots. Which is the best way of doing this?

I' have added the following code in the .htaccess file:

Header set X-Robots-Tag "noindex, nofollow, noarchive, nosnippet"

Is there any other way of doing this?
« Last Edit: April 28, 2014, 11:58:30 AM by Shanaya »

MilesWeb Hosting Forum

Stop website from being crawled by Google and other bots
« on: April 22, 2014, 12:10:17 PM »

Offline Brain Coffey

  • Jr. Member
  • **
  • Posts: 74
  • Karma: +1/-0
    • View Profile
Re: Stop website from being crawled by Google and other bots
« Reply #1 on: April 28, 2014, 12:00:05 PM »
The way that you mentioned is the best for stopping your website by getting crawled by Google and other bots. You can also try removing your website from Google Webmaster Tools.

Offline Kavita Menon

  • Newbie
  • *
  • Posts: 26
  • Karma: +0/-0
    • View Profile
Re: Stop website from being crawled by Google and other bots
« Reply #2 on: May 05, 2014, 01:17:12 PM »
You can also make use of the meta tags for blocking access to your website. In order to prevent all robots from indexing a page on your website, place the following meta tag into the <head> section of your web page:

<meta name="robots" content="noindex">

For allowing other robots to index the page on your site, preventing only Google's robots from indexing the web page:

<meta name="googlebot" content="noindex">

Offline Sarah

  • Newbie
  • *
  • Posts: 36
  • Karma: +0/-0
    • View Profile
Re: Stop website from being crawled by Google and other bots
« Reply #3 on: May 12, 2014, 02:46:07 PM »
Here's another technique:

In order to completely prevent a web page's contents from being listed in the Google web index even if other websites have linked to it. Use a 'noindex' meta tag. As long as Googlebot fetches the web page, it will come across the'noindex' meta tag and it will prevent that page from getting displayed in the web index.

Offline Shanaya

  • Administrator
  • Jr. Member
  • *****
  • Posts: 96
  • Karma: +1/-0
    • View Profile
Re: Stop website from being crawled by Google and other bots
« Reply #4 on: May 15, 2014, 05:49:56 PM »
Thanks everyone  :) That's like quite a lot of information and a great help for me.