I would like to know what is the solution if I don't want a website to get indexed by any Google bots or other bots. Which is the best way of doing this?
I' have added the following code in the .htaccess file:
Header set X-Robots-Tag "noindex, nofollow, noarchive, nosnippet"
Is there any other way of doing this?
The way that you mentioned is the best for stopping your website by getting crawled by Google and other bots. You can also try removing your website from Google Webmaster Tools.
You can also make use of the meta tags for blocking access to your website. In order to prevent all robots from indexing a page on your website, place the following meta tag into the <head> section of your web page:
<meta name="robots" content="noindex">
For allowing other robots to index the page on your site, preventing only Google's robots from indexing the web page:
<meta name="googlebot" content="noindex">
Here's another technique:
In order to completely prevent a web page's contents from being listed in the Google web index even if other websites have linked to it. Use a 'noindex' meta tag. As long as Googlebot fetches the web page, it will come across the'noindex' meta tag and it will prevent that page from getting displayed in the web index.
Thanks everyone :) That's like quite a lot of information and a great help for me.