MilesWeb
MilesWeb Hosting Forum

Please login or register.

Login with username, password and session length

 

News:

SMF - Just Installed!


Author Topic: What is Robots.txt file?  (Read 15433 times)

0 Members and 1 Guest are viewing this topic.

Offline Brain Coffey

  • Jr. Member
  • **
  • Posts: 74
  • Karma: +1/-0
    • View Profile
What is Robots.txt file?
« on: November 05, 2014, 11:35:15 AM »
Can anyone please tell me what exactly is a robots.txt file?

MilesWeb Hosting Forum

What is Robots.txt file?
« on: November 05, 2014, 11:35:15 AM »

Offline Drake Harrison

  • Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
    • View Profile
Re: What is Robots.txt file?
« Reply #1 on: November 20, 2014, 01:02:59 PM »
Robots.txt file is basically a text file that stops the web crawler software. For instance, with a robots.txt file, you can stop Googlebot from crawling some pages of your website. Robots.txt file consists of a list of commands like allow and disallow, as a result of which the web crawlers understand which URLs they can retrieve and which URLs they cannot retrieve.



Offline Shanaya

  • Administrator
  • Jr. Member
  • *****
  • Posts: 96
  • Karma: +1/-0
    • View Profile
Re: What is Robots.txt file?
« Reply #2 on: December 19, 2014, 12:32:06 PM »
You need to use a robots.txt file only if your website comprises of content that you don't want Google bots or any other search engine bots to index. If you want Google to index your entire website, you don't have to create a robots.txt file, not even an empty one.