Robots.txt is a text file which contains a set of instructions and tells the search engine crawlers or spider which parts of the website to be crawled or not to be crawled.

Who Voted for this Story





Comments


Log in to comment or register here.
Subscribe

Share your small business tips with the community!
Share your small business tips with the community!
Share your small business tips with the community!
Share your small business tips with the community!