Robots.txt generator is a Free SEO Tool used to generate robots.txt file instantly for your website. Whenever a search engine crawls any website, it always first looks for the robots.txt file that is located at the domain root level Ex : (www.example.com/robots.txt).
A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. The file give instructions about their site to web robots; this is called The Robots Exclusion Protocol. You can use it to prevent search engines from crawling specific parts of your website and to give search engines helpful tips on how they can best crawl your website. Although search engines understand robots.txt well, it is still a guideline and not a requirement.
Let's say , for example, you wanted to block all the bots except for Googlebot from crawling your site. Here's how you can go about it:
User-agent: * Disallow: / User-agent: Googlebot Allow: /
Programming without an overall architecture or design in mind is like exploring a cave with only a flashlight: You don’t know where you’ve been, you don’t know where you’re going, and you don’t know quite where you are.Danny Thorpe