Robots.txt Testing Tool - Free SEO Check
Robots.txt is a text file placed on web servers to instruct search engines and other web crawlers (also called robots or bots) which parts of a web page may be indexed and crawled and which may not. The robots.txt file is placed in the root directory of a website and can be read by the bots before crawling the website.
The robots.txt file consists of one or more "disallow" statements that specify certain directories, files, or entire sections of a web page that should not be indexed or searched. The syntax is relatively simple:
- "Disallow: /directory/" prevents search engines from crawling the specified directory.
- "Disallow: /page.html" prevents search engines from indexing and crawling a specified file.
- "Disallow: *" prohibits search engines from crawling and indexing the entire website.