Officially known as "robots.txt," it is a text file that instructs web crawlers (robots or bots) to restrict access to specific web pages and directories.

Return to glossary list