Robots.txt

The robots.txt file is the primary way of telling site visitors where robots can look and which places within the site to avoid. The structure of the file is extremely simple, and it can even be created in a basic text editor.

With many content management systems, the document is created automatically. However, if we do it manually, robots.txt should be placed in the main directory of the site in such a way that the file is accessible to search engine bots at e.g. https://przykladowastrona.pl/robots.txt.

According to official sources, this is where the robots visiting a website first look. For this reason, it is strongly suggested that you ensure the presence of the file described – even if you do not intend to block any access by robots. The instructions contained in the rules can refer to individual pages as well as entire folders. They can also be directed towards specific robots.

Back to glossary