A robots.txt file is a small text file at the root of your website that tells crawlers which parts of the site they may and may not fetch. It is simple, but a single wrong line can hide an entire site from search engines.
What Is robots.txt?
A plain text file placed in a site’s root directory that gives crawling instructions to search engine bots. See what is a TXT file.
What Is robots.txt For?
It guides crawlers away from areas you don’t want them to spend time on, such as admin areas, internal search results or endless filter pages. That helps manage crawl budget.
How to Create a robots.txt File
Write the rules in a text editor and save the file as robots.txt, or use our free robots.txt generator (Turkish). Upload it to the root of your domain so it is reachable at yourdomain.com/robots.txt. Files uploaded to subfolders are not valid.
robots.txt Syntax
User-agent: *
Disallow: /admin/
Disallow: /search/
Allow: /admin/public/
Sitemap: https://www.example.com/sitemap.xml- User-agent names the bot a rule applies to (
*means all). - Disallow blocks a path.
- Allow permits a path inside a blocked one.
- Sitemap points to your XML sitemap.
Which Pages Should Be Blocked?
Admin and login areas, internal search results, cart and checkout steps, endless filter combinations and other low-value pages. Don’t block pages you want indexed, or the CSS and JavaScript needed to render them.
robots.txt and the Sitemap
Listing your sitemap in robots.txt helps bots discover it faster, though you can also submit it in Search Console. See what is a sitemap.
Related Files
AI-focused guides like llms.txt are different from robots.txt: they guide, not block. See also technical SEO and our SEO consulting.
Frequently Asked Questions
Is a robots.txt file mandatory?
No. A site without robots.txt can still be crawled and indexed. But on large or complex sites it helps guide crawl behavior.
Does robots.txt completely hide a page from Google?
No. Robots.txt only blocks crawling; a page can still be known from other sources and indexed. To fully remove a page from search results, use a noindex meta tag or password protection.
Where should the robots.txt file be?
In the root directory of the website, reachable at sitename.com/robots.txt. Robots.txt files uploaded to subdirectories are not valid.
Is it required to list the sitemap in robots.txt?
No; you can also submit the sitemap in Google Search Console. But listing it in robots.txt can help bots discover it faster.
Can a wrong robots.txt harm SEO?
Yes. For example, accidentally using Disallow: / blocks crawling of the whole site and can seriously hurt organic visibility. Test after changes.
