Loading...

Knowledge Base
Home 
 
/
 How to Use Robots.txt
Categories: cPanel_Hosting_RetailPlesk_Hosting_RetailVPS_Dedi_Linux_Server_W_O_Control_Panel_RetailDedi_Windows_Server_W_O_Control_Panel_RetailWordPress_Hosting_RetailCommon_FAQ_Retail

How to Use Robots.txt

Learn how to create, edit, and manage your robots.txt file to control which search engines can crawl your website.

What is robots.txt?

The robots.txt file is a simple text file placed in your website’s root directory. It tells search engines (like Googlebot, Bingbot, Yandex, Baidu) which parts of your site they are allowed or not allowed to crawl and index.

Where to Place robots.txt

  1. Access your website’s root directory (often called public_html or www).
  2. Check if a robots.txt file already exists. If not, create a new text file named robots.txt.
  3. Open the file with a text editor and add your rules.
  4. Save the file and ensure it is readable by web crawlers (permissions typically set to 644).

 

Common robots.txt Rules and Examples

 

PurposeCode ExampleDescription
Block all botsUser-agent: *
Disallow: /		Prevents all search engines from crawling your site.
Block specific foldersUser-agent: *
Disallow: /private/
Disallow: /tmp/		Blocks bots from selected directories only.
Allow all botsUser-agent: *
Disallow:		Allows all bots to crawl your site.

How robots.txt Works

Most search engines respect the rules in robots.txt, but some may ignore them. Legitimate bots (Googlebot, Bingbot) will follow your instructions, but rogue bots may not. For advanced control, use webmaster tools provided by search engines.

For more details, see Google's official robots.txt documentation.

 

Troubleshooting robots.txt Issues

  • If search engines aren’t following your rules, check for typos or incorrect file placement.
  • Make sure robots.txt is in the root directory and accessible (e.g., https://yourdomain.com/robots.txt).
  • Check file permissions (should be readable by everyone).
  • Some bots may ignore robots.txt; use additional security measures if needed.

Limitations and Notes

robots.txt does not guarantee privacy—it only requests bots not to crawl certain areas. Sensitive data should not be exposed on public URLs. For subdomains, place a separate robots.txt in each subdomain’s root directory.

 

Video Guide

Watch this video for a walkthrough on editing robots.txt:
https://youtu.be/5MgekA7codQ

 

For further queries, please feel free to reach out to our support team via the preferred channel.

Loading...