What is the use of robots.txt file?

1 year ago
Digital Marketing

A robot.txt file helps Search Engines to crawl the webpages on your website.
Each website should have a robots.txt file to let the Search Engines know which pages it could crawl.
Robot.txt file allows you to disable some unwanted links on your website that you don’t want the search engine to crawl.
For example, if you have a WordPress website it can disable www.yoursite.com/wp-admin as it is a login page and isn’t required for search engines.

Sanisha Maharjan
Jan 15, 2022
More related questions

Questions Bank

View all Questions

Top