What is robots.txt?

robots.txt (consistently lowercase) is an ASCII-encoded text record put away in the root catalog of a site that tells web search tool crawlers (otherwise called web bugs) what content on the webpage ought not be ordered via web search tools and what can be filed.

Why use robots.txt

Many individuals may not need web search tools to incorporate however much as could be expected, why we likewise deny web indexes to incorporate a portion of our substance?

The first is to forestall Spider to visit some disconnected pages, causing tension on the worker.

The second can forestall the web crawler record some copy pages, with the goal that the heaviness of the website more thought, work on the positioning of the webpage.

WordPress blog how to utilize robots.txt

Coming up next is the current robots.txt document utilized by my blog.

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /author/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/

Momentarily, it just records the indexes that are taboo to be gotten to.

/cgi-canister/: This is the worker’s cgi registry, so it most certainly can’t be recorded.

/wp-administrator/: WordPress backend catalog, no compelling reason to file.

/wp-incorporates/: some fundamental libraries for WordPress program running, no compelling reason to file.

The/modules/(modules),/subjects/(topics) and/store/(reserve) in the/wp-content/registry ought not be filed.

The/wp-content/catalog is left with the fundamental/transfers/registry for pictures and connections, which ought to be ordered via web search tools, particularly picture web crawlers. The remainder of your custom registries, contingent upon your own necessities to set whether to let web crawlers file.

/writer/is a rundown of the multitude of articles of a specific writer, which is fundamentally copy content, so it isn’t permitted to be ordered all things considered.

/trackback/and */trackback/, the trackback catalog and the first text is essentially something similar, the specific copy content.

/feed/and */feed/, the feed is additionally copy content.

/remarks/and */remarks/, the message page is additionally copy content.

You can alter the substance of this record without help from anyone else to modify your site, then, at that point save it as robots.txt and transfer it to the foundation of your site.