Log In Sign Up

borsahub.com
robots.txt

Robots Exclusion Standard data for borsahub.com

Resource Scan

Scan Details

Site Domain borsahub.com
Base Domain borsahub.com
Scan Status Ok
Last Scan2024-10-10T08:12:25+00:00
Next Scan 2024-10-17T08:12:25+00:00

Last Scan

Scanned2024-10-10T08:12:25+00:00
URL https://borsahub.com/robots.txt
Domain IPs 78.142.208.148
Response IP 78.142.208.148
Found Yes
Hash aaae57aa67199b131333e835de2743aad323e07473543ef742fe32990eda9625
SimHash 6836da5066e2

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Allow /wp-admin/admin-ajax.php

badbot

Rule Path
Disallow /
Disallow /cgi-bin/
Disallow /tmp/
Disallow /junk/
Disallow /private.php
Disallow /config.php

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.borsahub.com/sitemap.xml
Back to top

Comments

  • Allow all bots to access the following files
  • Sitemaps
  • Blocking specific bots
  • Crawl delay for all bots
  • Specific directories to block
  • Specific files to block
Back to top