Log In Sign Up

webhostingtalk.com
robots.txt

Robots Exclusion Standard data for webhostingtalk.com

Resource Scan

Scan Details

Site Domain webhostingtalk.com
Base Domain webhostingtalk.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-26T03:40:42+00:00
Next Scan 2024-08-24T03:40:42+00:00

Last Successful Scan

Scanned2023-01-07T19:45:19+00:00
URL https://webhostingtalk.com/robots.txt
Redirect https://www.webhostingtalk.com/robots.txt
Redirect Domain www.webhostingtalk.com
Redirect Base webhostingtalk.com
Domain IPs 104.20.216.50, 104.20.217.50, 172.67.32.64, 2606:4700:10::6814:d832, 2606:4700:10::6814:d932, 2606:4700:10::ac43:2040
Redirect IPs 104.20.216.50, 104.20.217.50, 172.67.32.64, 2606:4700:10::6814:d832, 2606:4700:10::6814:d932, 2606:4700:10::ac43:2040
Response IP 104.20.217.50
Found Yes
Hash b05fce234dcc91209439d440ef1687401620ecb789ff43b7d8a088cf5f72212b
SimHash 6a095050c3c1

Groups

adbeat_bot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

Rule Path
Disallow /wp-admin/
Disallow /helpdesk/
Disallow /showthread.php?t=351529
Disallow /showthread.php?t=351432
Disallow /showthread.php?t=349120
Back to top

Other Records

Field Value
sitemap https://www.webhostingtalk.com/xmlsitemap.php
Back to top

Comments

  • User-agent: bingbot
  • Crawl-delay: 10
  • Disallow: /wp-includes/
  • Disallow: /wp-content/
  • Sitemap: https://www.webhostingtalk.com/sitemaps/sitemap_index.xml
  • Sitemap: https://www.webhostingtalk.com/xmlsitemap.php
Back to top