Log In Sign Up

suggestions.xhamster.com
robots.txt

Robots Exclusion Standard data for suggestions.xhamster.com

Resource Scan

Scan Details

Site Domain suggestions.xhamster.com
Base Domain xhamster.com
Scan Status Ok
Last Scan2024-05-22T23:42:24+00:00
Next Scan 2024-06-21T23:42:24+00:00

Last Scan

Scanned2024-05-22T23:42:24+00:00
URL https://suggestions.xhamster.com/robots.txt
Domain IPs 104.17.27.92, 104.17.28.92, 104.17.29.92, 104.17.30.92, 104.17.31.92, 2606:4700::6811:1b5c, 2606:4700::6811:1c5c, 2606:4700::6811:1d5c, 2606:4700::6811:1e5c, 2606:4700::6811:1f5c
Response IP 104.17.29.92
Found Yes
Hash 28aa87457557ddce43a5a7080deed74eb5ececfcdce2244791acdee03046d9e2
SimHash 875a0f11dce5

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

turnitinbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /admin/
Disallow /dashboard/
Disallow /assets/
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • MSN bot is prone to over-crawling
  • YandexBot bot is prone to over-crawling
  • 80legs is often abused
  • Disabled due to abuse
  • Disabled due to abuse
  • Disabled due to abuse
  • Please don't crawl the API
Back to top

Warnings

  • 2 invalid lines.
Back to top