idolpixels.net
robots.txt

Robots Exclusion Standard data for idolpixels.net

Resource Scan

Scan Details

Site Domain idolpixels.net
Base Domain idolpixels.net
Scan Status Ok
Last Scan2024-09-17T13:28:21+00:00
Next Scan 2024-10-17T13:28:21+00:00

Last Scan

Scanned2024-09-17T13:28:21+00:00
URL https://idolpixels.net/robots.txt
Domain IPs 104.21.234.50, 104.21.234.51, 2606:4700:3038::6815:ea32, 2606:4700:3038::6815:ea33
Response IP 104.21.234.50
Found Yes
Hash 159392ab8f82fb463aee8f8f2d2e278526b83230b30529968d6d05f7c53cd5c2
SimHash 4701d98356f5

Groups

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

googlebot
googlebot-image
yandex
msnbot
msnbot-media
bingbot
yahoo-blogs
yahoo-mmcrawler

Rule Path
Disallow /image/*
Disallow /tag/*

Other Records

Field Value
crawl-delay 10

Comments

  • robots.txt
  • disallow all
  • but allow only important bots
  • Directories