infignosmedia.com
robots.txt

Robots Exclusion Standard data for infignosmedia.com

Resource Scan

Scan Details

Site Domain infignosmedia.com
Base Domain infignosmedia.com
Scan Status Ok
Last Scan2024-11-01T16:47:04+00:00
Next Scan 2024-11-08T16:47:04+00:00

Last Scan

Scanned2024-11-01T16:47:04+00:00
URL https://infignosmedia.com/robots.txt
Domain IPs 91.247.172.166
Response IP 91.247.172.166
Found Yes
Hash 47aa316a1f10c3d87fcfebaf203fef3b7a9c9f919950da6f08781512a4e82c18
SimHash 631a3150bbd5

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /images/
Disallow /group/

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

spider72.yandex.ru

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

scoutjet

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mn12bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

ccbot

Rule Path
Disallow /

goose

Rule Path
Disallow /

Comments

  • Limit ScoutJet's crawl rate (example is to crawl no more than 1 page every 5 seconds)