Log In Sign Up

guide-spain.com
robots.txt

Robots Exclusion Standard data for guide-spain.com

Resource Scan

Scan Details

Site Domain guide-spain.com
Base Domain guide-spain.com
Scan Status Ok
Last Scan2024-11-09T18:41:42+00:00
Next Scan 2024-11-16T18:41:42+00:00

Last Scan

Scanned2024-11-09T18:41:42+00:00
URL http://guide-spain.com/robots.txt
Redirect http://www.guide-spain.com/robots.txt
Redirect Domain www.guide-spain.com
Redirect Base guide-spain.com
Domain IPs 79.98.111.192
Redirect IPs 79.98.111.192
Response IP 79.98.111.192
Found Yes
Hash f2895b444126788551e2d77330788b59e700f9306ce2c280038fb3d9aa8a6122
SimHash 023cd0632a71

Groups

*

Rule Path
Disallow /bot_trap_page

*

Rule Path
Disallow /administrator/bot

bubing

Rule Path
Disallow /

icc-crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

xovibot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

ahrefsbot

Rule Path
Disallow /

seokicks robot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mail.ru/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mail.ru

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

grapeshot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5

urlappendbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 500

turnitinbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Back to top

Warnings

  • 4 invalid lines.
Back to top