web-generalist.com
robots.txt

Robots Exclusion Standard data for web-generalist.com

Resource Scan

Scan Details

Site Domain web-generalist.com
Base Domain web-generalist.com
Scan Status Ok
Last Scan2024-09-24T21:42:35+00:00
Next Scan 2024-10-01T21:42:35+00:00

Last Scan

Scanned2024-09-24T21:42:35+00:00
URL https://web-generalist.com/robots.txt
Domain IPs 157.112.176.27
Response IP 157.112.176.27
Found Yes
Hash 275ed460cbef565c63395e423c281e915fca8c33047b1d905c139ec67af932b3
SimHash 40d3075173d4

Groups

slurp

Rule Path
Disallow /m/
Disallow /mt/
Disallow /main/calendar/

y!j-srd/1.0

Rule Path
Allow /m/
Disallow /mt/
Disallow /main/calendar/

y!j-mbs/1.0

Rule Path
Allow /m/
Disallow /mt/
Disallow /main/calendar/

googlebot

Rule Path
Disallow /m/
Disallow /mt/
Disallow /main/calendar/

googlebot-mobile

Rule Path
Allow /m/
Disallow /mt/
Disallow /main/calendar/

ichiro/mobile goo

Rule Path
Allow /m/
Disallow /mt/
Disallow /main/calendar/

ld_mobile_bot

Rule Path
Allow /m/
Disallow /mt/
Disallow /main/calendar/

moba-crawler

Rule Path
Allow /m/
Disallow /mt/
Disallow /main/calendar/

rfcrawler-mobile/1.0

Rule Path
Allow /m/
Disallow /mt/
Disallow /main/calendar/