planetanya.org.il
robots.txt

Robots Exclusion Standard data for planetanya.org.il

Resource Scan

Scan Details

Site Domain planetanya.org.il
Base Domain planetanya.org.il
Scan Status Ok
Last Scan2024-11-14T08:33:26+00:00
Next Scan 2024-11-21T08:33:26+00:00

Last Scan

Scanned2024-11-14T08:33:26+00:00
URL https://planetanya.org.il/robots.txt
Domain IPs 212.179.58.114
Response IP 212.179.58.114
Found Yes
Hash 48216e7c23f892eee5f0fad3d1047280bcf197d50c981ce87575291dcc1afa53
SimHash 2015dc825513

Groups

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

msnbot

Rule Path
Disallow

slurp

Rule Path
Disallow

teoma

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

nutch

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow

baiduspider

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow

psbot

Rule Path
Disallow

asterias

Rule Path
Disallow /

yahoo-blogs/v3.9

Rule Path
Disallow /

*

Rule Path
Disallow
Disallow /Common/
Disallow /EDITOR/
Disallow /*.swf$
Disallow /*.xml$