Log In Sign Up

rodinmuseum.org
robots.txt

Robots Exclusion Standard data for rodinmuseum.org

Resource Scan

Scan Details

Site Domain rodinmuseum.org
Base Domain rodinmuseum.org
Scan Status Ok
Last Scan2024-09-16T02:34:06+00:00
Next Scan 2024-10-16T02:34:06+00:00

Last Scan

Scanned2024-09-16T02:34:06+00:00
URL https://rodinmuseum.org/robots.txt
Domain IPs 15.197.184.37
Response IP 15.197.184.37
Found Yes
Hash 55ab016e03022b78e9845e4390769a39a15956f84beb192f70d07b0b153b4c65
SimHash 2234d6024706

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

flipboard

Rule Path
Disallow

flipboardproxy

Rule Path
Disallow

twitterbot

Rule Path
Allow /

mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)

Rule Path
Allow /

msnbot

Rule Path
Disallow

slurp

Rule Path
Disallow

teoma

Rule Path
Disallow

twiceler

Rule Path
Disallow

gigabot

Rule Path
Disallow

scrubby

Rule Path
Disallow

robozilla

Rule Path
Disallow

nutch

Rule Path
Disallow

ia_archiver

Rule Path
Disallow

baiduspider-ads

Rule Path
Disallow

naverbot

Rule Path
Disallow

yeti

Rule Path
Disallow

yahoo-mmcrawler

Rule Path
Disallow

psbot

Rule Path
Disallow

asterias

Rule Path
Disallow

yahoo-blogs/v3.9

Rule Path
Disallow

baiduspider

Rule Path
Disallow

mozilla/5.0(compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow

spider

Rule Path
Disallow

yandex

Rule Path
Disallow

yandex.com

Rule Path
Disallow

yandexbot/3.0

Rule Path
Disallow

sosospider+

Rule Path
Disallow

ezooms

Rule Path
Disallow

ezooms/1.0

Rule Path
Disallow

sogou spider

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://rodinmuseum.org/sitemap.xml
Back to top

Comments

  • baiduspider
  • yandex
  • sosospider
  • ezooms
  • sogou
Back to top