Log In Sign Up

eadventist.net
robots.txt

Robots Exclusion Standard data for eadventist.net

Resource Scan

Scan Details

Site Domain eadventist.net
Base Domain eadventist.net
Scan Status Ok
Last Scan2024-09-25T13:36:10+00:00
Next Scan 2024-10-25T13:36:10+00:00

Last Scan

Scanned2024-09-25T13:36:10+00:00
URL https://eadventist.net/robots.txt
Domain IPs 160.238.31.51
Response IP 160.238.31.51
Found Yes
Hash 8125d27d7e66a34dea074ef54e7ee7f39e2e4ec34fa5f2716f38c1413e83a1ec
SimHash 2059f9a576f5

Groups

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 600

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
slurp
yahoo-blogs
yahoo-mmcrawler

Rule Path
Disallow /logout/
Disallow /login/
Disallow /?q=admin%2F
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • disallow all
  • but allow only important bots
  • Directories
  • Disallow: /includes/
  • Files
  • Disallow: /CHANGELOG.txt
  • Paths (clean URLs)
  • Paths (no clean URLs)
Back to top