aemter.org
robots.txt

Robots Exclusion Standard data for aemter.org

Resource Scan

Scan Details

Site Domain aemter.org
Base Domain aemter.org
Scan Status Ok
Last Scan2025-05-19T10:47:45+00:00
Next Scan 2025-05-26T10:47:45+00:00

Last Scan

Scanned2025-05-19T10:47:45+00:00
URL https://aemter.org/robots.txt
Redirect https://www.aemter.org/robots.txt
Redirect Domain www.aemter.org
Redirect Base aemter.org
Domain IPs 62.141.57.151
Redirect IPs 62.141.57.151
Response IP 62.141.57.151
Found Yes
Hash c3d764e90ea902e3a5d725914dc57f8eb1e050977e65e4d892c7db01986e3468
SimHash 0f9497a1c6f0

Groups

adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
feedfetcher-google
google favicon
googlebot
googlebot-image
googlebot-mobile
googlebot-news
googlebot-video
google-read-aloud
apis-google
bingbot
bingpreview
mediapartners
mediapartners-google
mmsnbot_mobile
msnbot
msnbot-media
adidxbot
baiduspider
baiduspider-image
baiduspider-mobile
baiduspider-news
baiduspider-video
duckduckbot
duplexweb-google
exabot
facebot
fast-webcrawler
ia_archiver
scooter
slurp
teoma
yahooseeker/m1a1-r2d2
yandexbot
yandex

Rule Path
Allow /
Disallow /*?nid=*

*

Rule Path
Disallow /
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /sites/
Disallow /themes/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /admin/
Disallow /comment/reply/
Disallow /contact/
Disallow /logout/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=contact%2F
Disallow /?q=logout%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F

Comments

  • CRAWLING ALLOWED ONLY FOR
  • https://www.keycdn.com/blog/web-crawlers
  • ORIGINAL DRUPAL 5.23
  • Crawl-delay: 10
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)