administradorbrasil.com
robots.txt

Robots Exclusion Standard data for administradorbrasil.com

Resource Scan

Scan Details

Site Domain administradorbrasil.com
Base Domain administradorbrasil.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-05-20T21:40:58+00:00
Next Scan 2025-08-18T21:40:58+00:00

Last Successful Scan

Scanned2024-07-25T21:39:03+00:00
URL https://administradorbrasil.com/robots.txt
Domain IPs 84.19.191.142
Response IP 84.19.191.142
Found Yes
Hash 229bc7e7041d07af6afda255fb8847861973401ba659ea909aea149b8b74d602
SimHash 5d9477d1c6f9

Groups

adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
apis-google
feedfetcher-google
duplexweb-google
google favicon
googlebot
googlebot-image
googlebot-mobile
googlebot-news
googlebot-video
google-read-aloud
googleweblight
storebot-google
googleother
applebot
bingbot
bingpreview
mediapartners
mediapartners-google
mmsnbot_mobile
msnbot
msnbot-media
adidxbot
duckduckbot
duplexweb-google
exabot
facebot
fast-webcrawler
ia_archiver
scooter
slurp
teoma
yahooseeker/m1a1-r2d2
yandexbot
yandex
swiftbot
ccbot/2.0

Rule Path
Allow /
Disallow /*?nid=*

*

Rule Path
Disallow /
Allow /misc/*.css$
Allow /misc/*.css?
Allow /misc/*.js$
Allow /misc/*.js?
Allow /misc/*.gif
Allow /misc/*.jpg
Allow /misc/*.jpeg
Allow /misc/*.png
Allow /modules/*.css$
Allow /modules/*.css?
Allow /modules/*.js$
Allow /modules/*.js?
Allow /modules/*.gif
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.png
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /themes/*.css$
Allow /themes/*.css?
Allow /themes/*.js$
Allow /themes/*.js?
Allow /themes/*.gif
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.png
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F

Comments

  • CRAWLING ALLOWED ONLY FOR
  • https://www.keycdn.com/blog/web-crawlers
  • https://kinsta.com/de/blog/crawler-liste/#4-apple-bot
  • ORIGINAL DRUPAL 7.97
  • Crawl-delay: 10
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)