Log In Sign Up

imperialmetals.com
robots.txt

Robots Exclusion Standard data for imperialmetals.com

Resource Scan

Scan Details

Site Domain imperialmetals.com
Base Domain imperialmetals.com
Scan Status Ok
Last Scan2024-09-16T04:30:13+00:00
Next Scan 2024-10-16T04:30:13+00:00

Last Scan

Scanned2024-09-16T04:30:13+00:00
URL https://imperialmetals.com/robots.txt
Domain IPs 54.186.121.68
Response IP 54.186.121.68
Found Yes
Hash 68fa8265fe0cf7cba1976c95dd1a9604581ac097a1cc9a45eaacf39314336df2
SimHash 70838f6688b5

Groups

*

Rule Path
Disallow /index.php
Disallow /_add-ons
Disallow /_app
Disallow /_cache
Disallow /_config
Disallow /TRIGGER/honeybots/index

baiduspider-image
googlebot-image
msnbot-media
slurp
yahoo-mmcrawler
yahoo-mmcrawler/3.x
yandeximages
yandexmedia

Rule Path
Disallow /assets

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap /sitemap
Back to top

Comments

  • ___ ___ ___ ___
  • / /\ / /\ / /\ / /\ ___
  • / /::\ / /::\ / /::| / /::\ /__/\
  • / /:/\:\ / /:/\:\ / /:|:| / /:/\:\ \ \:\
  • / /::\ \:\ / /:/ \:\ / /:/|:|__ / /::\ \:\ \__\:\
  • /__/:/\:\_\:\/__/:/ \__\:|/__/:/ |:| /\/__/:/\:\ \:\ / /::\
  • \__\/ \:\/:/\ \:\ / /:/\__\/ |:|/:/\ \:\ \:\_\// /:/\:\
  • \__\::/ \ \:\ /:/ | |:/:/ \ \:\ \:\ / /:/__\/
  • / /:/ \ \:\/:/ |__|::/ \ \:\_\//__/:/
  • /__/:/ \__\::/ /__/:/ \ \:\ \__\/
  • \__\/ ~~ \__\/ \__\/
  • DISCOURAGE BOTS FROM SPIDERING YOUR ASSETS
  • DEFINE SITEMAP URL
  • DON'T CRAWL OFTEN
Back to top