Log In Sign Up

mavenart.com
robots.txt

Robots Exclusion Standard data for mavenart.com

Resource Scan

Scan Details

Site Domain mavenart.com
Base Domain mavenart.com
Scan Status Ok
Last Scan2024-06-06T16:25:11+00:00
Next Scan 2024-07-06T16:25:11+00:00

Last Scan

Scanned2024-06-06T16:25:11+00:00
URL https://mavenart.com/robots.txt
Redirect https://www.mavenart.com/robots.txt
Redirect Domain www.mavenart.com
Redirect Base mavenart.com
Domain IPs 172.66.40.151, 172.66.43.105, 2606:4700:3108::ac42:2897, 2606:4700:3108::ac42:2b69
Redirect IPs 172.66.40.151, 172.66.43.105, 2606:4700:3108::ac42:2897, 2606:4700:3108::ac42:2b69
Response IP 172.66.40.151
Found Yes
Hash a85d483be732907945b734cfca7ae3428476c45ad4f2da517db639fc81073a56
SimHash 8e2cbc5103f1

Groups

*

Rule Path Comment
Allow /*?p= -
Allow /index.php/blog/ -
Allow /media/catalog/ -
Disallow /404/ -
Disallow /app/ -
Disallow /cgi-bin/ -
Disallow /downloader/ -
Disallow /errors/ -
Disallow /includes/ -
Disallow /js/ -
Disallow /lib/ -
Disallow /magento/ -
Disallow /media/captcha/ -
Disallow /media/customer/ -
Disallow /media/dhl/ -
Disallow /media/downloadable/ -
Disallow /media/import/ -
Disallow /media/pdf/ -
Disallow /media/sales/ -
Disallow /media/tmp/ -
Disallow /media/xmlconnect/ -
Disallow /pkginfo/ -
Disallow /report/ -
Disallow /scripts/ -
Disallow /shell/ -
Disallow /stats/ -
Disallow /var/ -
Disallow /index.php/ -
Disallow /catalog/product_compare/ -
Disallow /catalog/category/view/ -
Disallow /catalog/product/view/ -
Disallow /catalog/product/gallery/ -
Disallow /checkout/ -
Disallow /control/ -
Disallow /contacts/ -
Disallow /customer/ -
Disallow /customize/ -
Disallow /newsletter/ -
Disallow /poll/ -
Disallow /review/ -
Disallow /sendfriend/ -
Disallow /tag/ -
Disallow /wishlist/ -
Disallow /cron.php -
Disallow /cron.sh -
Disallow /error_log -
Disallow /install.php -
Disallow /LICENSE.html -
Disallow /LICENSE.txt -
Disallow /LICENSE_AFL.txt -
Disallow /STATUS.txt -
Disallow /get.php Magento 1.5+
Disallow /*.php$ -
Disallow /*?SID= -
Disallow /rss* -
Disallow /*PHPSESSID -
Back to top

Other Records

Field Value
sitemap https://www.mavenart.com/sitemap.xml
Back to top

Comments

  • Website Sitemap
  • Crawlers Setup
  • Crawl-delay: 10
  • Allowable Index
  • Mind that Allow is not an official standard
  • Allow: /catalogsearch/result/
  • Directories
  • Disallow: /media/
  • Disallow: /media/catalog/
  • Disallow: /media/css/
  • Disallow: /media/css_secure/
  • Disallow: /media/js/
  • Disallow: /skin/
  • Paths (clean URLs)
  • Files
  • Paths (no clean URLs)
  • Disallow: /*.js$
  • Disallow: /*.css$
Back to top