Log In Sign Up

owboard.oraweb.it
robots.txt

Robots Exclusion Standard data for owboard.oraweb.it

Resource Scan

Scan Details

Site Domain owboard.oraweb.it
Base Domain oraweb.it
Scan Status Ok
Last Scan2025-10-05T05:34:11+00:00
Next Scan 2025-11-04T05:34:11+00:00

Last Scan

Scanned2025-10-05T05:34:11+00:00
URL https://owboard.oraweb.it/robots.txt
Domain IPs 104.21.60.186, 172.67.200.50, 2606:4700:3034::ac43:c832, 2606:4700:3035::6815:3cba
Response IP 104.21.60.186
Found Yes
Hash a770b10fad7b89c907dc698277f74c1dd7bc5b2f8d3942bd65b8f0b2ca59d544
SimHash b70e5940c2f1

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

petalbot

Rule Path
Disallow /

*

Rule Path
Disallow /copyright_license.html
Back to top

Comments

  • Simple Robots.txt 0.1
  • Crawlers Setup
  • Directories
  • Disallow: /404/
  • Disallow: /app/
  • Disallow: /cgi-bin/
  • Disallow: /downloader/
  • Disallow: /errors/
  • Disallow: /includes/
  • Paths (clean URLs)
  • Disallow: /index.php/
  • Disallow: /catalog/product_compare/
  • Disallow: /catalog/category/view/
  • Files
  • Disallow: /cron.php
  • Disallow: /cron.sh
  • Disallow: /sheduler_cron.sh
  • Disallow: /error_log
  • Disallow: /install.php
  • Disallow: /LICENSE.html
  • Disallow: /LICENSE.txt
  • Disallow: /STATUS.txt
  • Paths (no clean URLs)
  • Disallow: /*.js$
  • Disallow: /*.css$
  • Disallow: /*.php$
  • Disallow: /*?p=*&
  • Disallow: /*?limit=*
  • Disallow: /*?dir=*
  • Disallow: /*?order=*
  • Disallow: /*?l=*
  • Disallow: /*?SID=
Back to top

Warnings

  • 1 invalid line.
Back to top