soccorritori.it
robots.txt

Robots Exclusion Standard data for soccorritori.it

Resource Scan

Scan Details

Site Domain soccorritori.it
Base Domain soccorritori.it
Scan Status Ok
Last Scan2024-09-11T15:31:52+00:00
Next Scan 2024-10-11T15:31:52+00:00

Last Scan

Scanned2024-09-11T15:31:52+00:00
URL http://soccorritori.it/robots.txt
Domain IPs 178.79.136.90
Response IP 178.79.136.90
Found Yes
Hash 238d7b68f5c53485b3a4d1b68deaa64b943d89f45cfeeac010025f9826c9edd2
SimHash 639a9bb0c657

Groups

synthesio crawler

Rule Path
Disallow /

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 600

linguee

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
slurp
yahoo-blogs
yahoo-mmcrawler

Rule Path
Disallow /*-print/
Disallow /*/?nocache=1
Disallow /ajax.php
Disallow /calendar.php
Disallow /cannedreplies.php
Disallow /customavatars/
Disallow /customgroupicons/
Disallow /customprofilepics/
Disallow /faq.php
Disallow /includes/
Disallow /insights/
Disallow /install/
Disallow /iwt/
Disallow /login.php
Disallow /member.php
Disallow /memberlist.php
Disallow /members/
Disallow /misc.php
Disallow /moderator.php
Disallow /newreply.php
Disallow /newthread.php
Disallow /online.php
Disallow /poll.php
Disallow /postings.php
Disallow /printthread.php
Disallow /private.php
Disallow /profile.php
Disallow /register.php
Disallow /report.php
Disallow /reputation.php
Disallow /search.php
Disallow /search.php
Disallow /sendmessage.php
Disallow /showgroups.php
Disallow /subscription.php
Disallow /threadrate.php
Disallow /usercp.php
Disallow /usernote.php
Disallow /vbseocp.php

Comments

  • Tame search engines... it tends to eat a ton of resources without a delay
  • disallow all
  • but allow only important bots
  • Directories