Log In Sign Up

cavsi.com
robots.txt

Robots Exclusion Standard data for cavsi.com

Resource Scan

Scan Details

Site Domain cavsi.com
Base Domain cavsi.com
Scan Status Ok
Last Scan2026-01-15T08:03:56+00:00
Next Scan 2026-01-22T08:03:56+00:00

Last Scan

Scanned2026-01-15T08:03:56+00:00
URL https://cavsi.com/robots.txt
Domain IPs 104.21.35.105, 172.67.217.186, 2606:4700:3031::6815:2369, 2606:4700:3037::ac43:d9ba
Response IP 172.67.217.186
Found Yes
Hash 3a8857b4e6cb447a5708eee38fda718d56abcbe918f96c97371c2de149e22d85
SimHash 2916da800513

Groups

msnbot

Rule Path
Disallow

infonavirobot(f107)

Rule Path
Disallow

tv33_mercator_1-1.0

Rule Path
Disallow

avsearch-3.0

Rule Path
Disallow

teoma

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

nutch

Rule Path
Disallow /

scooter/2.0

Rule Path
Disallow

slurp

Rule Path
Disallow

slurp/2.0

Rule Path
Disallow

scrubby

Rule Path
Disallow

searchenginelicencesheep_v1.0

Rule Path
Disallow

shadow/2.0

Rule Path
Disallow

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow /

psbot

Rule Path
Disallow /

multitext/0.1

Rule Path
Disallow

fast-webcrawler/2.2.5

Rule Path
Disallow

atomz/1.0

Rule Path
Disallow

htdig/ (searchit@netmind.com)

Rule Path
Disallow

spider00.logika.net.

Rule Path
Disallow

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

*

Rule Path
Disallow /m/
Disallow /cgi-bin/
Disallow /espanol/cgi-bin/
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /espanol/wp-admin/
Allow /espanol/wp-admin/admin-ajax.php
Disallow /preguntasrespuestas/cgi-bin
Disallow /preguntasrespuestas/page
Disallow /questionsanswers/cgi-bin
Disallow /questionsanswers/page
Back to top

Other Records

Field Value
sitemap http://www.cavsi.com/sitemap.xml
Back to top

Comments

  • robots.txt for http://www.cavsi.com
Back to top